Sagas
A transaction is the right tool when every write is in one database. It is no help when the steps are in four places: charge a card, reserve stock, book a courier, send the receipt. A saga is the answer, and grit generate workflow writes one.
The problem it solves
The card does not roll back when the courier refuses. The process holding all of this in its head is exactly the thing that crashes, usually between the charge and the reservation, and what is left is a customer charged for nothing and no record of why.
A saga gives each step a Do and an Undo. The steps run in order, and when one fails for good the completed ones are undone newest first. Where the run got to is a row in saga_runs, not a stack frame, so a process that dies resumes rather than losing the thread.
Generating one
grit generate workflow Checkout --steps "charge,reserve_stock,book_courier,send_receipt"
That writes internal/sagas/checkout.go with the steps stubbed out and registers it. Each Do returns an error until you write it, so a half-built saga fails loudly rather than reporting success and doing nothing: the first you would otherwise hear of it is a customer saying the parcel never came.
func Checkout() saga.Definition {return saga.Definition{Name: "checkout",Steps: []saga.Step{{Name: "charge",Do: func(ctx context.Context, r *saga.Run) error {var in CheckoutInputif err := r.Input(&in); err != nil {return err}charge, err := payments.Charge(ctx, in.OrderID, payments.Idempotency(r.IdempotencyKey()))if err != nil {if payments.Declined(err) {return saga.Fatal(err) // retrying will not fix a declined card}return err}r.Set("charge_id", charge.ID)return nil},Undo: func(ctx context.Context, r *saga.Run) error {id := r.GetString("charge_id")if id == "" {return nil // the charge never landed, so there is nothing to refund}return payments.Refund(ctx, id)},},// ...},}}
Starting a run
run, err := saga.Start(ctx, db, "checkout", CheckoutInput{OrderID: order.ID},saga.Key("order:"+order.ID))
It records the run and returns at once. A runner advances it, and every replica runs one: a run is claimed before it is touched, so two of them cannot execute the same step and charge a card twice. The key makes the start idempotent, which is what you want behind a retried request or a redelivered webhook.
Start it in the same transaction as the write that justifies it and the two commit together, the way outbox.Enqueue does.
The three rules
- Every Do must be idempotent. A crash between the side effect and the record of it is not preventable, so a resumed run will sometimes repeat a step. Pass
r.IdempotencyKey()to whatever you are calling: it is stable for a given run and step, so a provider that deduplicates on it will not charge twice. - Every Undo must be idempotent too, and must tolerate a
Dothat never finished. Compensation runs because something went wrong, and that includes not knowing whether the charge landed. Check before you reverse. - Steps you cannot undo go last. An email cannot be unsent, so
Undo: nilis a legitimate thing to write. The consequence is the ordering: a step after it that fails will compensate everything before it and leave that email sent. Sending the receipt before the parcel is booked is a bug you only find in production.
r.Set and r.Get, because by the time the refund runs the process that made the charge is usually gone. What a step writes is saved even when that step then fails, which is deliberate: a charge id obtained just before a timeout is exactly what the refund needs.What the statuses mean
running: working through the steps.compensating: a step failed for good and the completed ones are being undone.done: every step completed.compensated: a step failed and everything before it was undone. The world is back where it started, which is the good outcome of a bad run.stuck: a compensation itself failed past its attempts. This is the one that needs a person. Something happened that could not be taken back, and the run says so instead of carrying on with a status that reads as resolved.
A compensation is never skipped, because skipping one is how money goes missing. A failed step is retried with exponential backoff and jitter; saga.Fatal(err) skips the retries for a failure that retrying will not fix.
Finding a stuck run
The admin has a screen at /system/sagas: every run, filtered by status, with its steps in order and the error that stopped each one. A stuck run gets a Retry button, which is the only state where a person pressing something is the right answer. It resumes compensating from the step it stopped on, because the thing that could not be undone is still not undone and whoever fixed the refund API wants that same undo attempted again.
Without the screen the only way to find a stuck run is SQL against saga_runs, which means nobody finds one until a customer complains.
A whole saga, end to end
Checkout: take the money, hold the stock, book the courier, email the receipt. Four systems, none of which rolls back because another one failed. The order matters and is explained under it.
package sagasimport ("context""myapp/apps/api/internal/saga")type CheckoutInput struct {OrderID string `json:"order_id"`Email string `json:"email"`}func Checkout() saga.Definition {return saga.Definition{Name: "checkout",Steps: []saga.Step{{Name: "charge",Do: func(ctx context.Context, r *saga.Run) error {var in CheckoutInputif err := r.Input(&in); err != nil {return err}// IdempotencyKey is stable for this run and this step, so a// replay after a crash reaches the same charge rather than a// second one.charge, err := payments.Charge(ctx, in.OrderID,payments.Idempotency(r.IdempotencyKey()))if err != nil {if payments.Declined(err) {// Retrying a declined card spends five attempts to// arrive at the same answer.return saga.Fatal(err)}return err}r.Set("charge_id", charge.ID)return nil},Undo: func(ctx context.Context, r *saga.Run) error {id := r.GetString("charge_id")if id == "" {// The Do never got far enough to charge anything.return nil}return payments.Refund(ctx, id)},},{Name: "reserve_stock",Do: func(ctx context.Context, r *saga.Run) error {var in CheckoutInputif err := r.Input(&in); err != nil {return err}ref, err := inventory.Reserve(ctx, in.OrderID)if err != nil {return err}r.Set("reservation", ref)return nil},Undo: func(ctx context.Context, r *saga.Run) error {ref := r.GetString("reservation")if ref == "" {return nil}return inventory.Release(ctx, ref)},},{Name: "book_courier",MaxAttempts: 8, // their API is flaky in a way ours is notDo: func(ctx context.Context, r *saga.Run) error {var in CheckoutInputif err := r.Input(&in); err != nil {return err}booking, err := courier.Book(ctx, in.OrderID)if err != nil {return err}r.Set("booking", booking.Reference)return nil},Undo: func(ctx context.Context, r *saga.Run) error {ref := r.GetString("booking")if ref == "" {return nil}return courier.Cancel(ctx, ref)},},{Name: "send_receipt",// No Undo: an email cannot be unsent. That is why it is last.Do: func(ctx context.Context, r *saga.Run) error {var in CheckoutInputif err := r.Input(&in); err != nil {return err}return mail.Receipt(ctx, in.Email, r.GetString("charge_id"))},},},}}
Why the receipt is last. A step with no Undo is allowed, and the ordering is the design. Put the receipt second and a courier failure compensates the charge and the stock, and leaves a customer holding an email for an order that no longer exists. The rule is: the things you cannot take back go after the things you can.
Why every Undo checks first. Compensation exists because something went wrong, and "went wrong" includes "I do not know whether the charge landed". Each Undo above reads what the Do recorded and returns cleanly when it finds nothing, because it will be called for steps that never finished.
Why state goes through r.Set. By the time a compensation runs, the process that charged the card is usually gone. The run's state is a column, so r.Get is the only thing an Undo has. A closure variable would be a different process's memory.
What the runner actually does
A background runner ticks every five seconds, claims the runs that are due, and advances each by one step. The claim is a row update, so with four replicas exactly one of them is advancing a given run: a step is never racing another copy of itself. The runner does not start at all when no saga is registered, rather than polling an empty table for the life of the process.
A failing step is retried with exponential backoff, five attempts by default, starting at five seconds and doubling to a fifteen-minute ceiling, with jitter so a thousand runs waiting on one outage do not retry in the same millisecond. Those are defaults on the definition; a step can override MaxAttempts for itself, and 1 means do not retry.
Past its attempts the run turns to compensating and the completed steps are undone newest first. If a compensation itself fails past its attempts the run goes stuck, which is the status that needs a person. A compensation is never skipped on the way past, because skipping one is how money goes missing.
Is this Temporal?
No, and the difference is worth being precise about, because the word "saga" names a pattern and Temporal is a product.
Both implement the same pattern, and the same flavour of it: orchestrated, where one coordinator calls each step in turn and decides when to compensate. The alternative is choreography, where services react to each other's events and no single component knows the whole sequence. Grit's engine and Temporal are both the first kind, so the shape of the code is recognisably similar: steps, compensations registered as you go, compensations run in reverse.
What differs is where the durability lives, and what you have to run.
| internal/saga | Temporal | |
|---|---|---|
| To run it | Nothing. Two tables in the database you already have. | A Temporal cluster, plus worker processes polling a task queue. |
| Where progress lives | A saga_runs row and a saga_steps row per step. | An event history in the cluster, replayed to rebuild workflow state. |
| How code resumes | The next step is read from the row and called. | The whole workflow function is replayed from its history. |
| Constraints on your code | A step must be idempotent. Nothing else. | Workflow code must be deterministic: no time.Now, no rand, no direct I/O. |
| Waiting and timers | Retry backoff. No durable sleeps or signals. | Durable timers, signals, queries, child workflows, versioning. |
| Scale it is built for | One application’s processes, sharing one database. | Many services and languages, millions of concurrent executions. |
The determinism row is the one that bites people. Temporal rebuilds a workflow's state by replaying its code against a recorded history, so that code may not do anything that could answer differently the second time: no time.Now(), no rand, no database calls outside an activity. Grit replays nothing. It reads which step is next out of a row and calls it, so a step is ordinary Go that may do whatever it likes, and the single rule is that running it twice must be safe.
That is the trade in both directions. If you need durable timers, signals into a running process, child workflows or many services in several languages sharing one engine, that is what Temporal is for and this is not a substitute. If what you have is one Go application whose checkout touches four APIs, a cluster is a large piece of infrastructure to operate for a problem two tables solve.
Every command
# Write a saga with its steps stubbed out, and register itgrit generate workflow Checkout --steps "charge,reserve_stock,book_courier,send_receipt"# Create the two tables (saga_runs, saga_steps)grit migrate# What the project has, and what is runninggrit routes | grep sagas
There is no command to start a run: a saga is started from your own code, because the thing that knows an order was placed is the handler that placed it.
run, err := saga.Start(ctx, db, "checkout", sagas.CheckoutInput{OrderID: order.ID,Email: order.Email,})
Watching a run
The admin has a screen at System → Sagas, listing runs by status with each run's steps and the error that stopped it. It is behind ADMIN or jobs.view. Finding a stuck run should not mean writing SQL against saga_runs, because nobody does that until a customer complains.
GET /api/v1/admin/sagas # runs, filterable by statusGET /api/v1/admin/sagas/definitions # what this project has registeredGET /api/v1/admin/sagas/:id # one run, with its stepsPOST /api/v1/admin/sagas/:id/retry # push a stuck run on again
retry is for the stuck ones: you have fixed whatever made the compensation fail, and you want the run to carry on undoing rather than stay half-undone forever.
Not the same as a workflow field
Workflows turn a status column into a state machine: the states are one record's own, and the transitions are usually a person clicking a button. A saga is the other thing: work that crosses systems, with nobody watching, that has to either finish or be undone. A project often wants both, and they do not overlap.
