The transactional outbox pattern in Go with Postgres
You write a row and publish an event. Two systems, no shared transaction, so there is a moment where one has happened and the other has not. The process can die in that moment, and at any volume, it eventually will.
The outbox makes the event part of the same transaction as the write, so they commit together or not at all, and a separate process delivers it afterwards. It is one table and about a hundred lines, and it converts a silent data-loss bug into a visible queue.
18 min read. Commands and output last run against Grit v3.560.0.
What you will have at the end
An outbox table, a relay with SKIP LOCKED, and delivery you can inspect.
At-least-once delivery, with the consumer-side idempotency that makes at-least-once usable.
A clear sense of when this is overkill, which is often.
The dual write
func (s *OrderService) Place(ctx context.Context, o *models.Order) error {if err := s.db.WithContext(ctx).Create(o).Error; err != nil {return fmt.Errorf("create order: %w", err)}// ←── the process can die here ──→return s.queue.Enqueue(ctx, jobs.NewOrderPlaced(o.ID))}
Three outcomes, and only one of them is the one you wanted:
- Both succeed. Fine, and this is almost always what happens, which is why the bug survives testing.
- The insert fails. Fine: nothing happened, the caller gets an error.
- The insert succeeds and the enqueue does not. The order exists. No confirmation email, no warehouse notification, no analytics event, and nothing anywhere records that this happened.
Reversing the order does not help: now you can send a confirmation for an order that was never created, which is worse. Retrying the enqueue in a loop does not help either, because the process can die during the retries.
Enqueuing inside the transaction is a different bug
Moving the enqueue inside db.Transaction looks like a fix and is not: Redis knows nothing about your transaction, so a worker can pick the job up before the commit, fail to find the row, and consume a retry. Worse, if the transaction then rolls back, you have published an event for something that never existed.
The pattern
The insight is small: the event is a row, so it is covered by the same transaction as everything else. Delivery becomes a separate, retryable problem that cannot lose anything, because the record of what is owed survives a crash.
The table
CREATE TABLE outbox_events (id bigserial PRIMARY KEY,aggregate_type text NOT NULL, -- 'order'aggregate_id bigint NOT NULL, -- 8821event_type text NOT NULL, -- 'order.placed'payload jsonb NOT NULL,created_at timestamptz NOT NULL DEFAULT now(),published_at timestamptz,attempts int NOT NULL DEFAULT 0,next_attempt_at timestamptz NOT NULL DEFAULT now(),last_error text);-- The relay's only query: unpublished, due, oldest first. A partial-- index keeps it small, because the published rows are the vast-- majority and the relay never looks at them.CREATE INDEX idx_outbox_pendingON outbox_events (next_attempt_at, id)WHERE published_at IS NULL;
Keeping the published rows is worth it: they are an audit trail of everything the system emitted, and the question "did we send that" is asked often. Prune them on a schedule rather than deleting on publish.
Writing the event
// The tx argument is the whole point. A Write that opened its own// transaction would be a dual write with extra steps.func Write(ctx context.Context, tx *gorm.DB, e Event) error {payload, err := json.Marshal(e.Payload)if err != nil {return fmt.Errorf("marshal %s: %w", e.Type, err)}return tx.WithContext(ctx).Create(&models.OutboxEvent{AggregateType: e.AggregateType,AggregateID: e.AggregateID,EventType: e.Type,Payload: payload,}).Error}
func (s *OrderService) Place(ctx context.Context, o *models.Order) error {return s.db.Transaction(func(tx *gorm.DB) error {if err := tx.Create(o).Error; err != nil {return fmt.Errorf("create order: %w", err)}// Same transaction. Both, or neither. There is no window.return outbox.Write(ctx, tx, outbox.Event{AggregateType: "order",AggregateID: o.ID,Type: "order.placed",Payload: OrderPlaced{OrderID: o.ID, Total: o.Total},})})}
Put the data in the payload, not just the id
This is the one place the usual advice reverses. A background job should carry an id and read the current row. An event describes something that happened at a moment, and a consumer reading the row later sees a state that may have changed three times since. Put what the event means in the payload, and version it.
The relay
func (r *Relay) tick(ctx context.Context) error {return r.db.Transaction(func(tx *gorm.DB) error {var batch []models.OutboxEvent// FOR UPDATE SKIP LOCKED is what lets several relay instances run// at once: each takes rows nobody else is holding, instead of// queueing behind them. Without it, two relays serialise, and// without the lock at all, two relays publish the same event.err := tx.Raw(`SELECT * FROM outbox_eventsWHERE published_at IS NULL AND next_attempt_at <= now()ORDER BY idLIMIT 100FOR UPDATE SKIP LOCKED`).Scan(&batch).Errorif err != nil {return fmt.Errorf("claim batch: %w", err)}for i := range batch {e := &batch[i]if err := r.publish(ctx, e); err != nil {// Not a failure of the tick. Record it, back off, carry on// with the rest of the batch.r.backoff(tx, e, err)continue}now := time.Now()if err := tx.Model(e).Update("published_at", now).Error; err != nil {return fmt.Errorf("mark published %d: %w", e.ID, err)}}return nil})}func (r *Relay) backoff(tx *gorm.DB, e *models.OutboxEvent, cause error) {e.Attempts++// Exponential with a ceiling, and jitter so a provider coming back// up is not hit by every parked event in the same second.delay := time.Duration(math.Pow(2, float64(e.Attempts))) * time.Secondif delay > 15*time.Minute {delay = 15 * time.Minute}delay += time.Duration(rand.Int63n(int64(delay / 4)))tx.Model(e).Updates(map[string]any{"attempts": e.Attempts,"next_attempt_at": time.Now().Add(delay),"last_error": cause.Error(),})}
Poll every second or two. For lower latency, have the writer send a notification after commit so the relay wakes immediately; keep the poll as the fallback, because the notification is the part that can be lost and the poll is the part that cannot.
-- Optional: the relay LISTENs, the writer NOTIFYs after commit.-- Latency drops from seconds to milliseconds, and correctness still-- rests on the poll underneath.NOTIFY outbox_new;
At-least-once means the consumer must cope
An outbox guarantees that an event is delivered at least once. It cannot guarantee exactly once: the publish can succeed and the process die before the row is marked, and the next tick will send it again. That is not a flaw to engineer around, it is the contract.
// Every event carries its id, and the consumer records what it has// processed. The unique index does the arbitration, so two concurrent// deliveries cannot both proceed.type ProcessedEvent struct {EventID uint64 `gorm:"primaryKey"`ProcessedAt time.Time}func (c *Consumer) Handle(ctx context.Context, e Event) error {return c.db.Transaction(func(tx *gorm.DB) error {err := tx.Create(&ProcessedEvent{EventID: e.ID, ProcessedAt: time.Now()}).Errorif isUniqueViolation(err) {return nil // already done, by us or by a duplicate delivery}if err != nil {return fmt.Errorf("claim event %d: %w", e.ID, err)}// The work, in the same transaction as the claim, so a crash// rolls back both and the event is retried cleanly.return c.apply(ctx, tx, e)})}
Claim and work in one transaction. Claim first and work after, in a separate transaction, and a crash between them loses the event permanently, which is the bug the outbox existed to prevent, reintroduced at the other end.
Operating it
- Alert on the oldest pending event, not on the count. A thousand events enqueued a second ago is healthy; one event pending for two hours is a stuck relay or a permanently failing publish.
- Park after a threshold. After roughly ten attempts, stop retrying and flag it. An event retrying forever consumes capacity and hides the queue depth that would have told you something was wrong.
- Prune published rows on a schedule. The partial index keeps the relay fast regardless, but the table itself will grow without bound.
- Order is per aggregate, not global. If a consumer needs
order.placedbeforeorder.shipped, either publish in id order within an aggregate and process serially per aggregate, or make the consumer tolerate arrival out of order. Global ordering across a parallel relay does not exist. - Watch the payload size. An outbox row with a megabyte of JSON in it turns your main database into a message broker with poor ergonomics.
When not to do this
The outbox is for when losing an event is a real problem. Much of what people put through one is not.
- A welcome email can be lost occasionally. Log the enqueue failure and move on; an outbox here is machinery for an incident that costs an apology.
- Cache invalidation usually has a TTL underneath it, which is a correctness floor the outbox is not adding to.
- Anything with a reconciliation job already has a recovery path. A nightly sweep that finds orders with no shipment is simpler than an outbox and covers more failure modes.
Use it for the events that move money, notify another system of record, or cannot be reconstructed later: payment captured, order placed, subscription cancelled, document signed.
Generated
# The outbox table, its migration and the relay are scaffolded into# every project. Writing an event is a call, not a command:## outbox.Write(ctx, tx, outbox.Event{Type: "order.placed", ...})$grit migrate
That writes the table, the migration, the relay with SKIP LOCKED and jittered backoff, the parked-event handling, and an admin screen showing pending, retrying and parked events with their last error and a retry button.
The screen is the part worth having. An outbox with no visibility is a queue you cannot reason about, and the first time something is stuck you will want the last error rather than a row count.
The outbox reference covers the generated relay and the admin screen.
Frequently asked questions
- Why not use a queue that supports transactions?
- Very few do across a different database, and those that do, such as a Postgres-backed queue like River, are in effect an outbox with the relay built in. That is a good answer: if your queue lives in the same Postgres as your data, you can enqueue inside the transaction and the whole problem disappears. The pattern here is what you need when the queue is Redis, a broker, or someone else's HTTP endpoint.
- Does this give me exactly-once delivery?
- No, and nothing does across a network. It gives at-least-once with no loss, which is the strongest guarantee available. Exactly-once processing is achievable, and it is achieved at the consumer, with a processed-events table and a unique index, as above.
- How do I keep events in order?
- Per aggregate, by publishing in id order and processing one aggregate at a time. Global ordering is not achievable with a parallel relay, and wanting it is usually a sign the consumer should be tolerant of order instead. Designing consumers to handle arrival out of order is cheaper than enforcing a total order.
- What about change data capture instead?
- Reading the write-ahead log with Debezium captures every change with no application code, which is powerful and couples your consumers to your schema: they see table rows rather than domain events, so a column rename becomes a breaking change for them. An outbox publishes an event you designed and can version. Worth the extra table for anything consumers outside your team rely on.
Next steps
- Background jobs in GoThe queue this delivers into, and why enqueuing in a transaction is wrong.
- Sagas and compensating transactionsWhen the work spans several services and some of it has to be undone.
- Outbox referenceThe generated table, relay and admin screen.
- Webhooks: verification and idempotencyThe same at-least-once problem, seen from the receiving end.
