Background jobs (asynq)

Some work shouldn't happen inside a request: sending emails, resizing images, generating PDFs, syncing third-party data. Background jobs handle it asynchronously — your handler enqueues, returns instantly, a worker processes later. Grit ships asynq as the job engine.

Why we need it

Three reasons to push work to a job:

• Speed up requests. Sending email = 200ms; the user shouldn't wait. Enqueue, return in 5ms.
• Retries on failure. External API hiccupped? The job retries automatically with backoff. The handler would have just 500'd.
• Scheduled work. Cron-like — nightly digest, hourly cleanup, weekly report. Without a job system, you spawn a cron container that's hard to monitor.

The two halves

The API enqueues; the worker dequeues. They're separate Go processes. Both connect to Redis. You can scale workers independently — 10x more worker pods if your queue grows without touching the API.

Where it lives

apps/api/internal/jobs/
├── client.go        ← Enqueue(taskName, payload)
├── handlers.go      ← Register handlers for each task name
├── server.go        ← Worker process bootstrap
└── cron.go          ← Cron schedule definitions
cmd/worker/main.go   ← Worker entry point (separate binary)

How it's implemented

package jobs

import (
  "context"
  "encoding/json"
  "github.com/hibiken/asynq"
)

type Client struct {
  inner *asynq.Client
}

func NewClient(redisURL string) *Client {
  opt, _ := asynq.ParseRedisURI(redisURL)
  return &Client{inner: asynq.NewClient(opt)}
}

func (c *Client) Enqueue(ctx context.Context, taskName string, payload any) error {
  b, err := json.Marshal(payload)
  if err != nil { return err }
  _, err = c.inner.EnqueueContext(ctx, asynq.NewTask(taskName, b))
  return err
}

func (c *Client) EnqueueIn(ctx context.Context, taskName string, payload any, delay time.Duration) error {
  b, _ := json.Marshal(payload)
  _, err := c.inner.EnqueueContext(ctx, asynq.NewTask(taskName, b), asynq.ProcessIn(delay))
  return err
}

Two operations: enqueue now, enqueue in delay (great for "send reminder in 24h").

Defining a job handler

type Handlers struct {
  mail  *mail.Service
  store *storage.Service
}

func (h *Handlers) Register(mux *asynq.ServeMux) {
  mux.HandleFunc("send_welcome",     h.SendWelcome)
  mux.HandleFunc("resize_image",     h.ResizeImage)
  mux.HandleFunc("cleanup_uploads",  h.CleanupUploads)
}

func (h *Handlers) SendWelcome(ctx context.Context, t *asynq.Task) error {
  var p struct{ To, Name string }
  if err := json.Unmarshal(t.Payload(), &p); err != nil {
    return fmt.Errorf("bad payload: %w", err)
  }
  return h.mail.Send(ctx, p.To, "welcome.html", "Welcome!", p)
}

Each task name maps to a handler function. Return nil for success; return an error to trigger retry. asynq re-runs the job up to N times with exponential backoff, then dead-letters it.

The worker process

func main() {
  cfg := config.Load()
  db := db.MustOpen(cfg)
  services := services.New(db, cfg)

  srv := asynq.NewServer(
    asynq.RedisClientOpt{Addr: cfg.RedisAddr},
    asynq.Config{
      Concurrency: 10,                                   // workers per process
      Queues: map[string]int{
        "critical": 6,                                  // priority weight
        "default":  3,
        "low":      1,
      },
    },
  )

  mux := asynq.NewServeMux()
  jobs.NewHandlers(services).Register(mux)

  log.Fatal(srv.Run(mux))
}

Run this as a separate process: go run cmd/worker/main.go in dev, or a separate deploy unit in prod. Two processes, one DB, one Redis.

Cron — scheduled jobs

func RegisterCron(s *asynq.Scheduler) error {
  // Every day at 2am
  _, err := s.Register("0 2 * * *", asynq.NewTask("nightly_digest", nil))
  if err != nil { return err }

  // Every hour
  _, err = s.Register("0 * * * *", asynq.NewTask("cleanup_uploads", nil))
  if err != nil { return err }

  return nil
}

Standard cron syntax. The scheduler runs as part of the worker; it enqueues the task at the right time, the same workers process it.

The Jobs admin page

Grit's admin panel at /admin/system/jobs shows:

• Queue depth per queue (active, pending, retry, dead).
• Recent runs with status + duration.
• Failed jobs — click to inspect payload + error, re-queue.
• Throughput chart (jobs/min).

When something's wrong, this is the first place to look.

How you call it from a service

func (s *OrderService) Create(ctx context.Context, in CreateOrderInput) (Order, error) {
  order := models.Order{...}
  if err := s.db.WithContext(ctx).Create(&order).Error; err != nil { return Order{}, err }

  // 3 jobs, fire-and-forget — never block the create.
  // The IdempotencyKey makes a same-order retry safe — the receipt won't
  // fire twice even if the caller retries, even if a load-balancer
  // double-delivers, even if the job worker crashed mid-process.
  _ = s.jobs.EnqueueSendEmail(ctx, in.Email, "Receipt", "receipt",
      map[string]any{"order_id": order.ID},
      jobs.EnqueueOption{IdempotencyKey: "receipt:" + order.ID},
  )
  _ = s.jobs.Enqueue(ctx, "update_inventory",
      map[string]any{"order_id": order.ID},
      jobs.EnqueueOption{IdempotencyKey: "inventory:" + order.ID},
  )
  _ = s.jobs.Enqueue(ctx, "ask_for_review",
      map[string]any{"order_id": order.ID},
      jobs.EnqueueOption{
          Delay:          7 * 24 * time.Hour,
          IdempotencyKey: "review_request:" + order.ID,
      },
  )

  return order, nil
}

Receipt: now. Inventory update: now. Review request: in 7 days. Customer's checkout response still returns in 50ms — and none of these jobs fire twice for the same order.

Idempotency keys — the safety net for retries

Retries cause duplicates. The most common production incident: a payment job fires, the worker crashes mid-flight, the next worker picks it up and charges the customer twice. Idempotency keys fix this at the queue level.

Pass an IdempotencyKey in EnqueueOption and the queue refuses to enqueue a second task with the same key during the window (default 24h). The first call wins; the second silently returns success (the operation is already on its way, which is what the caller wanted).

// First call enqueues
err := s.jobs.Enqueue(ctx, "charge_card", payload, jobs.EnqueueOption{
    IdempotencyKey: "charge:" + invoice.ID,
})
// err == nil

// A retry within the window — dedup'd
err = s.jobs.Enqueue(ctx, "charge_card", payload, jobs.EnqueueOption{
    IdempotencyKey: "charge:" + invoice.ID,
})
// err == nil — the EnqueueSendEmail/EnqueueProcessImage helpers swallow
// the ErrDuplicateTask sentinel so you don't have to check for it on
// fire-and-forget paths. Use the generic Enqueue if you want to detect it.

Picking a good key

The key should be a natural business identifier for the action — not the payload, not a timestamp, not random bytes:

• "receipt:" + order.ID — one receipt per order.
• "welcome:" + user.ID — one welcome email per user.
• "charge:" + invoice.ID — one charge per invoice.
• "digest:" + user.ID + ":" + today — one digest per user per day.

Retry backoff — what actually happens on failure

Return an error from a job handler and asynq re-runs it with exponential backoff. Grit's explicit schedule (set via ExponentialBackoff in workers.go):

Attempt 1:  immediate
Attempt 2:  +1s     (after first failure)
Attempt 3:  +2s
Attempt 4:  +4s
Attempt 5:  +8s
Attempt 6:  +16s
Attempt 7:  +32s
Attempt 8:  +64s
Attempt 9:  +128s
Attempt 10: +256s
…           capped at 5 minutes
After DefaultMaxRetries failures (5 by default): dead-letter queue

The total budget at default settings is ~16 minutes — long enough to ride out a typical downstream outage, short enough that the dead queue doesn't fill with hours-old work. Tune MaxRetries per-task when the failure mode warrants it (e.g., a webhook delivery to a flaky third party might want MaxRetries: 25).

How to modify this battery

• Add a new task — register the handler in handlers.go, enqueue it from a service. That's it.
• Add a queue priority — edit the Queues map in server.go. Higher weight = more worker attention.
• Increase concurrency — bump Concurrency: 10 to 50. Watch DB connection pool — if you exhaust it, raise the pool too.
• Change retry policy — pass asynq.MaxRetry(3) at enqueue, or set a default in the server config.

What's next

Last battery — AI (Claude + OpenAI). Streaming chat, embeddings, and how Grit lets you swap providers without touching feature code.