Background jobs, cron and transactional email

Some work must not happen inside a request. Resizing twelve images, emailing four hundred people, reconciling yesterday's payments: if the user is waiting for it, you have built a timeout.

This chapter is the queue, the scheduler and the mailer, and most of it is about the two questions that separate a job system from a go func(): what happens when it fails, and what happens when it runs twice.

Why not just a goroutine

go sendTheEmails(users)   // please do not

Three things are wrong with it, and the third is the one that hurts.

It dies with the process. A deploy, a crash, a scale-down, and the work is gone with no record that it existed.

It cannot be retried. The error goes to a log, if anywhere.

It cannot be seen. Nobody can answer "did the invoice reminders run last night", which is the question you will actually be asked.

A job, generated

grit generate job SendInvoiceReminders --cron "0 9 * * *"

That writes three things: the job file, a handler registration in the worker's mux, and, with --cron, a schedule entry that the admin's cron page lists.

The job file is yours from the moment it is written, and a second generaterefuses rather than overwrites.

const TypeSendInvoiceReminders = "send_invoice_reminders"

type SendInvoiceRemindersPayload struct {
    Day string `json:"day,omitempty"`
}

func handleSendInvoiceReminders(deps WorkerDeps) func(
    context.Context, *asynq.Task) error {

    return func(ctx context.Context, task *asynq.Task) error {
        var p SendInvoiceRemindersPayload
        if len(task.Payload()) > 0 {
            if err := json.Unmarshal(task.Payload(), &p); err != nil {
                return fmt.Errorf("decoding: %v: %w",
                    err, asynq.SkipRetry)
            }
        }
        // deps.DB, deps.Mailer, deps.Storage, deps.Cache
        return nil
    }
}

Two conventions in that template are the whole discipline.

The payload is IDs and small values. It is serialised into Redis, and a retry sees exactly what the first attempt saw. Putting a whole struct in the payload means a retry works from a snapshot that may now be stale.

*An undecodable payload is wrapped with asynq.SkipRetry.* A payload that does not parse now will not parse in four minutes either. Retrying a permanent failure is how a queue fills up with work that can never succeed.

Return an error to retry, return nil to stop

return nil   // done, or can never succeed
return err   // retry with backoff

That is the entire protocol, and the second half of the first line is the part people get wrong. A task that fails for a reason that will never change, a deleted record, a malformed input, a file that is not an image, should returnnil and log loudly, or wrap with SkipRetry. Returning the error buys you the same failure five more times with exponential backoff.

The two kinds of "it ran twice"

A queue guarantees at least once, not exactly once. The retry that happens after a worker was killed mid-task is the normal case, not the pathological one.

So every job has to answer: what if this runs again?

Idempotency at the enqueue. The same work queued twice is refused:

queue.EnqueueSendInvoiceReminders(ctx, payload,
    jobs.EnqueueOption{
        IdempotencyKey: "reminders:" + day,
        Window:         24 * time.Hour,
    })

A second enqueue inside the window returns ErrDuplicateTask rather than running. This is how a nightly job that is triggered twice, by a scheduler and by an impatient human, stays a single run.

Idempotency in the work. Even one enqueue can execute twice. So the work itself should be safe to repeat: mark the invoice as reminded in the same transaction that records the send, and skip the ones already marked.

Cron, and the five fields

grit generate job NightlyReconcile --cron "30 23 * * *"
grit generate job Heartbeat --cron "@every 15m"
grit generate job Cleanup --cron "@daily"

The spec is validated when you generate, not at three in the morning:

--cron wants five fields (minute hour day-of-month month
day-of-week), e.g. "30 23 * * *" for 23:30 every day

Scheduled runs arrive with an empty payload, which is why the template handles that case. The schedule is registered with a retry limit, and every registered task appears on the admin's cron page with its name, its spec and its type, so "did it run last night" is a screen rather than a grep.

Mail, and the drivers

MAIL_MAILER = smtp | resend | mailgun | postmark |
              sendgrid | ses | log | failover

Three behaviours in the configuration are worth copying into whatever you build next.

An unset driver does something sensible. In development the mail goes to Mailhog from Docker Compose, falling back to the log when Mailhog is not running, so a password reset link is never lost.

A named but misconfigured driver is an error, not a fallback. An application that believes it sends through SES must not quietly write to a log instead.

Production with no mailer configured returns nothing, and the code that sends says so, rather than pretending.

Queued or inline, decided once

Transactional mail goes through one function, and the decision is made there:

func dispatchMail(ctx context.Context, mailer *mail.Mailer,
    queue *jobs.Client, key string,
    opts mail.SendOptions) error {

    if queue != nil {
        return queue.EnqueueSendEmail(ctx, opts.To,
            opts.Subject, opts.Template, opts.Data,
            jobs.EnqueueOption{IdempotencyKey: key})
    }
    if mailer == nil {
        return ErrNoMailer
    }
    sendCtx, cancel := context.WithTimeout(
        context.WithoutCancel(ctx), 15*time.Second)
    defer cancel()
    return mailer.Send(sendCtx, opts)
}

With Redis, the mail is queued and retried. Without it, the mail is sent inline with a timeout. The caller does not care, which is why a project can start without Redis and gain it later without touching the sending code.

context.WithoutCancel is the detail worth stealing. The send is allowed to finish after the request it came from has been answered, but not to run forever: the request's context is cancelled the moment the response is written, and inheriting that cancellation would kill the send mid-flight.

Seeing it work

Three places, and knowing they exist saves an afternoon:

• Admin: Jobs
• Queued, active, retrying and dead tasks, with payloads
• Admin: Cron
• Every registered schedule and when it last ran
• Admin: Mail
• A preview of each template, rendered, without sending

The mail preview is the one that changes how you work. Email templates are miserable to iterate on through actual sends, and a page that renders them with sample data turns a twenty minute loop into a reload.

grit generate job SendInvoiceReminders --cron "0 9 * * *"

Assignment

Commit your work with the message Jobs and Mail - Assignment One, exactly as written, so the leaderboard can score it.