Delivery

Transactional Email System Design

Getting a password reset into an inbox, through whichever provider is configured, without the request waiting for it.

internal/mailinternal/handlersinternal/jobs

1.Problem statement

Email is where authentication actually lives. A password reset that does not arrive is an account that cannot be recovered. A verification message that lands in spam is a sign-up that does not complete. The feature is not "send mail", it is "the user gets the message", and the gap between those two is most of the work.

The naive version has four faults, and this project shipped all of them. Mail left every handler from a bare goroutine, so nothing bounded the concurrency, nothing retried a provider blip, a deploy dropped whatever was in flight, and no failure surfaced because the handler had already returned 200. Meanwhile the job client that handles all four properly was never called from anywhere.

There is also a provider problem. Hardcoding one provider means local development needs real credentials and real messages, tests send real mail or are not written, and changing provider is a change to every call site. The provider is a configuration detail and should be one in the code too.

And there is a content problem: an HTML email is written against a rendering engine from 1997 that lives inside several different mail clients, none of which agree.

The system has to be able to:

Send from one place in the application, not from each handler.
Queue the send, so the request does not wait and a failure can be retried.
Swap the provider by configuration, including a log driver that sends nothing.
Fall back to a second transport when the first fails.
Render templates with a layout, so every message looks like the application.
Preview a rendered template without sending it.
Test without a network, through a fake transport that records what was sent.
Show what was queued, what was sent and what failed.

2.System requirements

Functional requirements

A mailer with a transport chosen once from configuration.
Transports for SMTP, Resend, SES and a log driver that prints instead of sending.
A failover transport, so a provider outage falls through to another.
HTML templates with a shared layout and a plain text alternative.
All sending routed through one dispatch function in the handler layer.
Sending done in a background job, with the queue’s retries and dead letter.
A preview page rendering each template with sample data.
A fake transport for tests that records messages rather than sending them.

Non-functional requirements

Never in the request path. a sign-up does not wait for a mail provider. The enqueue is milliseconds; the send is somebody else’s latency.
Retried, not hoped for. a provider down for a minute must not lose a password reset. That is what the queue is for, and it is what a goroutine cannot do.
One way out. twelve handlers each sending mail is twelve places to get bounding, retry and shutdown wrong. One dispatch function is one place to get them right.
Provider agnostic. the call site does not change when the provider does. The transport is chosen once, from configuration, at startup.
Sends nothing by default in development. the log driver means a fresh clone can register a user and read the verification link in the terminal, with no account anywhere.
Visible. a failed send is in the jobs dashboard with its error. The previous failure mode was total silence.

3.Capacity estimation

Numbers for a mid-sized deployment. They are here to size the thing, not to predict your traffic: change an assumption and the sums below move with it.

Assumptions

Parameter	Value
Emails per day	~40,000
Peak rate	~25 per second during a campaign or an incident
Average message size	~45 KB with HTML and inline styles
Provider latency	150 to 800 ms per send
Provider rate limit	typically 10 to 100 per second

Worker capacity

0.4 s average provider latency

10 concurrent slots = 25 sends/s per worker process

two worker processes cover the peak with headroom

Mail is almost entirely waiting on a network call, so concurrency rather than CPU is what sets throughput, and the provider rate limit caps it before the worker does.

Inline, for comparison

0.4 s added to every request that sends mail

a sign-up going from 80 ms to 480 ms

and a provider timeout turning it into 30 s

The 30 second case is the one that matters. An inline send means a provider incident is an application incident.

Queue during an outage

25/s x 600 s of provider downtime = 15,000 queued

at ~46 KB each = ~690 MB in Redis

Which is the argument for payloads carrying a template name and data rather than rendered HTML. The same backlog of references is a few megabytes.

Rate limit interaction

provider allows 20/s, workers can do 50/s

without a limiter: 429s, which consume retry attempts

with one: the queue drains at 20/s and nothing fails

4.High level design

One dispatch function, one queued job, one mailer, and a transport chosen from configuration.

Core components

Mail dispatch. the single place mail leaves a handler. Added after the review found every email going out from a bare goroutine.
Send job. the queued task. Inherits the queue’s retries, timeout, dead letter and dashboard, which is four properties for free.
Mailer. renders and sends. Holds a transport and a default from address, and reports which driver it is using.
Transports. SMTP, Resend, SES, log, and a failover wrapper that tries one and then another. Chosen once at startup from MAIL_MAILER.
Templates. HTML with a shared layout and a text alternative, rendered with the data the job carried.
Preview. an admin page rendering each template with sample data, because the only other way to check a change is to send yourself mail.
Fake transport. records messages instead of sending them, so a test can assert that a reset mail was sent to the right address.

Request flow

A password reset, from request to inbox

1A user asks for a password reset. A token is created and stored, which is the part that must happen before the response.
2The handler calls the one dispatch function. It does not start a goroutine, call a provider or render anything, because each of those was a separate bug when handlers did them.
3A send job is queued with the template name, the recipient and the data. The template name rather than rendered HTML, so the payload is small and a template fix applies to work already queued.
4The response goes back immediately. The user is told to check their email while nothing has yet been sent, which is correct: the alternative is making them wait for a third party.
5A worker picks the job up, with five retry attempts and a timeout. A provider down for a minute costs a few retries rather than a lost reset.
6The template is rendered with its layout, producing HTML and a plain text alternative. One layout, so every message looks like the application rather than like whoever wrote that handler.
7The configured transport sends it: SMTP, Resend, SES, or the log driver that prints it to the terminal, which is what a fresh clone uses so that nothing needs credentials to work. A failover transport tries a second provider when the first errors.
8It arrives, or the job retries, or after the last attempt it sits in the dead queue with its error where somebody can see it. Visible failure is the entire improvement over the goroutine version, where nobody ever found out.

Data flow

The job payload carries a template name and data, not rendered HTML. A queue full of rendered messages is large, and a template fix cannot reach them.
Nothing in a payload is assumed still to exist when the job runs. The user may have been deleted between the enqueue and the send.
Transports are selected once at startup. Branching per send would mean the behaviour could differ between two calls in the same process.
The log driver is a real transport rather than a flag, so the path taken in development is the same path taken in production with a different endpoint.

5.Technology stack

Component	What it is
Transports	SMTP, Resend, SES, log, and failover between them
Selection	MAIL_MAILER, read once at startup
Delivery	a queued job, 5 retries, 5 minute timeout
Templates	Go html/template with a shared layout and a text alternative
Development	log driver by default, nothing sent, nothing configured
Testing	a fake transport that records messages
Preview	an admin page per template with sample data

6.API design

Mail administration

Method	Endpoint	What it does
GET	/api/v1/admin/mail/templates	The templates and their sample data
GET	/api/v1/admin/mail/preview/:template	Rendered HTML, without sending
GET	/api/v1/admin/mail/queue	Queued, sent and failed sends

The only way mail leaves a handler

// Not a goroutine, not a provider call: one dispatch function that
// queues the send, so a provider blip is a retry and not a lost reset.
if err := handlers.DispatchMail(c, mail.Message{
    To:       user.Email,
    Template: "password-reset",
    Data:     map[string]any{"name": user.Name, "link": resetURL},
}); err != nil {
    return fmt.Errorf("queueing reset mail: %w", err)
}

7.Low level design

Core types

mail.Mailerinternal/mail/mail.go

Sends through a Transport. The transport is picked once from configuration, so the code that sends mail does not change when the provider does.

SendDriverFrom

mail.Transport

The interface. SMTP, Resend, SES, log, and a failover wrapper named failover(smtp,log) so the driver string says what is actually happening.

mail.FromConfig

The constructor that honours MAIL_MAILER. The older New is kept sending through Resend so existing code compiles unchanged.

handlers.DispatchMailinternal/handlers/mail_dispatch.go

The one place mail leaves a handler. Its own file and its own writer so an upgrade can deliver it whole, because the repaired handlers do not compile without it.

mailtest

The fake transport. Records messages so a test can assert the recipient and the template without a network or a provider account.

Design principles applied

One exit point. the fix for mail being sent from twelve handlers was not twelve careful handlers, it was one function they all call.
Queue, do not go. a goroutine gives you concurrency and nothing else. A queue gives you bounding, retries, survival across deploys and a dashboard.
The provider is configuration. including the one that sends nothing. A development setup requiring a mail account is a development setup people work around.
Reference, do not render, into the queue. a template name and data keeps payloads small and lets a template fix reach mail that is already queued.

Patterns

Pattern	Where it is used
Strategy	the transport interface with a provider per implementation
Failover	a transport that wraps two others
Null object	the log driver as a real transport
Facade	one dispatch function in front of the mailer and the queue
Test double	a recording transport instead of a network

8.Scalability and performance

Mail is I/O bound, so throughput follows worker concurrency until the provider rate limit caps it, and the provider limit is almost always the real ceiling.
A rate limiter in front of the transport is worth more than more workers, because a 429 consumes a retry attempt and a limiter does not.
Payload size decides what an outage costs. Template names and data make a ten minute backlog a few megabytes; rendered HTML makes it most of a gigabyte.
A dedicated mail queue keeps a campaign from delaying a password reset, which is the difference between slow marketing and locked-out users.
Bulk sending is a different system. Transactional mail is one message per event, and running a campaign through it will meet the provider limit first and the reputation limit second.
Deliverability does not scale with any of this. SPF, DKIM and DMARC on the sending domain decide whether the message arrives, and no amount of throughput substitutes.

9.Bottlenecks and improvements

What breaks first

Provider rate limits. exceeding one produces 429s that look like failures and burn retry attempts, so the queue drains slower the harder it tries.
Deliverability. mail that is sent and not delivered is the failure nobody sees in any dashboard, because the send succeeded.
Bounces and complaints. repeatedly sending to a dead address damages the sending reputation of the domain, which affects every other message.
HTML rendering. mail clients disagree about almost everything, so a template that looks right in one is broken in another and the preview only shows one.
Templates as a single point of failure. a template that fails to render fails every message using it, and it fails in a worker rather than in a test.

What to do about it

Rate limit before the transport. match the provider’s allowance so the queue paces itself instead of being throttled, and retries stay available for real failures.
Authenticate the domain. SPF, DKIM and DMARC, and a subdomain for transactional mail so marketing cannot damage its reputation.
Process bounce webhooks. suppress hard-bounced addresses rather than sending again. This is an inbound webhook, so it already has a verified path to arrive on.
Always send a text alternative. it renders everywhere, it improves spam scoring, and it is the version that works when the HTML does not.
Render every template in a test. a test that renders each template with its sample data catches the break at build time rather than in a worker at midnight.