For senior developers

Caching in Go: cache-aside, jittered TTLs and stampede protection

A cache is a correctness problem disguised as a performance improvement. The naive version works perfectly in development and has two failure modes that only appear under load, both of which make the system slower than having no cache at all.

This covers cache-aside in Go, the stampede that follows an expiry, why a fixed TTL synchronises your misses, and the invalidation strategies in order of how much trouble they cause.

17 min read. Commands and output last run against Grit v3.560.0.

What you will have at the end

A Remember helper with single-flight protection and jittered expiry.

An answer to invalidation that does not rely on remembering every call site.

A clear idea of what should not be cached, which is most things.

Measure before you cache

Caching is the second answer. The first is an index, and it is right far more often than its reputation suggests: a query doing a sequential scan on a million rows is a missing index, not a caching opportunity, and caching it hides the problem while adding staleness.

  • Is it slow because of an N+1? Fix the query. A cached N+1 is five hundred cache lookups.
  • Is it slow because of a missing index? Add the index. It is exact, it has no staleness, and it helps every query against that column.
  • Is it genuinely expensive and read far more than it is written? Now cache it.

Cache-aside, and why the simple version is wrong

// The version in every tutorial.
func (s *Service) Product(ctx context.Context, id uint) (*Product, error) {
if hit, err := s.cache.Get(ctx, key(id)); err == nil {
return decode(hit)
}
p, err := s.repo.ByID(ctx, id)
if err != nil {
return nil, err
}
s.cache.Set(ctx, key(id), encode(p), 10*time.Minute)
return p, nil
}

Two problems, neither visible until there is traffic.

The stampede

The key expires. Five hundred requests arrive in the same second, all miss, and all run the expensive query. The database, which was comfortable because the cache was absorbing the reads, now receives five hundred of them at once. The slower it gets, the longer the window during which more requests miss.

This is why a cache can make an outage worse rather than better: the system is sized for the cached load and the cache is exactly what disappears under pressure.

Synchronised expiry

Warm a thousand keys at deploy time with a fixed ten-minute TTL and all thousand expire in the same second, ten minutes later. The cache turns a steady load into a sawtooth, with a spike at a predictable interval.

One helper that gets both right

internal/cache/remember.go
package cache
import (
"context"
"math/rand"
"time"
"golang.org/x/sync/singleflight"
)
type Cache struct {
rdb redis.UniversalClient
group singleflight.Group
}
// Remember returns the cached value, or computes it once and caches it.
//
// "Once" is the important word: singleflight collapses concurrent calls
// for the same key in this process into one execution of fn, and gives
// every caller the same result. Five hundred simultaneous misses become
// one query.
func Remember[T any](
ctx context.Context, c *Cache, key string, ttl time.Duration,
fn func(context.Context) (T, error),
) (T, error) {
var zero T
if raw, err := c.rdb.Get(ctx, key).Bytes(); err == nil {
var out T
if json.Unmarshal(raw, &out) == nil {
return out, nil
}
// A value that will not decode is a value from an older shape.
// Drop it rather than failing the request forever.
c.rdb.Del(ctx, key)
}
v, err, _ := c.group.Do(key, func() (any, error) {
val, err := fn(ctx)
if err != nil {
return nil, err
}
if raw, err := json.Marshal(val); err == nil {
// Jitter: up to 20% on top of the TTL, so keys written
// together do not expire together.
c.rdb.Set(ctx, key, raw, withJitter(ttl))
}
return val, nil
})
if err != nil {
return zero, err
}
return v.(T), nil
}
func withJitter(d time.Duration) time.Duration {
return d + time.Duration(rand.Int63n(int64(d/5)))
}

singleflight is per process

It collapses concurrent calls inside one instance. Ten instances still produce ten queries rather than five hundred, which is usually fine. When it is not, take a short distributed lock in Redis around the recompute, and have the losers wait briefly and re-read rather than computing. Do not reach for that first: it adds a failure mode, the lock holder dying, that singleflight does not have.

A cache that is down must not take the service with it

// Every cache call has a short timeout of its own, separate from the
// request's. A Redis that is slow should cost two milliseconds and a
// cache miss, not the whole request.
cctx, cancel := context.WithTimeout(ctx, 50*time.Millisecond)
defer cancel()
raw, err := c.rdb.Get(cctx, key).Bytes()
if err != nil && !errors.Is(err, redis.Nil) {
// Log it as a miss, not as a failure. The source of truth is the
// database, and it is still there.
metrics.CacheError.Inc()
}

The rule: a cache error is a cache miss. Code that returns a 500 because Redis is unavailable has made an optional dependency into a required one, which is the opposite of what a cache is for.

Invalidation, in order of how much trouble it causes

1. A short TTL and nothing else

Boring and correct. If sixty seconds of staleness is acceptable, this is the whole design and there is no invalidation code to get wrong. Start here and only move when somebody states a requirement that rules it out.

2. Delete on write

func (s *ProductService) Update(ctx context.Context, p *Product) error {
if err := s.repo.Save(ctx, p); err != nil {
return fmt.Errorf("save product: %w", err)
}
// Delete, do not overwrite. Overwriting with the value you just
// wrote races with a concurrent update and can leave the older of
// the two in the cache.
return s.cache.Del(ctx, productKey(p.ID), productListKey(p.CategoryID))
}

The difficulty is remembering every key a write affects, and lists are where it breaks: a product appears in its own key, in a category listing, in a search result and in a dashboard count.

3. Version the key instead of deleting it

// Bump a counter and every key derived from it is unreachable at once.
// No key enumeration, no scanning, and the old values expire on their
// own TTL.
func (c *Cache) listKey(ctx context.Context, categoryID uint) string {
v, _ := c.rdb.Get(ctx, fmt.Sprintf("ver:category:%d", categoryID)).Int64()
return fmt.Sprintf("products:cat:%d:v%d", categoryID, v)
}
func (c *Cache) BumpCategory(ctx context.Context, categoryID uint) error {
return c.rdb.Incr(ctx, fmt.Sprintf("ver:category:%d", categoryID)).Err()
}

This is the technique worth knowing. It turns "delete every key that might contain this" into one increment, and it is safe under concurrency because the version is part of the key rather than something checked separately.

Never KEYS, and be careful with SCAN

KEYS pattern blocks Redis while it walks the entire keyspace. On a production instance that is a stall measured in seconds, during which every other command waits. SCAN is incremental and safe, and is still a poor invalidation strategy: key versioning avoids needing either.

4. Pub/sub, for local caches

Only when you have an in-process cache in front of Redis. One instance writes, all instances hear about it and drop their local copy. It is a real technique with a real cost: a dropped message means one instance is stale indefinitely, so the local copy needs a short TTL anyway, which raises the question of whether it was worth it.

Keys, and the mistakes in them

// Namespaced, versioned, and the scope is IN the key.
"app:v2:product:8821"
"app:v2:org:412:invoices:page:1:status:draft"
"app:v2:user:88:permissions"
  • Put the tenant in the key. A cached list keyed by page number and not by organisation serves one customer's data to another, and it is a particularly bad bug because it is intermittent and looks like a database problem.
  • Put the version in the prefix. When the cached shape changes, bumping v2 to v3 invalidates everything instantly, which is otherwise a deploy that reads old structures into new types.
  • Include every input that changes the answer: filters, sort, page size, and the viewer's permissions if those affect what is returned.
  • Never cache a response that varies by user under a key that does not. This is the same mistake as the tenant one, and it is how a CDN serves one person's dashboard to everybody.

What to cache, and what not to

Good candidates

  • Expensive aggregates: counts, reports, dashboard figures.
  • Reference data that changes rarely: categories, settings, feature flags, plans.
  • Responses from third parties, which are slow and rate limited.
  • Permission sets, which are read on every request and written rarely.

Poor candidates

  • Anything a user has just changed. Seeing a stale version of your own edit is the worst kind of staleness, because it reads as a bug rather than as a delay.
  • Data that is already fast. A primary key lookup on a warm table is under a millisecond. Caching it adds a network hop and a staleness window to save nothing.
  • Anything where stale means wrong: stock levels at checkout, account balances, permission denials.
  • Large objects read occasionally. They evict the small, hot entries that were doing the real work.

Operating it

  • Measure the hit rate per key prefix, not overall. An overall rate of 85% can hide a prefix at 5% that is pure overhead.
  • Set maxmemory and an eviction policy. allkeys-lru for a pure cache. Without a policy, Redis eventually refuses writes instead of evicting, which is a confusing outage.
  • Know whether you are also using Redis as a queue. If so, do not use allkeys-lru, because it will evict your jobs. Separate the instances, or use volatile-lru and set TTLs only on cache keys.
  • Watch the key count and the memory, not just the hit rate. A growing key count with a steady hit rate usually means keys that include something unbounded, a timestamp or a search string, which is a slow memory leak.

Generated

// Remember with single-flight and jitter, ready to use.
products, err := cache.Remember(ctx, c, key, 10*time.Minute,
func(ctx context.Context) ([]Product, error) {
return repo.ListFeatured(ctx)
})
Terminal
# Nothing to install: the Redis client, Remember, the response
# middleware and the key helpers are scaffolded into every project.
$grit start

There is also response caching as middleware for public read endpoints, keyed on the full URL. One caution that comes from this project's own history: that cache is keyed by URL and not by viewer, so an endpoint that varies by who is asking must not be on it. A writing or personalised endpoint on a URL-keyed cache group is a data leak wearing a performance improvement.

The caching reference covers the helper, the middleware and the invalidation helpers.

Frequently asked questions

In-memory or Redis?
In-memory is faster and is per instance, so each one has a different view and invalidation means telling all of them. Redis is shared, which makes invalidation a single delete, at the cost of a network hop. Start with Redis: a consistent cache is easier to reason about than a fast one, and the hop is well under a millisecond.
How long should a TTL be?
As long as the staleness you can defend to the person who will notice. Write it down as a sentence: "the dashboard may be up to five minutes behind". If that sentence is acceptable, the TTL is five minutes. If no sentence is acceptable, the thing should not be cached. Add jitter regardless.
Is singleflight enough, or do I need a distributed lock?
Enough for almost everything. With ten instances, a popular key expiring produces ten queries rather than five hundred, and ten is not a stampede. A distributed lock is worth it when the recompute is genuinely heavy, tens of seconds, and it brings its own failure mode in the lock holder dying mid-compute.
Why is my cache making things slower?
Three usual causes. A hit rate low enough that most requests pay the lookup and then query anyway. Values large enough that serialising and transferring them costs more than the query did. Or a cache call with no timeout of its own, so a struggling Redis adds its latency to every request rather than being skipped.

Next steps