Caching System Design

Most read traffic asks the same questions. A dashboard count, a settings lookup, a permission set, a list that changes hourly and is read thousands of times an hour. Recomputing each of them from the database is work the database did not need to do.

A cache is easy to add and easy to get wrong in four specific ways. It can serve a stale answer after a write. It can become a dependency, so that losing it takes the application down rather than slowing it. It can stampede, where a hot key expires and every concurrent request rebuilds it at once. And a deploy can write ten thousand keys in the same second, which then all expire in the same second and hand the database the whole working set back at once.

Each of those has a known remedy, and the value of a cache layer in a framework is that the remedies are already in it rather than being rediscovered per project.

The system has to be able to:

• Read through: ask for a value, get it from the cache or compute and store it, in one call.
• Fall through to the loader when the cache is unavailable, so an outage is a slowdown.
• Spread expiry times so a synchronised write does not become a synchronised miss.
• Let one caller rebuild a hot key while the others wait briefly rather than all rebuilding.
• Invalidate by key and by prefix when the underlying data changes.
• Run with no cache at all, because a small deployment should not need Redis.

Functional requirements

• A Remember call taking a key, a time to live, a destination and a loader.
• Get, Set, Delete and DeleteByPrefix for the cases that are not read-through.
• A response cache for whole GET responses on routes that opt in.
• Automatic invalidation on write for generated resources, by prefix.
• Jitter of up to ten per cent added to every time to live.
• Single-flight rebuilding of an expired key.
• Complete absence as a supported configuration: no Redis URL means no cache and no errors.

Non-functional requirements

• Never load-bearing. every read path works with the cache removed. That is what makes it acceptable for a cache to be a network dependency at all.
• Correct before fast. a write invalidates before it returns. Serving a value the caller just changed is worse than not caching.
• Keyed completely. a key includes everything that varies the answer, including the owner and the tenant. A key that omits them is a data leak, not a performance bug.
• Bounded. everything has a time to live. A cache with no expiry is a second database with no migrations.
• Observable. hit rate and eviction are visible. A cache nobody measures is a cache nobody can tune.

Numbers for a mid-sized deployment. They are here to size the thing, not to predict your traffic: change an assumption and the sums below move with it.

Assumptions

One read-through call covers most uses. The response cache and the invalidation hooks are layered on top of the same store.

Core components

• Cache client. the store interface. Backed by Redis when configured and by nothing when not, so the absence case needs no branches at the call sites.
• Remember. the read-through call. Cache-aside with the three things a hand-written version usually misses: fall-through on failure, jittered expiry and single-flight rebuilds.
• Response cache. middleware that stores a whole GET response for routes that opt in, keyed by path, query and identity.
• Invalidation hooks. generated resources delete their prefix on create, update and delete, so a write cannot leave a stale list behind.
• Key builder. the convention that makes keys include the owner and the tenant, so no key can be shared across a boundary.

Request flow

1. 1A read request arrives on a route whose answer is worth caching.
2. 2The handler calls Remember with a key, a lifetime, a destination and a function that can compute the value.
3. 3The cache is asked. A hit returns immediately. A cache that is unreachable is treated as a miss rather than an error, which is what keeps it from being load-bearing.
4. 4On a miss, single-flight ensures one caller runs the loader while the others wait for its result instead of all running the same query.
5. 5The loader computes the value, usually a query the handler would have run anyway.
6. 6The result is stored with the requested lifetime plus up to ten per cent of jitter, so keys written together do not expire together.
7. 7Separately, a write to the underlying resource deletes the cache prefix for it.
8. 8Invalidation happens before the write returns, so a client that reads immediately after writing cannot see the value it just replaced.

Data flow

• Keys carry everything that varies the answer. For an owned resource that includes the owner, and in a multitenant deployment the organisation.
• Nothing is cached without a lifetime. An entry that never expires is state that no migration will ever fix.
• Invalidation is by prefix rather than by enumerating keys, because the set of list keys for a resource is unbounded in query parameters.
• A cache miss and a cache outage take the same path, so there is no error handling at the call site to get wrong.

There is no schema. The design question is the key, and the convention is what keeps it correct.

Core types

Design principles applied

• Make the correct call the shortest one. Remember is one line and handles the four classic mistakes. Get and Set are available and longer to write, which is the right incentive.
• A dependency that may be absent. the client treats unreachable as empty. There is no error to handle at the call site and therefore no error handling to get wrong.
• Invalidate at the write, not on a timer. relying on a short lifetime to hide staleness means choosing between stale data and no caching. Invalidating removes the choice.

Patterns

• A cache hit removes a database round trip, which is the cheapest capacity available until the hit rate stops improving.
• Hit rate times cacheable share is the real number. A 99% hit rate on 5% of traffic is worth less than 80% on 60%.
• Memory grows with the working set, not with total data. Sizing follows from distinct hot keys, not from table sizes.
• Redis is single-threaded per instance. Very hot single keys are the limit, which single-flight helps with on the application side.
• Prefix invalidation is a scan on some backends. Keeping prefixes narrow keeps that cost bounded.
• With no cache configured the application is slower and entirely correct, which is the right default for a small deployment.

What breaks first

• Keys that omit the owner. the worst cache bug there is, because it does not look like a bug. It looks like one user occasionally seeing another user’s data.
• The cache becoming load-bearing. a path that errors when the cache is unreachable has turned an optional component into a required one, usually without anybody deciding to.
• Invalidation gaps. a write path that does not go through the service does not invalidate, and the stale value lives for a full lifetime.
• Unbounded key growth. list keys hashed from arbitrary query parameters can be generated without limit by a caller who varies them.

What to do about it

• Build keys, do not write them. a helper that takes the context and appends the owner and tenant makes the complete key the easy one.
• Test with the cache off. running the suite with no cache configured proves no path depends on it, which is the property that is otherwise assumed.
• Invalidate in the service. putting the hook where every writer passes, rather than in the handler, closes the gap for jobs and commands.
• Whitelist the query parameters. hashing only the parameters that actually vary the answer bounds the key space regardless of what a caller sends.