Picture Omkar at ten past ten on a sale Saturday, watching the homepage p99 jump from 180 milliseconds to four seconds, drop back, then do it again ten minutes later. There is no cron behind it. There is one hot Redis key with a 600-second TTL.
That is a Redis cache stampede: a hot key expires under load, and every concurrent miss runs the same expensive rebuild at once. A hundred misses means a hundred copies of the same query.
This post walks through the three fixes in plain Redis and Node:
- a lock
- early expiration
- stale-while-revalidate
My pick is stale-while-revalidate, and I'll show why.
- Normal180 ms
- Key expires under load4,000 ms
How one expired key becomes a hundred queries
Traffic reads the key a few hundred times a second. At second 600 the TTL hits zero and Redis deletes it.
The next request misses and starts the rebuild, a query with two joins and an ORDER BY that takes 300 milliseconds. During those 300 milliseconds the key is still missing, so the next hundred requests also miss, and also start the rebuild.
A hundred copies of the same query land on the database within a third of a second. They all return the same rows and write the same value. Ninety-nine were waste, and every other query queued behind them paid too.
The painful part is when it happens. At 3am the same expiry is one miss and nobody notices. At peak it is a crowd.
The problem scales with concurrency, so it is invisible in staging and arrives exactly when the database has the least to spare. I covered the Laravel version in Laravel caching with Redis; here are the three approaches underneath.
Fix 1: lock and let one request rebuild
The obvious fix, and the one Omkar ships from his phone at twenty past ten. The first request to miss takes a lock and rebuilds. Everyone else waits.
const LOCK_TTL_MS = 10_000;
async function getPopular(): Promise<Listing[]> {
const cached = await redis.get("listings:popular");
if (cached) return JSON.parse(cached);
const token = randomUUID();
const locked = await redis.set("listings:popular:lock", token, "PX", LOCK_TTL_MS, "NX");
if (locked === "OK") {
try {
const value = await computePopular(); // the 300 ms query
await redis.set("listings:popular", JSON.stringify(value), "EX", 600);
return value;
} finally {
await releaseLock("listings:popular:lock", token); // Lua: only if token matches
}
}
for (let i = 0; i < 20; i++) { // someone else holds it
await sleep(50);
const now = await redis.get("listings:popular");
if (now) return JSON.parse(now);
}
return computePopular(); // gave up waiting; compute rather than error
}NX and PX on Redis SET do the job in one round trip: set only if absent, expire in so many milliseconds. Two details carry the rest:
- The lock has a TTL, because a rebuild that dies holding the lock must not block everyone forever. Set it to the worst case of your rebuild, times three.
- The release checks the token, so a slow rebuild that outlives its lock cannot release the next holder's lock. That is a well-known mistake, and I have still made it.
The weakness is the wait. Those polls are up to a second of latency on a hundred requests.
A lock turns a herd into a queue, and the queue still has to stand somewhere.
Fix 2: recompute before the key expires (XFetch)
A different idea: have one request refresh the key slightly early, chosen at random with a probability that rises as expiry gets closer.
This is probabilistic early expiration, from Vattani, Chierichetti and Lowenstein's 2015 VLDB paper, Optimal Probabilistic Cache Stampede Prevention, usually called XFetch:
recompute if: now - delta * beta * ln(random()) >= expirydelta is how long the recompute takes and beta is a knob, 1.0 by default. Far from expiry the condition almost never fires; near it, it almost always does.
interface Wrapped<T> { value: T; delta: number; expiry: number } // ms
async function getPopular(beta = 1.0): Promise<Listing[]> {
const raw = await redis.get("listings:popular");
const now = Date.now();
if (raw) {
const w: Wrapped<Listing[]> = JSON.parse(raw);
if (now - w.delta * beta * Math.log(Math.random()) < w.expiry) return w.value;
}
const start = Date.now();
const value = await computePopular();
const w = { value, delta: Date.now() - start, expiry: now + 600_000 };
await redis.set("listings:popular", JSON.stringify(w), "EX", 660);
return value;
}No lock and no waiting. The herd never forms because the key never expires under load.
A slower recompute pushes delta up, which moves the refresh earlier on its own. The cost is that one request per refresh still pays the full 300 milliseconds inline. Fine for a homepage, not for a checkout.
Fix 3: serve stale and refresh in the background
The third approach gives up on the idea that a reader should ever wait.
- Store a soft expiry inside the value and set the Redis key's hard TTL well past it.
- When a read finds the soft expiry passed, return the stale value at once and refresh in the background behind a lock.
async function getPopular(): Promise<Listing[]> {
const raw = await redis.get("listings:popular");
if (raw) {
const w = JSON.parse(raw);
if (Date.now() > w.softExpiry) void refreshInBackground(); // not awaited
return w.value; // stale is fine
}
return refreshNow(); // cold start, the only time a reader waits
}
async function refreshInBackground() {
const locked = await redis.set("listings:popular:refresh", "1", "PX", 10_000, "NX");
if (locked !== "OK") return;
try { await refreshNow(); } finally { await redis.del("listings:popular:refresh"); }
}
async function refreshNow(): Promise<Listing[]> {
const value = await computePopular();
const w = { value, softExpiry: Date.now() + 600_000 };
await redis.set("listings:popular", JSON.stringify(w), "EX", 3600); // hard TTL: one hour
return value;
}- Plain TTL~100
- Lock (the rest wait)1
- XFetch (one reader waits)1
- Stale-while-revalidate (nobody waits)1
Nobody blocks except on a cold cache. The database sees one query per soft TTL regardless of concurrency.
The result? If the database is down, the stale value keeps serving for up to an hour.
The costs are memory held longer (see what large values do to Redis) and a promise to the product team that data is at most one refresh old.
Which stampede fix to pick
Stale-while-revalidate, every time, for anything that can be stale at all.
- The lock makes readers queue, and queues under load are what you were avoiding.
- XFetch is clever, and I would use it where I cannot wrap the value, but it still charges one reader the full rebuild.
- Stale-while-revalidate charges nobody after the first request and is the easiest of the three to read at 2am.
What would change my mind is a value that must not be stale. Then I would take the lock, with the token check, and accept the queue, or ask why that value is cached at all.
The mistake I made for longer than I like was treating this per key, when a spike showed up. It is a property of the cache wrapper.
Conclusion
Put the soft TTL and the background refresh in the one function every cached read goes through. Then the next hot key is protected before anyone finds it on a graph, and Omkar's next sale morning is boring.
Before that, read the query plan for your slowest rebuild: a 300-millisecond query that becomes 30 shrinks every window in this post by ten.
Which of your cache keys would hurt most if it expired at peak?