Case study · System design
An order system that survives a sale day
Anyone can draw boxes. The design is in what you refuse to put inside them.
The brief
Normal days are easy. The design is for one afternoon a year.
A shop selling into three countries runs one sale a year. On that afternoon it takes roughly forty times its normal traffic for about two hours. The rest of the year the system is close to idle.
That single fact decides almost everything below. A design tuned for the average is a design that falls over on the only day anyone remembers.
| Normal | ~3 orders/minute, ~40 page views/second |
|---|---|
| Sale peak | ~120 orders/minute (2/second), ~1,600 page views/second |
| Read : write | Roughly 800 : 1. Almost nobody buys; almost everybody browses. |
| Order row | ~400 bytes + items. 250k orders/year is well under a gigabyte. |
| Catalogue | ~12,000 products. The entire thing fits in memory. |
| Budget for a page | 300 ms at the 95th percentile, on a phone, on a bad connection. |
What these numbers rule out. Two writes per second is not a distributed-database problem — one Postgres primary does that without noticing. The pressure is entirely on reads, and reads of a catalogue that barely changes. That is a caching problem wearing a scaling problem's coat.
Shape
Draw the line first, then the boxes.
Before anything else: decide what is allowed to make a customer wait. Everything on the synchronous side has a latency budget and a blast radius. Everything on the other side can fail, retry, and try again in ten minutes without anyone noticing.
Why the app tier is stateless
Not for elegance — so that scaling out on sale day is adding machines rather than a migration. Sessions live in Redis, uploads go straight to object storage, and nothing on a box's disk matters. Any app server can be shot at any moment and the only cost is one retried request.
Why one primary, and not something cleverer
Two writes a second. A single Postgres primary handles that with room to spare, and gives transactions, foreign keys and a unique index — three things the write path below depends on completely. Sharding would buy write throughput nobody needs and cost the guarantees everybody does.
The honest limit: this design has one machine whose loss stops writes. With a hot standby and automated failover that is around thirty seconds of degraded checkout. For this shop that is the right trade. For a payments company it would not be.
Data
The schema is the argument.
Most arguments about a system are really arguments about its tables. Get these five right and the code becomes obvious; get them wrong and no amount of clever service boundaries will save it.
Stock as a ledger, not a number
The tempting design is products.stock, decremented on sale.
It is smaller, faster, and it loses. Two checkouts land in the same
millisecond and you have sold a thing twice; a warehouse correction
arrives and there is nowhere to record why the number changed.
inventory_moves stores every change with a reason. Current
stock is a sum, and a sum is cheap when you keep a rolled-up snapshot
beside it. What you buy is the ability to answer “where did those
eleven go?” — which is the question that actually gets asked.
Money as integers
total_minor is a count of pence, not a decimal of pounds.
Floating-point money is a bug with a release date: it works through
testing and shows up as a one-penny discrepancy in a reconciliation
report nine months later.
Writes
Assume every request arrives twice.
A phone on a train sends the checkout request, loses signal before the response, and the customer presses the button again. On a normal day that is rare. On sale day, with everything slower, it is constant.
The whole trick is one unique index
The client generates an idempotency key and sends it with the order. The
insert is ON CONFLICT (idempotency_key) DO NOTHING, followed
by a select. First request: a row is created. Second request: the insert
does nothing and the select returns the same order. The customer gets one
confirmation, and the database — not the application —
enforced it.
Doing this in application code means reading, deciding, then writing, and two workers can do that at the same time. Pushing it into a constraint means the race cannot be lost, because there is no gap to lose it in.
At-least-once, everywhere
Queues that promise exactly-once delivery are either lying or very slow. Assume at-least-once and make the consumer idempotent: every worker writes keyed on the order id, so a duplicate delivery is a no-op instead of a second shipment.
Reads
Cache the boring 99%.
Eight hundred reads for every write, of a catalogue that changes a few times a day. Three layers, each catching what the one before it missed:
| CDN, 5 minutes | Product and category pages, for signed-out visitors. Takes the overwhelming majority of sale-day traffic before it reaches a server at all. |
|---|---|
| Redis, 60 seconds | Rendered fragments and the catalogue itself. Written on publish, not on expiry, so a sale-day stampede cannot all miss at once. |
| Postgres replicas | Everything personal — order history, saved baskets. Always a read, never a write. |
Invalidation is by event, not by timer. A price change
publishes product.updated; a worker rewrites the fragment
and purges the CDN path. Waiting five minutes for a wrong price to
expire is how you sell a television for nine pounds.
The cache is not allowed to be load-bearing
Redis going down should make the site slow, not broken. Every cached read has a path to the database behind it, and that path is exercised deliberately in staging — because a fallback nobody has run is not a fallback, it is a hope.
When it breaks
What fails first, and what that looks like.
A design is not finished when it works. It is finished when you can say what happens when each piece stops working, and you are content with every answer.
| An app server dies | The load balancer stops sending to it within one health check. Nothing is lost; in-flight requests are retried by the client. |
|---|---|
| Redis dies | Every read falls through to Postgres. Pages get slower and the database gets busy. The site stays up. This is the failure that is designed for. |
| A replica dies | Reads move to the remaining one. Capacity halves, which on a normal day is invisible and on sale day is not. |
| The primary dies | Writes stop. Checkout returns a clear error rather than half-succeeding, and failover takes roughly thirty seconds. Browsing continues from the replicas throughout. |
| The queue backs up | Nothing customer-facing breaks. Confirmation emails and search freshness lag. This is exactly why they are below the line. |
| The warehouse API is down | Fulfilment jobs retry with backoff for hours. Orders are still taken and still paid for. Nobody at checkout finds out. |
Honesty
What I would argue about in review.
The single primary. It is the right call at two writes a second and the wrong call the moment this shop becomes a marketplace. The migration is real work, and pretending otherwise is how a design becomes a trap.
The ledger. It costs a rollup and a nightly check that the rollup still matches the sum. If the team is one person, a plain counter and an accepted risk might genuinely be the better engineering.
The CDN window. Five minutes is a guess. It should be set from how often prices actually change, and I would want a week of real numbers before defending it.
None of the above is a diagram, and all of it is the design. A drawing that cannot tell you what it gives up is a picture, not a plan.
Got a system that only breaks on the busy day?
That is usually the interesting one. Tell me what it does and where it hurts.