System designEasy19 min readruns in the simulator
Scale a site from one server to millions, one part at a time
How a site grows from one server to millions of users: a separate database, a load balancer, shared sessions, a cache, a CDN, replicas, autoscaling and a queue, each added when the last version breaks.
- load-balancing
- caching
- cdn
- replication
- autoscaling
- throughput
- 5,267 rps
- page p99 latency
- 179 ms
- cost
- $4,395/mo
Requirements
Functional
- Read a question and its answers, browse the home page and tag lists. Most requests are these.
- Sign in and stay signed in, whichever server answers; signing out works at once.
- Ask, answer, comment and vote; the author sees their post at once.
- Get an email or a notice when someone answers you.
- Upload an image to a post.
Non-functional
- Pages are fast: p99 under 200 ms at the daily peak.page p99 latency < 200 ms
- Writes are quick: p99 under 500 ms, however slow email is.write p99 latency < 500 ms
- Scripts, styles and images load fast for readers anywhere: p99 under 200 ms.static p99 latency < 200 ms
- 5 million daily users and 50 million page views a day; about 1,750 page views a second at the daily peak.page successful requests ≥ 1700 rps
- 99.95% of requests succeed, including while one server or one availability zone is down.
- When the database primary dies, writes pause only for the failover and nothing written is lost.
- Pay for what is used: the server count follows the traffic, not the peak.
The requirements with a measurable target are checked live in the simulator: break something and watch them fail.
Back-of-the-envelope estimates
daily users → requests, servers and database load
| Quantity | Value |
|---|---|
| Daily users input | 5,000,000 |
| Pages per user a day input | 10 |
| Peak ÷ average input | 3 |
| Database reads per page input | 8 |
| Cache hit ratio input | 0.9 |
| Pages a second per web server input | 150 |
| Page views a second at peak | 1,736 rps |
| Writes a second at peak (1%) | 17 rps |
| Database reads a second, no cache | 13,889 rps |
| Database reads a second, with the cache | 1,389 rps |
| Web servers at the peak (60% busy) | 19 |
| Web servers at the quietest hour | 2 |
| Session memory (1 KB each) | 5 GB |
Every value is computed from the inputs. Change them in the simulator and the model re-sizes itself.
High-level architecture
Peopleclient · web
Five million people a day; at the daily peak about 1,750 page views a second.
Almost everyone reads: about 99 page views for every answer, comment or vote. Each page also pulls about two static files.
CDNcdn
Serves scripts, styles and images from edges near readers; 98% never leave the edge.
Static files never change between releases, and readers are far from the region. An edge near the reader answers them in milliseconds, and the servers never see them.
Load Balancerlb · nginx
Spreads page requests over the web servers and stops sending to one that dies.
More than one web server needs something in front to spread requests and route around a dead one. Because the servers keep no state, any of them can take any request.
Web Serversserver · api
Renders pages. Twenty servers at the daily peak, two at the quietest hour; none keeps anything between requests.
A web server that remembers nothing between requests can be copied, removed or replaced without anyone noticing. That is what lets the load balancer spread the load and the autoscaler change the count.
Session Storedb · redis
Who is signed in: about 5 GB of sessions in memory, with a replica that takes over in seconds.
Sessions are the one piece of state a request needs from the last one. Kept in a shared store instead of a web server’s memory, they survive that server going away. Eviction is off: a dropped session signs someone out.
Cachecache · redis
Hot rows and rendered fragments in memory; answers 90% of page reads before they reach the database.
Without it, the peak needs about 14,000 database queries a second. A few questions get most of the views, so a memory cache in front of the database takes most of that load for a fraction of the cost.
PostgreSQLdb · postgres
Users, questions, answers, votes: one primary for writes, a standby in another zone, and replicas for reads the cache misses.
The data is relational and fits one machine: about a terabyte after five years and about 17 writes a second at the peak. So the database scales up and copies itself instead of splitting: replicas share the reads and take over when the primary dies.
Object StorageobjectStore · s3
Uploaded images and the site’s static files; the CDN serves them.
Files don’t belong in the database or on a web server’s disk, which the autoscaler may throw away. Object storage keeps them durably and the CDN fetches from it.
Job Queuequeue
Emails and notices waiting to be sent. A write adds a job and returns at once.
Sending email calls an outside service that can be slow or down. In a queue, that slowness becomes a backlog, not a slow or failed write.
Workersworker
Takes jobs off the queue and sends emails and notices.
Workers do the slow part of a write after the person has their answer. They can be added or paused without touching the web servers.
Email Providerexternal
An outside email service. It answers in about 300 ms, until it doesn’t.
Outside services fail in ways you can’t fix. Keeping them off the request path means their bad day isn’t yours.
Request paths
Open a page
Load BalancerWeb ServersSession StoreCacheon a missPostgreSQL
Opening a page
- Any server will do.
The browser asks for a question page. The load balancer picks any healthy web server: they are all the same, and none remembers this reader.
- Who is this?.
The server reads the session cookie and looks it up in the session store: signed in as someone, or anonymous. One memory read, about a millisecond.
- The cache first.
It asks the cache for the question, its answers and their votes. Nine times in ten everything is there.
- A miss reads a replica.
On a miss it reads the rows from one of the database copies and puts them in the cache for the next reader.
- Static files from the edge.
The page then pulls its scripts, styles and images from a CDN edge near the reader, which almost always has them.
Ask, answer or vote
Load BalancerWeb ServersSession StorePostgreSQLasyncJob QueueWorkersEmail Provider
Posting an answer
- Commit to the primary.
Someone posts an answer. The server writes it to the primary, which waits for the standby in the other zone to confirm, so a lost machine can’t lose it.
- The email waits in line.
The server puts a job on the queue, "tell the asker", and returns. The answer is posted in about a fifth of a second.
- A worker sends it.
A worker takes the job and calls the email provider. If the provider is slow today, the queue grows; nobody posting waits for it.
Upload an image
Load BalancerWeb ServersSession StoreObject StoragePostgreSQL
Load a script, style or image
CDNon a missObject Storage
Opening a page
- Any server will do.
The browser asks for a question page. The load balancer picks any healthy web server: they are all the same, and none remembers this reader.
- Who is this?.
The server reads the session cookie and looks it up in the session store: signed in as someone, or anonymous. One memory read, about a millisecond.
- The cache first.
It asks the cache for the question, its answers and their votes. Nine times in ten everything is there.
- A miss reads a replica.
On a miss it reads the rows from one of the database copies and puts them in the cache for the next reader.
- Static files from the edge.
The page then pulls its scripts, styles and images from a CDN edge near the reader, which almost always has them.
How the design evolved
Stage 1
One server
The web app and its database run on one machine, sharing its CPU and memory. It’s the right way to start: cheap and simple. But the two fight for the same machine, so pages slow down early, and when the machine dies the whole site goes with it.
breaks at 48 rpswhen page p99 latency < 200 ms stops holding$146/mo
Stage 2
The database on its own machine
The database moves to its own machine, sized for memory and disk, and the web server gets all of its CPU. Each can grow on its own. But there is still one web server, and it is the ceiling.
breaks at 97 rpswhen page p99 latency < 200 ms stops holding$847/mo
Stage 3
A load balancer, more servers, shared sessions
Sessions move to a shared store so the web servers keep no state, and a load balancer spreads pages over many of them. The web tier now grows by adding copies. Every page still reads the database, and the database is one machine.
breaks at 460 rpswhen page p99 latency < 200 ms stops holding$3,811/mo
Stage 4
A cache and a CDN
A shared cache answers most page reads from memory, and a CDN serves static files from edges near readers. The database now sees only the misses. But it is still one machine: a cold cache or a dead primary takes the site with it, and the web fleet is fixed at its peak size.
breaks at 5,700 rpswhen page p99 latency < 200 ms stops holding$4,212/mo
Stage 5
Replicas and a standby
The database gets copies in another zone: they share the reads and one takes over when the primary dies. Reads are now safe. But every answer still sends its email before it returns, so writes are as slow as the email provider, and fail with it.
breaks at 525 rpswhen write p99 latency < 500 ms stops holding$4,212/mo
Final design
Autoscaling and a queue
The web tier grows and shrinks with traffic, and its floor is raised ahead of known launches. Emails and notices go through a queue to workers, so writes return at once. This is the design on the board.
Key decisions
What is the first thing to split off?
Chosen
The database, onto its own machine: it needs memory and disk, the web server needs CPU, and only the web server can be copied.
Why not
- A bigger single machine. It buys time, but it still fails as one and hits a ceiling.
Source: Scaling up to your first 10 million users (notes on the AWS talk)
Where do sessions live?
Chosen
In a shared in-memory store with eviction off and a replica, so any web server can answer any request.
Why not
- Sticky sessions. A dying or removed server signs its users out, and load piles up unevenly.
- Signed cookies only. A stolen or signed-out session can’t be cancelled at once.
How do reads stay off the database?
Chosen
A shared cache-aside cache in front of it, answering about 90% of page reads.
Why not
- Only read replicas. At most 15 per primary, each a full database to pay for, and slower per read than memory.
How does the database grow?
Chosen
Up, then copied: one primary for writes, a standby in another zone, and replicas for reads. No shards at this size.
Why not
- Shard now. About 17 writes a second and a terabyte fit one machine; shards would make every query know where its data lives.
- Read from the primary only. A cold cache or a dead primary then takes the pages down.
Try the alternative in the simulator →
Sources: Stack Overflow: The Architecture, 2016 Edition, How Figma’s databases team lived to tell the scale
What happens when the primary dies?
Chosen
A replica in another zone is promoted automatically, in about a minute; writes pause, reads carry on.
Why not
- Wait for the primary. Writes fail for as long as the restart takes.
Who serves static files?
Chosen
A CDN, from object storage: edges near readers answer almost all of them.
Why not
- The web servers. Two thirds of requests would be bytes that never change, paying the full trip to the region.
How many web servers?
Chosen
As many as traffic needs, from 2 to 100, aiming at 60% busy; the floor is raised ahead of a known launch.
Why not
- A fixed fleet sized for the peak. It pays for the peak all night.
- Autoscaling alone for launches. New servers take time to start, so a fast climb outruns it.
When are emails sent?
Chosen
After the write, by workers reading a queue: the answer returns at once and a slow provider becomes a backlog.
Why not
- Inside the request. Every write waits for the provider and fails with it.
One region or several?
Chosen
One region, every tier across at least two availability zones.
Why not
- Several regions. It survives a lost region, but cross-region failover can split writes; it is a later lesson.
One part at a time
Start with one machine
Every site starts on one machine, and it should: one server is cheap, simple and fast enough for the first users. The trouble is that it does everything at once, so the first limit it hits stops everything. The evolution below adds one part each time the site breaks, and only that part.
The order, and what forces each step
- **A separate database:** the web server needs CPU, the database needs memory and disk, and they fight over one machine.
- **A load balancer and more web servers:** one web server runs out of CPU. Copies only work if the servers keep no state, so sessions move to a shared store.
- **A cache and a CDN:** now the database is the limit; most reads repeat, and static files never change.
- **Replicas and a standby:** the database is one machine, and a cold cache or a dead primary takes everything with it.
- **Autoscaling and a queue:** a fixed fleet is wrong at every hour but one, and slow outside services sit in the write path.
Stateless servers, and where sessions go
Why state has to leave the web servers
A web server that remembers who you are can’t be replaced without signing you out. So before the load balancer can spread requests freely, and before an autoscaler can add and remove servers, the one piece of state a request needs, the session, moves to a shared store that every server reads.
The alternatives
- **Sticky sessions** pin each user to one server: no store needed, but a dying server signs its users out and load piles onto busy servers.
- **Signed cookies only** carry the session in the browser: no store at all, but a stolen or signed-out session can’t be cancelled at once.
- **A shared session store** with eviction off and a replica: one memory read per request, and any server can answer anyone.
One primary goes a long way
Reads and writes need different fixes
Reads can be copied: a cache in front, replicas behind. Writes can’t: every replica replays every write, so the primary carries all of them. That makes the real question “do the writes fit one machine?”, and for a site like this one the answer is yes for a long time: about 17 writes a second at the peak and about a terabyte after five years.
What real sites did
Stack Overflow served about 66 million page loads a day in 2016 from two database primaries, with a cache doing most of the reading. Figma ran on a single PostgreSQL primary on the largest machine AWS sold, and sharded only when tables reached several terabytes and writes neared the machine’s limit. Sharding is the answer to too many writes, not to too many readers.
Autoscaling follows, it doesn’t lead
What it’s good at
Traffic has a daily shape: at the quietest hour the site needs a fraction of its peak servers. Autoscaling follows that shape so you don’t pay for the peak all night. It only works on servers that keep no state, which is why it comes after the session store.
What it isn’t
A new server takes time to start, and autoscaling reacts after the load arrives. A launch that climbs in a minute outruns it. The launch story shows the fix: for a spike you know about, raise the floor ahead of time and let autoscaling bring it back down.
Why one region
Zones, not regions
Every tier here spans at least two availability zones, separate buildings with their own power, in one region. That survives a lost machine or a lost building. Surviving a lost region needs copies across regions, and failing writes over across regions is where split writes happen: in 2018 a 43-second network cut left GitHub degraded for a day after writes landed on both coasts. It’s a later lesson.
What happens when it breaks
Launch day
A launch brings five times the peak over one minute.
| Reading | Healthy | During the failure |
|---|---|---|
| p99 latency | 173 ms | 173 ms |
| error rate | 0% | 0% |
| throughput | 5,267 rps | 5,267 rps |
| cost | $4,395/mo | $4,395/mo |
- Pages are fast: p99 under 200 ms at the daily peak. (still holds)
- Writes are quick: p99 under 500 ms, however slow email is. (still holds)
- Scripts, styles and images load fast for readers anywhere: p99 under 200 ms. (still holds)
- 5 million daily users and 50 million page views a day; about 1,750 page views a second at the daily peak. (still holds)
The launch starts in five seconds. Autoscaling is on. What do you do?
- Raise the floor to 100 servers before it starts.
Every page works: p99 stays at or under about 184 ms all the way up. The extra servers cost about $16,000 a month while they run, so the floor comes back down after the launch.
- Leave it to autoscaling.
Autoscaling reacts, but each new server needs 30 seconds and the fleet at most doubles per step. For about a minute more than 80% of pages fail; the site recovers at about 1:50 with 99 servers.
- Turn autoscaling off and keep 20 servers.
The 20 servers run at nearly three times their capacity: 91% of pages fail for as long as the launch lasts.
Autoscaling follows traffic; it can’t lead it. New servers take time to start, so a sudden climb outruns it. For a launch you know about, raise the floor ahead of time and let autoscaling bring it back down; for surprises, keep headroom.
Trade-offs
Sending email
| Option | Gain | Cost |
|---|---|---|
| Through a queuedefault | Writes never wait for the provider; a slow day becomes a backlog. | Emails can go out late, and there is a queue and workers to run. |
| In the request | The email is sent before the answer is confirmed. | Every write waits about 300 ms more, and fails when the provider fails. |
Database reads
| Option | Gain | Cost |
|---|---|---|
| Spread over replicasdefault | Four copies share the reads, and reads survive the primary. | A replica can be a fraction of a second behind. |
| Primary only | Every read sees the latest write. | One machine carries every read; a cold cache or a dead primary takes pages down. |
Fixes and what they cost
| Fix | What it does | Trade-off |
|---|---|---|
| Double the web servers | Twice the servers right now. | Costs more, and autoscaling may take them away again when traffic is low. |
| Raise the floor to 100 servers | Starts 100 web servers and keeps at least that many. | About $16,000 a month while it lasts: right for a known launch, wasteful as a habit. |
| Four times the workers | The queue drains four times as fast. | About $440 a month more, mostly idle on a good day. |
| Double the database machines | Bigger machines for the primary and its replicas. | Twice the database bill, for headroom the cache usually covers. |
| Spread reads over the replicas | Reads go to all four copies. | A replica can be a fraction of a second behind the primary. |
| Read from the primary only | Every read sees the latest write. | One machine carries every read, and reads fail with it. |
| No automatic failover | A person decides when a replica takes over. | Writes fail until someone acts or the primary restarts. |
| Turn the cache off | Every page read goes to the database. | About ten times the database reads. |
| Turn the CDN off | Every static file comes from object storage, far from the reader. | Each file pays the long trip, and object storage takes every request. |
| Turn autoscaling off | The web tier stays at its current size. | Sized for the peak it wastes money all night; sized for the night it fails at the peak. |
Interview questions
What is the first thing to scale in a web app?
Move the database to its own machine, then put several web servers behind a load balancer. That only works once the web servers keep no state, so sessions move to a shared store first.
When do you need to shard a database?
When writes or table sizes outgrow the biggest machine you can buy. Reads are handled by a cache and read replicas. Many large sites ran on one primary for years: Figma sharded only when tables reached terabytes and writes neared the machine’s limit.
Sticky sessions or a session store?
A session store. Sticky sessions sign users out when their server dies or is removed, and block autoscaling from removing servers. A shared store costs one fast memory read per request.
Cache or read replicas?
Both, for different jobs. The cache answers most reads from memory and is far cheaper per read; the replicas carry the reads when the cache is cold and keep reads alive when the primary dies.
Is autoscaling enough for a traffic spike?
Not for a fast one. New servers take time to start, so autoscaling lags a sudden climb. Raise the minimum ahead of a known launch, and keep some headroom for surprises.
Scale up or scale out?
Both, in order. Scale up the parts that are hard to copy, like the database primary, and scale out the parts that are easy to copy, like stateless web servers. A bigger machine buys time; copies buy capacity and survival.
Sources
- Stack Overflow: The Architecture, 2016 Edition, Nick Craver
- Scaling up to your first 10 million users (notes on the AWS talk), Scott Lowe
- Scaling Pinterest: from 0 to 10s of billions of page views a month in two years, High Scalability
- Instagram architecture: 14 million users, terabytes of photos, High Scalability
- How Figma’s databases team lived to tell the scale, Figma
- Failover process for Amazon RDS Multi-AZ, AWS
- RDS for PostgreSQL supports up to 15 read replicas, AWS
- October 21 post-incident analysis, GitHub