System designCore8 min
Concurrency
How one server with a handful of processor cores keeps thousands of requests in flight at once, the three ways to do it, and the bugs that appear the moment two things happen at the same time.
1 · The idea
Concurrency is juggling; parallelism is more hands
Watch a good barista. They start an espresso, take the next order while it pours, steam milk for a third. Five orders are in progress, and one person is making them all. That is concurrency: dealing with many things at once. Hire a second barista and two drinks are truly being made at the same instant. That is parallelism: doing many things at once.
A server works the same way. With 8 processor cores it can run at most 8 things at the same instant. Yet a busy server has thousands of requests in progress. How? Look at where one request's time goes. Take a checkout API: each request does about 1 ms of real work, then waits about 50 ms for the database to answer.
About 98% of the request's life is waiting. A processor core that sat idle through the wait would be wasting nearly all of its time.
Servers spend most of their time waiting. Concurrency is how they use it.
2 · The tools
Processes, threads and event loops: three ways to juggle
Every server needs some way to keep many requests in progress. There are three main ones, plus a modern blend:
Way | How it works | What it costs |
|---|---|---|
Processes | Separate programs, each with its own memory | The heaviest; a crash in one leaves the others running |
Threads | Many paths of execution inside one program, sharing its memory | Megabytes of reserved stack each; the operating system switches between them in a microsecond or two |
Event loop | One thread that never waits: it starts each slow operation and moves on, returning when the answer arrives | Almost nothing per request, but any long computation blocks everyone |
Lightweight threads | Written like threads, run like an event loop: the language runtime parks them cheaply while they wait | A few kilobytes each, so hundreds of thousands are practical |
Threads are not as expensive as their reputation. A modern machine runs 10,000 threads in one process without trouble. The difference shows at larger numbers, and in what happens while a request waits, which is the next section.
3 · Waiting
Blocking parks a whole thread; non-blocking parks only the request
In the classic design, each request gets its own thread, and the thread simply stops while the database answers. This is blocking I/O: easy to write, because the code reads top to bottom. The price is one parked thread per waiting request. How many? Requests in flight = arrival rate × time each takes, so for the checkout API:
- At 2,000 requests a second: 2,000 × 0.051 s ≈ 100 threads, nearly all of them waiting.
- If the database slows to 1 second per answer: 2,000 × 1 s = 2,000 threads, for the same traffic.
An event loop uses non-blocking I/O instead. It sends the query, asks the operating system to report back when the answer arrives, and goes straight on to the next request:
While A's query is out, the same single thread serves other requests. Waiting costs a note of what to do next, not a parked thread.
The catch is the reverse: an event loop hates long computation. If one request spends 300 ms resizing a photo on the loop, every other request on that loop waits 300 ms. Heavy work must be handed to separate worker threads or processes.
One more trap: some languages let only one thread run their code at a time. The standard build of Python is the best-known example. There, threads give you concurrency for waiting, but not parallelism for computing; CPU-heavy work needs several processes.
4 · Pools
Pools keep the expensive things few and reused
Some things are too costly to create for every request. Opening a database connection takes network round trips and a login. So servers keep a pool: a fixed set of open connections (or threads), borrowed for one request and handed back.
When every connection is in use, new requests wait in line for one, and give up if the wait exceeds a timeout.
How big should the pool be? The same rule as before gives the average in use: if the checkout API handles 1,000 requests a second and each holds a connection for 25 ms, then 1,000 × 0.025 = 25 connections are busy on average. Leave room above that for bursts.
But a bigger pool is not automatically faster. The database behind it has only so many cores and disks; past that point, extra connections just queue inside the database instead of in your server. Pools are bounded on purpose: they protect the thing behind them. When a pool runs dry, the waiting line forms, and Throughput, utilization & queueing explains what that line does.
5 · The danger
Two things at once can break what one at a time never would
Concurrency brings a new kind of bug. One pair of sneakers is left in stock, and two customers press Buy at the same moment. Each request runs the same three steps: read the stock, check it, write the new value.
Both read before either wrote, so both saw one pair left. Run the steps one after the other and the bug cannot happen; interleave them and it does.
This is a race condition: the result depends on the timing. This one is called a lost update, because one write silently erased the other. The fix is to make read, check and write a single step that nothing can interrupt:
- A lock (or mutex): only the request holding it may touch the stock count; the other waits its turn.
- An atomic operation: "subtract 1 only if the count is above 0", done by the processor or the database as one indivisible action.
Locks bring their own trouble:
- Deadlock. A holds lock 1 and waits for lock 2; B holds lock 2 and waits for lock 1. Neither ever moves. The usual cure: always take locks in the same order.
- Contention. A lock that every request needs lets only one through at a time. It is exactly the shared part that caps a system in Scalability.
Check yourself
5 questions
Takeaways
Remember this
- Concurrency is dealing with many things at once; parallelism is doing many at once.
- Servers spend most of each request waiting, and concurrency is how they use that time.
- Blocking I/O parks a thread per waiting request; an event loop parks only a note, but long computation stalls it.
- Pools keep expensive connections few and reused; size them from arrival rate × holding time, and keep them bounded.
- Interleaved read, check, write loses updates; make it one step with a lock or an atomic operation, and watch for deadlock and contention.