System designEasy22 min readruns in the simulator
Design Pastebin: object storage, a CDN and a few good limits
How a Pastebin-like paste site is designed: paste text in object storage, unguessable links, a CDN for viral pastes, expiry, and limits for spammers and link guessers, with a simulator.
- object-storage
- cdn
- postgres
- security
- read-heavy
- throughput
- 533 rps
- view p99 latency
- 154 ms
- cost
- $1,000/mo
Requirements
Functional
- Paste text and get a link at once: up to 512 KB for guests and free accounts, 10 MB for paid accounts. About 10 pastes a second at peak.create successful requests ≥ 9.5 rps
- Choose who can see a paste: anyone (public), anyone with the link (unlisted), or only its signed-in owner (private).
- Choose when a paste expires, from 10 minutes to never; a year by default. An expired paste is gone everywhere within a minute, cached copies included.
- Burn after reading: the first view shows the paste, then it is gone for everyone. Optionally protect a paste with a password.
- Nobody finds unlisted pastes by guessing links: a guesser is refused long before guessing could pay off.scrape refused over quota > 90%
- Spam and stolen data are caught: limits on creating, a scan of every new paste, reports, and takedowns that also clear cached copies.
Non-functional
- Opening a public or unlisted paste is fast: p99 under 200 ms at peak.view p99 latency < 200 ms
- Opening a private or protected paste, which is never cached, is fast too: p99 under 200 ms.view-direct p99 latency < 200 ms
- Creating a paste is quick: p99 under 500 ms, however long abuse scanning takes.create p99 latency < 500 ms
- 300,000 new pastes and 15 million views a day in five years, three times the average at peak: about 520 views and 10 creates a second.throughput ≥ 520 rps
- Viewing keeps working while one database server or one app server is down; creating may pause for a failover.
- Once its link is shown, a paste survives the loss of any one machine or disk, until it expires or is deleted.
The requirements with a measurable target are checked live in the simulator: break something and watch them fail.
Back-of-the-envelope estimates
new pastes a day → requests, storage and what is kept
| Quantity | Value |
|---|---|
| New pastes a day input | 300,000 |
| Views per paste input | 50 |
| Peak ÷ average input | 3 |
| Average size (KB) input | 10 |
| Compression input | 3 |
| Share kept a year (the default) input | 0.8 |
| Share kept forever input | 0.1 |
| Years input | 5 |
| New pastes a second at peak | 10 rps |
| Views a second at peak | 521 rps |
| Text written a year | 365 GB |
| Pastes kept | 142,350,000 |
| Text kept | 475 GB |
| Database size | 42.7 GB |
| Expiries a second, after the first year | 3 rps |
Every value is computed from the inputs. Change them in the simulator and the model re-sizes itself.
High-level architecture
Peopleclient · web
People pasting text and opening pastes: about 520 views and 10 new pastes a second at peak.
Almost everyone reads. A paste is created once and opened about fifty times, mostly by people who were sent the link, so the design is built around serving reads cheaply and keeping writes simple.
Link Guesserclient
Someone trying random links, hoping to find unlisted pastes. Refused at the gateway.
Unlisted pastes are protected only by their link. A guesser tries links at random; the design makes guessing pointless with long random keys, and cheap to refuse with a budget of not-found answers per address.
Spam Botnetclient
Five thousand infected machines posting spam links. Quiet until the spam story.
Paste sites are a cheap place to host spam and malware links. A botnet spreads its posts over thousands of addresses, so each one looks like a guest posting a few pastes.
Every Minuteclient · schedule
A timer that starts the expiry job every minute.
Expired pastes must disappear for good, not only stop being shown. Once a minute a scheduler starts the job that deletes them.
CDNcdn
Serves public and unlisted pastes from the edge. A viral paste is fetched from the servers about once per edge location.
A paste never changes after it is saved, which makes it ideal to cache. The CDN answers most views in milliseconds near the reader, and a paste linked from a busy page costs the servers almost nothing. Private, password-protected and burn-after-reading pastes are never cached.
Load Balancerlb · nginx
Spreads requests over the paste servers and drops floods from a single address.
More than one paste server means something must spread the load and route around a dead one. The balancer also applies a coarse per-address limit, far above what an office sends, so a flood from one address stops here.
API GatewayapiGateway
Applies the paste site’s limits: views per address, pastes per address per day, pastes for everyone together, and a budget of not-found answers.
Pastebin’s abuse is cheap to send and expensive to serve. The gateway holds every limit in one place, before any paste server does work: guests get 10 pastes a day per address, creates are capped for everyone together, and an address that keeps asking for pastes that don’t exist is refused.
Paste Serviceserver · api
Creates pastes, checks visibility, expiry and passwords, and returns the text. Three servers, no state of their own.
The servers keep nothing between requests, so any of them can serve any paste, and adding more is just adding more. A view reads the paste’s details from the database and its text from object storage; the browser highlights the code.
PostgreSQLdb · postgres
Each paste’s details: key, owner, visibility, expiry, password hash, size. A primary, a standby that confirms every write, and reads from the replicas.
The details are small, about 300 bytes a paste, and the questions are relational: my pastes, newest first; pastes expired before now; mark this burn-after-reading paste read if nobody has. About 45 GB after five years fits one primary with room to spare, so the copies are for staying up, not for scale.
Object StorageobjectStore · s3
The paste text itself, compressed: about 0.5 TB after five years, kept with copies in several places.
Pastes can be up to 10 MB. Object storage keeps large values durably and cheaply, about two cents a gigabyte a month, so the database stays small. Its first byte is slow next to memory, which is one more reason the CDN sits in front.
Scan Queuequeue
New pastes waiting to be scanned. It lets creates finish at once and absorbs bursts.
Scanning takes about 300 ms a paste. On the request it would push creates past a second at p99; in a queue it costs nobody a wait, and a burst of spam becomes a backlog instead of an outage.
Abuse Scannerworker
Reads each new paste and looks for credential lists, personal data and malware. A hit hides the paste and purges it from the CDN.
Paste sites carry leaked passwords and malware payloads, and people watch them closely: leaked pastes are often picked up within a minute. Every new paste is checked, and reports from readers go through the same scanners.
Expiry Jobworker
Deletes expired pastes every minute: their rows, then their text. About 3 a second once the first year has passed.
With a one-year default, about 240,000 pastes expire every day. Views already refuse an expired paste, so the job is not about hiding them; it keeps the stored data level instead of growing forever. It walks an index on the expiry time in batches.
Request paths
Open a public or unlisted paste
CDNon a missLoad BalancerAPI GatewayPaste ServicePostgreSQLObject Storage
Opening a paste
- The edge first.
The link goes to the CDN. If a reader near you opened this paste recently, the edge answers in milliseconds: that’s about four views in five.
- A miss goes to the servers.
On a miss the edge asks the servers. The paste server reads the row from a replica: is it expired, is it public or unlisted?
- The text.
It fetches the text from object storage and returns it with a cache lifetime no longer than the paste has left to live. The browser highlights the code.
Open a private or protected paste
Load BalancerAPI GatewayPaste ServicePostgreSQLObject Storage
Create a paste
Load BalancerAPI GatewayPaste ServiceObject StoragePostgreSQLasyncScan QueueAbuse ScannerObject Storage
Creating a paste
- Send the text.
Someone pastes a log and presses Create. The gateway checks this address has pastes left today and that creates as a whole are under their cap.
- Text to object storage.
A paste server picks a random 10-character key, compresses the text and stores it in object storage under that key.
- Details to the database.
Then it writes the row: key, owner, visibility, expiry a year from now, size. The write waits for one replica to confirm, so a single lost machine can’t lose the paste.
- The link comes back.
The link is returned at once, in about a fifth of a second at p99.
- Scanned after.
Meanwhile the key went onto the scan queue. A scanner reads the paste and looks for credential lists and malware; a hit hides it and purges it from the CDN.
Guess random links
Load BalancerAPI GatewayPaste ServicePostgreSQL
Post spam
Load BalancerAPI GatewayPaste ServiceObject StoragePostgreSQLasyncScan QueueAbuse ScannerObject Storage
Delete expired pastes
Expiry JobPostgreSQLObject Storage
How the design evolved
Stage 1
One server, one database
The first version is one paste server and a PostgreSQL database holding everything, text included. It is simple and works for a small site. Every view reads a whole paste from the database through the one server, so that server is the ceiling, and nothing is left when it or the database dies.
breaks at 325 rpswhen view p99 latency < 200 ms stops holding$453/mo
Stage 2
Text in object storage, servers behind a load balancer
The text moves to object storage, so the database holds only small details, and the servers keep no state, so a load balancer spreads views over three of them. It holds today’s peak. But every view still reaches a server and object storage, so a busy day, or one viral paste, fills the servers.
breaks at 785 rpswhen view p99 latency < 200 ms stops holding$628/mo
Stage 3
A CDN in front
Pastes never change, so a CDN caches public and unlisted ones near readers and a viral paste stops reaching the servers. Views are now fast and cheap. But nothing refuses anyone: a single guesser trying random links is served in full, at every load.
breaks at 58.5 rpswhen scrape refused over quota > 50% stops holding$774/mo
Final design
Limits, scanning and replicas
An API gateway holds the limits: views per address, 10 pastes a day per guest address, a cap on all new pastes, and a budget of not-found answers that makes guessing cheap to refuse. New pastes are scanned for leaked data and malware after saving, through a queue. The database gets two replicas that serve reads and take over when the primary dies, and a job deletes expired pastes every minute. This is the design on the board.
Key decisions
Where does the paste text live?
Chosen
In object storage, compressed, with the paste’s details in PostgreSQL. The database stays small and the text is kept durably for about two cents a gigabyte a month.
Why not
- In the database row. Every replica, backup and restore carries the text. It works at first and is outgrown, not wrong.
- A key-value store alone. DynamoDB caps items at 400 KB, below the 512 KB and 10 MB limits.
Source: Best practices for storing large items and attributes
How are links made?
Chosen
10 random letters and digits, checked by the database’s unique key, with a retry on the rare clash. Guessing finds a paste about once in six billion tries.
Why not
- A counter turned into letters. Consecutive links can be listed one by one, which exposes every unlisted paste.
- 8 random characters. With 140 million pastes, a guesser with 1,000 addresses would find about 1,100 a day.
- A separate key service. Pre-generating keys pays off at thousands of writes a second; this site makes about 10.
How is the database run?
Chosen
One PostgreSQL primary and two replicas: each write waits for one replica, reads go to the replicas, and a replica is promoted automatically when the primary dies.
Why not
- Read from the primary only. Pastes never change, so nothing is gained, and views fail with the primary.
- Shards or a NoSQL cluster. About 45 GB after five years and 10 writes a second fit one machine.
How are views served fast?
Chosen
Public and unlisted pastes through a CDN, cached at most a day and never past their expiry. Private, password-protected and burn-after-reading pastes are never cached.
Why not
- More servers instead. A viral paste is one key in object storage, which serves about 5,500 reads a second per prefix.
- A memory cache at the servers only. Readers still wait for the round trip to the servers, and a viral paste still reaches them.
How is abuse limited?
Chosen
At the gateway: views per address, 10 new pastes a day per guest address, a cap on all new pastes together, and a budget of not-found answers per address.
Why not
- Per-address limits only. A botnet or a guesser spread over a thousand addresses stays under every one.
- More servers instead of limits. It pays to serve abuse, and spam still fills the database and the scanners.
Try the alternative in the simulator →
Source: Pastebin FAQ (archived copy)
When are pastes scanned for abuse?
Chosen
After saving, through a queue: the link comes back at once, and a burst of spam becomes a backlog.
Why not
- Before the link is returned. Every create waits about 300 ms more, and a spam flood slows or fails honest creates.
How do pastes expire?
Chosen
A view refuses an expired paste at once, CDN copies never outlive it, and a job deletes expired pastes every minute in batches. The default expiry is one year.
Why not
- Storage lifecycle rules. They act in days, not minutes.
- The cleanup job alone. A paste would stay visible until the next run.
- Never by default. Stored text would grow to about 1.8 TB in five years instead of leveling off near 0.5 TB.
Where is code highlighted?
Chosen
In the browser: the server sends the text and its language, so every reader gets the same cacheable response.
Why not
- On the server. It spends server time on every view that misses the CDN, for no difference the reader sees.
Can the server read pastes?
Chosen
Yes: pastes are encrypted in transit and at rest, and the server can read them, so it can scan for leaked data and malware.
Why not
- End-to-end encryption, the key in the link. It protects users from the operator, but the operator can no longer scan for this site’s main abuse.
Where the text lives
Small details, big text
A paste is two very different things: a few hundred bytes of details (key, owner, visibility, expiry) and the text itself, a few kilobytes usually and up to 10 MB. The details are queried in many ways; the text is only ever fetched whole, by key. So they live apart: the details in PostgreSQL, the text compressed in object storage.
Why not one store?
- **Everything in PostgreSQL** works at first: large values are moved out of the row and compressed. But every replica, backup and restore then carries the text, and the database grows by about 365 GB a year instead of about 9 GB.
- **A key-value store alone** fails on size: DynamoDB caps an item at 400 KB, below the 512 KB limit for free accounts.
- **Details in the database, text in object storage**: the database stays small enough to sit in memory, and the text is kept durably for about two cents a gigabyte a month.
Links nobody can guess
Unlisted means the link is the password
Most pastes are unlisted: anyone with the link can open them, nobody else can find them. That makes the link a secret, so it must be random. A counter turned into letters, the way many URL shorteners make keys, lets anyone list every paste in order.
How long?
| Characters | Possible links | Found a day by 1,000 addresses at 20 guesses a second |
|---|---|---|
| 8 | About 218 trillion | About 1,100 pastes |
| 10 | About 840 million billion | About one every three days |
Length and a budget
With 140 million pastes kept, 8 characters is not enough against someone with rented addresses. Ten characters makes each guess about a one-in-six-billion shot. The not-found budget adds the second half: an address that keeps asking for pastes that don’t exist is refused, so guessing is both hopeless and cheap to stop.
A CDN for pastes that never change
Immutable means cacheable
A paste is never edited, so a cached copy is never stale until the paste expires or is deleted. Public and unlisted pastes go through a CDN: most views are answered near the reader, and a paste linked from a busy page costs the servers one fetch per edge location. The viral story shows what happens without it: one key in object storage can’t be split across servers.
What must never be cached
- **Private pastes**: only the owner may see them, so they are checked on every request.
- **Password-protected pastes**: the password is checked on every request.
- **Burn after reading**: a cached copy would be a second reading.
- **Everything else** is cached for at most a day and never past its expiry, and a delete or takedown purges it by URL.
Who abuses a paste site
Four kinds of abuse
- **Spammers** post links by the thousand: 10 pastes a day per guest address, and a cap on all new pastes together.
- **Leakers** post stolen passwords and personal data: every new paste is scanned, readers can report, and a takedown also purges the CDN.
- **Malware authors** host payloads: the same scan, looking for known patterns.
- **Guessers and scrapers** hunt for unlisted pastes: long random links and a budget of not-found answers per address.
Why scanning happens after saving
Scanning a paste takes about 300 ms. Before the link is returned, that time is added to every honest create, and a flood of spam slows everyone. After saving, through a queue, nobody waits and a flood becomes a backlog. The price is a window, seconds normally and a minute or so in a flood, in which a bad paste is reachable. Leaked data is often picked up by watchers within a minute, so the queue is kept short and reports jump it.
Expiry, burn after reading and deletes
Gone within a minute, everywhere
Three things together make an expired paste disappear. A view refuses any paste past its expiry time, at once. The CDN never caches a paste longer than it has left to live. And every minute the expiry job deletes expired rows and their text in batches, using an index on the expiry time. Storage lifecycle rules can’t do this job: they work in days.
A year by default
Most people never choose an expiry, so the default decides how much is kept. With "never" as the default the text would grow to about 1.8 TB in five years; with a year it levels off at about 0.5 TB, and "never" stays a choice for the pastes that need it.
Shown exactly once
Burn after reading is a race: two people can open the link at the same moment. One conditional update settles it, setting the paste as read only if it wasn’t already. Exactly one view wins and gets the text; the paste is never cached.
What happens when it breaks
A paste goes viral
A paste is linked from a busy front page: 5,000 views a second on one key.
| Reading | Healthy | During the failure |
|---|---|---|
| p99 latency | 158 ms | 72.9 ms |
| error rate | 0% | 0% |
| throughput | 533 rps | 5,533 rps |
| refused over quota | 50 rps | 50 rps |
- Paste text and get a link at once: up to 512 KB for guests and free accounts, 10 MB for paid accounts. About 10 pastes a second at peak. (still holds)
- Opening a public or unlisted paste is fast: p99 under 200 ms at peak. (still holds)
- Opening a private or protected paste, which is never cached, is fast too: p99 under 200 ms. (still holds)
- Creating a paste is quick: p99 under 500 ms, however long abuse scanning takes. (still holds)
- 300,000 new pastes and 15 million views a day in five years, three times the average at peak: about 520 views and 10 creates a second. (still holds)
- Nobody finds unlisted pastes by guessing links: a guesser is refused long before guessing could pay off. (still holds)
The CDN is carrying the viral paste. Could more paste servers do the same job without it?
- Keep the CDN.
5,480 views a second at a p99 of about 69 ms. The paste servers stay at about 13%.
- Turn the CDN off and run five times the servers.
The servers reach 92% and object storage passes 100%: one paste is one key, and a key prefix serves about 5,500 reads a second. 83% of views fail, for about $700 a month more.
A paste never changes, so the cheapest copy is the one nearest the reader. Servers can be added; one hot key in object storage can’t be split. The CDN turns 5,000 reads of one key into a handful.
Trade-offs
Abuse scanning
| Option | Gain | Cost |
|---|---|---|
| After savingdefault | Creating a paste never waits for the scan, and a burst becomes a queue. | A bad paste is reachable for the seconds, or minutes in a flood, before its scan. |
| Before the link | Nothing is reachable before it is scanned. | Every create waits about 300 ms more, and a burst of spam slows or fails honest creates. |
Database reads
| Option | Gain | Cost |
|---|---|---|
| From replicasdefault | Views keep working while the primary is down, with three times the read capacity. | A replica can be a fraction of a second behind, which is harmless for pastes that never change. |
| Primary only | Every read sees the latest write. | Reads fail with the primary, and it carries every read alone. |
Fixes and what they cost
| Fix | What it does | Trade-off |
|---|---|---|
| Double the paste servers | Twice the room for views that miss the CDN, private views and creates. | Costs more, and does nothing for one hot key in object storage. |
| Four times the scanners | The scan queue drains four times as fast. | About $350 a month more, mostly idle outside an attack. |
| Double the database | Bigger database machines. | Twice the cost, for a database that is almost idle unless something gets past the limits. |
| Limit not-found answers per address | An address that keeps asking for pastes that don’t exist is refused. | A reader who mistypes many links in a row is refused for a few minutes. |
| Drop the not-found budget | Guesses are limited only per address, like any view. | A guesser spread over many addresses reaches the servers in full. |
| Drop the cap on all new pastes | Only the per-address daily limit holds creates. | A botnet’s first pastes from every address all land at once. |
| Turn the CDN off | Every view goes to the paste servers. | The servers and object storage take every view, viral pastes included. |
| Read from the primary only | Every read sees the latest write. | Pastes never change, so nothing is gained, and reads fail while the primary is down. |
| No automatic failover | A person decides when a replica takes over. | New pastes fail until someone acts or the primary restarts. |
Interview questions
Should Pastebin store pastes in a database or in object storage?
Both: the details (key, owner, visibility, expiry) in a relational database, and the text in object storage. Pastes can be up to 10 MB, which bloats every replica and backup of a database, and some key-value stores cap items at 400 KB. Object storage keeps the text durably and cheaply, and the database stays small and fast.
Source: Best practices for storing large items and attributes
How long should a Pastebin link be?
Long enough that guessing can’t find unlisted pastes. With 140 million pastes, 8 random letters and digits let someone with 1,000 addresses find about 1,100 pastes a day; 10 characters cut that to one every few days. Use random keys, never a counter, and limit how many not-found answers an address gets.
How does a paste site delete expired pastes?
A view refuses an expired paste at once, the CDN never caches a paste past its expiry, and a job runs every minute to delete expired rows and their text in batches, using an index on the expiry time.
How does burn after reading work?
The first view marks the paste as read with a conditional update that succeeds only if it wasn’t read already, so exactly one of two simultaneous views gets the text. Such pastes are never cached by the CDN.
Why put a CDN in front of Pastebin?
Pastes never change, so cached copies are never stale. A CDN answers most views near the reader, and a paste that goes viral costs the servers one fetch per edge location instead of thousands of reads of the same key a second.
Should pastes be encrypted end to end?
It depends on the requirements. End-to-end encryption, with the key in the link, protects users from the operator, but then the operator can’t scan pastes for leaked passwords or malware, the main abuse of a public paste site. A public paste site keeps pastes readable by the server and encrypts them in transit and at rest.
Sources
- Pastebin.com, Wikipedia
- Pastebin surpasses 10 million active pastes, TechCrunch
- Pastebin FAQ (archived copy), Scribd
- Pastebin API wrapper documentation, pbwrap
- What is Pastebin?, Fortinet
- Watching Have I Been Pwned Pastebin, Troy Hunt
- PrivateBin configuration defaults, PrivateBin
- Best practices design patterns: optimizing Amazon S3 performance, AWS
- Best practices for storing large items and attributes, AWS