Disaster recovery & backups

A payroll service pays the staff of 40,000 small businesses. Its database has a synchronous standby in a second zone and an asynchronous replica in another region. It survives a dead server, a dead zone, even a dead region. Then, at 14:00 on the 27th, a migration with a bug in its WHERE clause sets every employee's bank account to the same value. Within a second the standby and the remote replica have faithfully copied the damage.

Redundancy protects against copies disappearing. It does nothing about copies being wrong:

None of these rows is hypothetical:

• 2017: an engineer at a code-hosting company ran a delete on the primary database instead of a replica. About 300 GB was gone in seconds; about 6 hours of data was lost and the site was down for about 18 hours, partly because the regular backups had been silently failing.
• March 2021: a fire destroyed one data-center building in Strasbourg and badly damaged another. Customers whose paid backups were stored in the same building lost them with it, and a court later ordered the provider to compensate two of them.

Two numbers define what recovery must achieve:

• Recovery point objective (RPO): the maximum acceptable time between the last recovery point and the disaster, which is the data you agree to lose.
• Recovery time objective (RTO): the maximum acceptable time between the interruption and restoring service.

They come from the business, through a business impact analysis: what does an hour of downtime, or an hour of lost data, cost? Then they are weighed against the cost of achieving them. A recovery strategy that costs more than the loss it prevents is not worth building, unless a regulator requires it.

The same system can deserve different targets at different times. The payroll service on the 3rd of the month can be down for a day and nobody notices. On the 27th, with payments due on the 28th, an hour matters. Different parts deserve different targets too:

One subtlety changes everything about corruption. When data is corrupted, the usable recovery point is before the corruption started, not before it was noticed. A bug that quietly damages records for three days before anyone notices gives a recovery point three days old, whatever the backup schedule says, unless you can repair forward from logs. Keep backups far enough back to reach behind the slowest discovery you can imagine.

At 14:00 the bad migration ran. The fix is to rebuild the database as it was at 13:59, then re-apply the legitimate changes made since. That needs point-in-time recovery (PITR): a full base backup taken periodically, plus every change since, kept in the database's write-ahead log (Storage engines: B-trees, LSM trees & WAL):

How far back you can go, and how much you lose, depends on the method:

A restore usually creates a new database rather than rewinding the old one. That is a feature: you can inspect the restored copy, and production keeps running while you do. Plan for where it runs, how the app is pointed at it, and how its data reaches production.

A backup in the same building, the same account or under the same credentials shares the original's fate. The oldest rule of thumb is 3-2-1: keep 3 copies of important data, on 2 different kinds of storage, with 1 copy off-site. Modern threats add two more requirements:

Why each part matters:

• Another region survives the loss of a site, unlike the Strasbourg backups.
• Another account survives stolen production credentials. Ransomware routinely looks for reachable backups to delete or encrypt them before demanding payment.
• Write-once storage survives even an administrator. S3 Object Lock in compliance mode, for example, prevents anyone, including the account's root user, from deleting a locked version until its retention date.
• Offline copies are the standard defense against ransomware, together with regular tests that they can actually be restored. Immutable cloud storage is an alternative, configured with care.

Back up more than the database. A restore needs everything that rebuilds the service: infrastructure as code, configuration, machine images, secrets, and the encryption keys for the backups themselves. A backup encrypted with a key that lived only in the lost region can't be read.

A backup sets the recovery point; the restore sets the recovery time. The payroll ledger is 4 TB. Restore time is roughly size ÷ throughput, before replaying any logs:

The copy is only one step. The full recovery time adds:

• Deciding to restore, and to which moment, which is often the slowest step in a real incident.
• Provisioning the database and servers, fastest with infrastructure as code.
• Replaying logs from the base backup to the chosen moment; a week of changes takes far longer than an hour of them.
• Warming up: some managed restores keep loading data in the background, at reduced performance until done.
• Repointing the application, DNS and caches at the restored copy.

In the 2017 deletion, the only usable copy sat on cheap network disks in a staging environment, throttled to around 60 megabits per second. Copying it back to production took about 18 hours, most of the outage. Nobody had measured it in advance. If the measured restore time is longer than the RTO, either the strategy changes or the RTO does.

Shorter recovery times mean more of the recovery is done in advance. The standard ladder of strategies, cheapest first:

The line between pilot light and warm standby is whether the recovery site can take a request immediately: a warm standby can, at reduced capacity; a pilot light must first start its servers. The standby kinds themselves, and why the idle copy is also untested, are in Redundancy & failure domains; running every region at once is Multi-region replication.

Two warnings apply to every rung:

• Even active/active needs backups. It recovers from a lost region in seconds, but corruption and deletion replicate to every site, so their recovery point is still the last good backup.
• Automatic failover to a recovery region is risky. A false alarm costs real data and downtime, so many teams start failover by hand, with every step after the decision automated so that starting it is one command. Recovery should rely on systems that keep running when the primary site fails, not on the failed site's management tools.

The 2017 outage had several backup methods. One had been failing for some time: its version didn't match the database, and its failure emails were being rejected, so nobody knew. Another had never been turned on for the database servers. The lesson in that postmortem is the one in every disaster-recovery guide: backups exist only once they have been restored.

A disaster-recovery plan is part of a wider business continuity plan. Restoring the payroll service quickly helps only if the bank connections, the support team and the people who approve payments are available too. Write down who decides, how customers are told, and what happens if the person who knows the runbook is unreachable.