System designCore10 min

Data lifecycle: tiering, TTL & compression

Most data is read heavily for a few days and then almost never, but it keeps costing money every month until someone deletes it. A lifecycle decides, for each kind of data, where it lives as it ages, how small it is stored, and when it goes away, and then lets rules, TTLs and partition drops do the work.

1 · The idea

Every kind of data has a lifecycle, and most reads happen early

A home security camera service has 500,000 cameras. Each records about 20 motion clips a day of 3 MB, so 30 TB of video arrives daily, and owners can scroll back 30 days. Almost every replay happens within a day or two of the clip, when someone checks what set off the alert. After that the clips sit untouched, costing the same as on day one.

Clips aren't the only data, and each kind ages differently. Writing down, per dataset, how it is read over time and when it can go is the start of a lifecycle:

Data	Read mostly	Kept for	Ends by
Motion clips, 30 TB a day	First two days	30 days	An object storage lifecycle rule
Clip index rows, 10 million a day	With the clips	30 days	Dropping a daily partition
Camera logs	During an incident, within days	14 days	Kafka retention, then a search index policy
Shared clip links	Within hours	24 hours	A TTL on each item

The levers are the same everywhere: move aging data to cheaper storage, store it smaller, and delete it on schedule. Deletion is the one that stops the bill from growing forever (Object & blob storage).

2 · Tiers

Colder storage is cheaper to keep and dearer to read

Object stores sell the same durability at several prices. Google Cloud Storage's classes, for example, cut the monthly price per gigabyte in steps, and charge for it in two other ways: a fee for every gigabyte read back, and a minimum storage duration. Deleting an object earlier still bills it for the minimum:

Class	Per GB-month (Iowa)	Per GB read	Minimum duration
Standard	$0.020	Free	None
Nearline	$0.010	$0.01	30 days
Coldline	$0.004	$0.02	90 days
Archive	$0.0012	$0.05	365 days

Nearline fits data read about once a month, Coldline once a quarter, and Archive less than once a year. When a lifecycle rule changes the class, the days already spent count toward the new class's minimum. For the camera service, that changes which class wins:

All 30 days in Standard: 18,000 USD
Nearline from day 3: 9,900 USD
Coldline from day 3: 12,240 USD

Monthly cost at steady state, 900 TB of clips. Nearline: 90 TB × $20 + 810 TB × $10 per TB. Coldline is cheaper per gigabyte, but clips deleted at day 30 are still billed to day 90, adding $7,200.

Replays after day 3 pay the read fee: if owners replay 1% of the older clips, that's 8.1 TB × $10 = $81 a month, small next to the saving. Data read often should never be tiered this way, because the read fees grow with every access.

3 · Rules

Let rules move and delete data by age, and expect them to run late

Nobody should run a nightly script that lists 300 million clips. Object stores apply lifecycle rules themselves: conditions such as age, prefix or current class, and an action, either change the class or delete:

Cloud Storage lifecycle for the clips bucket

json

{
  "lifecycle": {
    "rule": [
      {
        "action": {"type": "SetStorageClass", "storageClass": "NEARLINE"},
        "condition": {"age": 3, "matchesStorageClass": ["STANDARD"]}
      },
      {
        "action": {"type": "Delete"},
        "condition": {"age": 30}
      }
    ]
  }
}

Rules are asynchronous. Lifecycle actions can lag behind their conditions, and in Cloud Storage a changed configuration can take up to 24 hours to apply, so apps shouldn't depend on the timing. They also have per-object costs: S3, by default, doesn't transition objects smaller than 128 KB, because the per-object transition charge can exceed the saving. When access is unpredictable, S3 Intelligent-Tiering moves each object itself, to a cheaper tier after 30 days without access and a cheaper one again after 90.

Search indexes have the same idea. Elasticsearch's index lifecycle management moves each index through phases by age, rolling over to a new index whenever the current one reaches a set age or size. The camera logs use three:

Elasticsearch defines hot, warm, cold, frozen and delete phases. Each phase can make the index read-only, merge its segments, move it to cheaper nodes or snapshots, or delete it.

4 · TTL

A TTL deletes items for you, but not at the moment they expire

Databases can expire individual items. Owners share a clip with a link that works for 24 hours; each link is a row with an expiry time, and the database removes it later. How much later varies a lot between systems:

Database	How expiry works	Expired items linger
DynamoDB	A per-item timestamp; deletes use no write capacity	Typically up to a few days
MongoDB	A TTL index; a background task runs every 60 seconds	A minute or more under load
Cassandra	The value turns into a tombstone, a deletion marker	Tombstones stay at least 10 days by default

So the TTL is cleanup, not the rule itself. The link service checks the expiry time on every read and refuses an expired link, whether or not the row is still there. So filter expired items out of queries.

Cassandra shows why deletes are hard in a replicated database. A delete is a write: a tombstone that must reach every replica, and it is kept for the grace period, gc_grace_seconds, ten days by default, so a replica that was down can't bring the old value back. Data that expires in large volumes leaves many tombstones, which stay on disk until the grace period has passed and a compaction removes them (Storage engines: B-trees, LSM trees & WAL).

5 · Bulk deletes

Delete in bulk by dropping whole partitions, not row by row

The clip index gets 10 million rows a day. Deleting the 31st day back with DELETE ... WHERE recorded_at < ... touches 10 million rows, writes all of that to the log, and leaves dead rows for VACUUM to clean up. If the table is partitioned by day, the same cleanup is one quick statement:

One partition per day; retention is a DROP

sql

CREATE TABLE clip_events (
  camera_id   bigint,
  recorded_at timestamptz,
  label       text
) PARTITION BY RANGE (recorded_at);

CREATE TABLE clip_events_2026_10_03 PARTITION OF clip_events
  FOR VALUES FROM ('2026-10-03') TO ('2026-10-04');

-- 30 days later: the whole day disappears at once
DROP TABLE clip_events_2026_09_02;

A daily job keeps the window moving: add tomorrow's partition, drop the one that just turned 31 days old. Queries for recent clips only touch the partitions their time range covers.

In PostgreSQL, dropping a partition is far faster than a bulk delete and avoids the VACUUM overhead entirely. DETACH PARTITION is the gentler variant: the day leaves the table but stays readable, for a final backup or a summary, before it's dropped. A job creates tomorrow's partition ahead of time and drops the oldest one each day (Time-series databases does this with chunks).

Kafka is built the same way. Each topic partition is a series of segment files, and retention deletes whole segments once they're older than retention.ms, 7 days by default. The camera logs topic keeps the default; the search index keeps 14 days.

6 · Compression

Compression trades CPU for space, and the codec is a dial

Text-like data such as logs and index rows compresses well; the clips don't, because video codecs have already removed the redundancy (Video streaming: transcoding & adaptive bitrate). For the rest, the choice of codec is a trade between how small, how fast to write, and how fast to read. The zstd project's benchmark on a standard mixed corpus, on one desktop CPU core:

Codec	Size reduction	Compress	Decompress
zstd, level 1	2.90×	510 MB/s	1,550 MB/s
zlib, level 1	2.74×	105 MB/s	390 MB/s
lz4	2.10×	675 MB/s	3,850 MB/s
snappy	2.09×	520 MB/s	1,500 MB/s

zstd at level 1 compresses better than zlib, about five times faster. lz4 gives up some size for the fastest reads, which suits data read constantly. zstd's higher levels shrink data further at much slower compression, while decompression stays about as fast, a good fit for data written once and kept for months. Your data's ratio will differ from this corpus, so measure on a sample.

Uncompressed: 28 TB
lz4: 13.3 TB
zstd level 1: 9.7 TB

14 days of camera logs at 2 TB a day, if they compress like the benchmark corpus. Kafka topics can store messages with gzip, snappy, lz4 or zstd.

Where data is stored by column, compression goes further, because similar values sit next to each other (OLTP vs OLAP & columnar storage). Metrics can also shrink by losing detail on purpose, keeping per-minute averages instead of every point.

7 · In practice

Write the lifecycle down for each dataset, then automate it

A lifecycle is mostly a decision, made once per dataset, and then machinery that carries it out without anyone remembering to. The legal side, what you must keep and what you must erase, comes from privacy and retention requirements (Privacy, residency & retention).

List datasets and their reads by age
How often each is read on day 1, day 7, day 30.
Pick tiers with the fees in mind
Minimum durations and read fees can make the cheaper class cost more.
Make expiry a filter
Check expiry times on read; TTLs and rules clean up later.
Partition by time
Retention becomes dropping a partition, a segment or an index.
Compress with a chosen codec
lz4 for hot reads, zstd for data kept long; skip already-compressed media.
Expire every copy
Replicas, caches and search indexes need the same retention as the source.

Check yourself

4 questions

1. Clips are deleted after 30 days. Why can moving them to Coldline on day 3 cost more than Nearline, though Coldline is cheaper per gigabyte?

2. A shared link has a DynamoDB TTL of 24 hours. What should the link service do when it reads a link that is 26 hours old?

3. Why is dropping a daily partition better than DELETE for removing 10 million old rows in PostgreSQL?

4. Which codec fits logs written once and kept for months but rarely read?

Takeaways

Remember this

Decide per dataset how it is read as it ages and when it can be deleted; deletion is what stops storage from growing forever.
Colder storage classes cost less to keep but charge for reads and for a minimum duration, which can make them cost more.
Lifecycle rules, index lifecycle policies and TTLs automate moving and deleting, but run late, so check expiry on read.
Partition by time so retention drops a whole partition or segment instead of deleting rows one by one.
Compression trades CPU for space: lz4 for fast reads, zstd for long-term data, nothing for already-compressed media.