System designDeep dive14 min
Lambda vs Kappa architecture
A data system has to answer fast and has to be fixable: when the code that computed last month's numbers had a bug, the numbers must be computed again. Lambda gets both by running a batch system and a stream system side by side. Kappa gets both from one stream job and a log long enough to replay. The choice decides how many copies of your logic you maintain.
1 · The problem
Fresh answers and fixable answers pull in different directions
A power utility has 2 million smart meters. Each sends a reading every 15 minutes, 96 a day, so 192 million readings arrive every day. Customers open an app to see how much electricity they have used today, and each month the same readings decide their bill.
Those two uses want different things from the system that turns readings into usage:
Aspect | "Today so far" in the app | The monthly bill |
|---|---|---|
Needs to be | Minutes old at most | Exactly right |
Natural fit | A stream job, adding each reading as it arrives | A batch job, summing a closed period |
When the code has a bug | Already shown; can't be taken back | Must be recomputed before the bill goes out |
Then a bug is found. The job has been grouping readings by UTC day instead of the customer's local day. The utility runs on Central time, so for months each customer's "today" has reset at 7 p.m. (6 p.m. in winter), and every daily total in the history is shifted by five or six hours.
Fixing the code is easy. Fixing the numbers means running the corrected logic over every reading since the bug began. A stream job only ever saw each reading once, as it passed. So the real question for the architecture is: where does the system recompute from, and with what code? Lambda and Kappa are the two standard answers.
2 · Lambda
Lambda runs a batch layer and a speed layer, and merges them when you ask
The Lambda architecture starts from one rule: keep every input record, unchanged, forever. Readings are facts about a moment; a new reading never makes an old one untrue. So the system only ever adds data to a master dataset, and everything users see is computed from it. Three layers do the work:
The batch layer recomputes everything from the master dataset; the speed layer covers only the hours the last batch run hasn't reached. A query reads both views and combines them.
- Batch layer. A batch job reads the whole master dataset and recomputes each household's daily usage from scratch, writing a batch view (Batch processing). It is slow, hours old by the time it finishes, and simple: every run starts from the raw facts.
- Speed layer. A stream job covers the hours since the last batch run, incrementing counters in a read-write database as readings arrive. That is the realtime view.
- Serving layer. A query reads both views and merges them.
212.4 + 9.6 = 222.0 kWh. When tonight's batch run covers October 15, the realtime view's entries for that day are thrown away.
The design's real strength is correction. Everything the speed layer computes is overwritten by the batch layer within a day, so a mistake there is temporary. The speed layer may even use an approximate algorithm, such as an estimate of unique visitors, because the batch layer replaces it with the exact answer. And the UTC-day bug has a clear repair: fix the batch code and let the next run recompute everything from the untouched readings.
3 · The catch
The same logic written twice drifts apart
Look at where the UTC-day bug lives in a Lambda system. The rule "group readings by the customer's local day" is written once in the batch job and once in the stream job. They usually run on different frameworks, in different styles, built and tested by different people. The fix has to land in both.
Suppose it lands in the batch code first and the stream fix ships a week later. For that week, the customer sees this:
One day, two numbers: the speed layer still counts UTC days, the batch layer counts local days. Every difference between the two codebases shows up as a number that changes overnight.
This is the cost that pushed teams away from Lambda. Keeping two implementations in two complex distributed systems producing the same result is as painful as it sounds, and running and debugging two systems doubles the operational work. Frameworks that compile one program to both systems were tried; they tend to leak, so engineers still need to understand both underneath.
Cost | What it looks like at the utility |
|---|---|
Two codebases | Every rule (local day, meter swaps, estimated readings) written and tested twice |
Drift | Numbers that change when the batch run lands |
Two systems to run | Two clusters, two on-call skill sets, two sets of failures |
Merging logic | Each query must know where the batch view ends and the realtime view starts |
None of this means the ideas were wrong. Keeping the raw input unchanged, and treating every output as something you can recompute, is what made the bug fixable. The question is whether recomputing needs a second system.
4 · Kappa
Kappa keeps the log and replays it through one job
A stream job is not inherently approximate. Stream frameworks once lagged behind batch ones, but nothing prevents a stream job from giving the same guarantees as a batch job (Stream processing covers checkpoints and exactly-once state). If it can, then reprocessing is just the stream job run again over old input. The Kappa architecture drops the batch layer and does exactly that:
- Keep the input log
Retain every reading you might need to reprocess. To be able to fix 90 days, keep at least 90 days.
- Start the fixed job beside the old one
Usage job v2, with the local-day fix, starts reading from the oldest retained reading and writes to a new table, usage_v2. Version 1 keeps serving the app.
- Let it catch up
v2 works through 90 days of readings, then reaches the live end of the log and keeps up like v1.
- Switch the app
Point the app at usage_v2. The switch is one configuration change, and switching back is too.
- Retire the old version
Stop v1 and delete usage_v1.
Two versions of one job run side by side until the new one catches up. There is only ever one codebase; reprocessing happens only when the code changes.
The replay has to be fast, or the fix takes 90 days to arrive. Live, the job handles 192 million readings a day, about 2,222 a second. To work through 90 days, 17.28 billion readings, in 12 hours, v2 must run 180 times faster:
17,280,000,000 ÷ 43,200 s = 400,000 readings a second, 180 times the live rate. The replay runs with far more parallel instances than the live job, then scales down once it has caught up.
Kappa has its own costs:
- Double output storage while usage_v1 and usage_v2 both exist.
- An output database that can take the replay's write rate, roughly 180 times the usual for those 12 hours, without slowing the live reads.
- A log long enough to replay, which is the next section's problem.
5 · The limit
You can only replay what the log still holds
Kappa moves the master dataset from batch storage into the log. A Kafka topic deletes old segments once they pass the topic's retention time or size, so the retention setting becomes the furthest back you can ever fix. The utility has three ways to keep more:
Option | What it keeps | Catch |
|---|---|---|
Long retention on broker disks | Every reading, for the retention period | Months of history on the brokers' own disks |
Tiered storage | Recent segments on brokers, older ones in object storage | Not supported for compacted topics |
Archive in a data lake | Readings copied to files, kept as long as you like | Replay must read files instead of the log |
Tiered storage in Kafka copies each completed log segment to a remote store such as S3 or HDFS. Live consumers read the tail from the brokers, so the local retention can drop to hours while the remote tier holds months. Storage then grows without adding broker memory or CPU.
The same topic, two tiers with two retention periods. Live tail reads are served from the brokers; a replay reads old segments from S3 and then joins the tail on the brokers.
One shortcut doesn't work. A compacted topic keeps at least the last value for each key and discards older ones. That is enough to rebuild a table of current state, such as each meter's latest tariff, but useless for recomputing usage: the readings the fix needs are exactly the ones compaction threw away.
The archive route has a precedent. Uber backfills a streaming pipeline, about nine days and roughly 10 TB, by treating a Hive table as a streaming source in Spark, so the same job code reads the archive. Replaying that much data back into Kafka would have overloaded its Kafka infrastructure and put live consumers behind.
Replay also changes the order of arrival. Live, readings arrive roughly in time order; read back from files, they come in whatever order the files hold. A job using event-time windows waits only as long as its watermark allows (Stream processing), so a watermark tuned for live lateness can drop correct readings as late during a replay. Uber relaxed its watermark from ten seconds to two hours for backfills. Test the replay path, not just the live one.
6 · One codebase
Unified engines run one program on bounded and unbounded input
The deeper fix for Lambda's two codebases is an engine that runs the same program over a closed dataset or an endless stream. Today's major engines are built that way:
Engine | How one program runs both ways |
|---|---|
Apache Flink | One DataStream program, run in BATCH mode on bounded input or STREAMING mode on either |
Apache Beam | A unified model; most transforms apply to bounded and unbounded data alike |
Spark Structured Streaming | A stream is an unbounded table, queried with the same DataFrame API as a static one |
Flink spells out what "the same" means. On bounded input, a program gives the same final results in either mode. A STREAMING run reaches them through incremental updates, like upserts into a table; a BATCH run writes one final result at the end, and can run more efficiently because it knows the input ends. In Flink, set the mode when you submit the job, not in the code, so one program serves both.
For the utility, the rule lives in one function, called by the live job and by the reprocessing job:
One usage rule for live and replay (PySpark, simplified)
from pyspark.sql import functions as F
def daily_usage(readings):
# The fix: group by the customer's local day, not the UTC day
local_day = F.to_date(F.from_utc_timestamp("read_at", "America/Chicago"))
return (readings
.withColumn("day", local_day)
.groupBy("meter_id", "day")
.agg(F.sum("kwh").alias("kwh")))
# Live: the log as an unbounded table (parse and upsert_usage defined elsewhere)
live = parse(spark.readStream.format("kafka")
.option("kafka.bootstrap.servers", "broker:9092")
.option("subscribe", "meter-readings")
.load())
daily_usage(live).writeStream.outputMode("update").foreachBatch(upsert_usage).start()
# Reprocessing: the archived readings, through the same function
archive = spark.read.parquet("s3://meter-archive/readings/")
daily_usage(archive).write.mode("overwrite").saveAsTable("usage_v2")This blurs the old line. A "batch job" can be the stream program pointed at bounded input, and Spark's available-now trigger runs a streaming query over everything available and then stops. What remains of the Lambda vs Kappa debate is not batch versus stream engines, but one codebase or two, and where the history you replay is kept.
7 · In practice
Keep the raw input, write the logic once, and replay to fix
Both architectures keep the raw input and treat every output as recomputable. They differ in what runs the recompute:
Aspect | Lambda | Kappa |
|---|---|---|
Recompute with | A batch system, on every run | The stream job, only when the code changes |
Codebases | Two, kept in sync | One |
History lives in | Batch storage | The log, tiered storage or an archive |
Live mistakes | Overwritten by the next batch run | Stay until a replay fixes them |
Fits when | Live numbers may be approximate and the exact ones come later | One job can meet both latency and correctness |
The practical advice is to pick one: a batch framework if you can wait hours, a stream framework if you can't, and both at once only when you truly must. A Lambda-shaped design still earns its keep when live numbers are deliberately approximate, say an estimated count of unique visitors, and exact ones arrive later. Even then, a unified engine can keep the logic in one place.
For the utility, the checklist:
- Keep every reading for as long as you might need to recompute, in the log, its remote tier or an archive.
- Write each rule once, and run it through the same code live and on replay.
- Version the output: a fix writes usage_v2 beside usage_v1, and the app switches when it has caught up.
- Size the replay: parallel instances for the catch-up rate, and an output database that can take the burst.
- Test the replay path, including arrival order and watermarks.
- Don't count on compacted topics for history; they keep only the latest value per key.
Check yourself
5 questions
Takeaways
Remember this
- Lambda gets fresh and fixable answers by running a batch layer over all the data and a speed layer over the latest hours, merged when a query asks.
- Lambda's price is the same logic in two systems: two codebases that drift and two systems to operate.
- Kappa reprocesses by running the fixed stream job from the start of a retained log into a new table, then switching to it.
- You can only replay what you kept: retain the log, tier it to object storage, or archive it and read the archive with the same code; compacted topics keep no history.
- Unified engines run one program on bounded and unbounded input, so the real choice is one codebase and a replayable history.