System designCore8 min
Batch processing
Some answers don't need to be instant, but they do need to cover everything: every event of the day, every row of the year. Batch processing takes a fixed pile of input, splits the work over many machines, and writes a derived output. Failures are handled by simply running pieces again, which only works if every run gives the same result.
1 · The idea
A batch job turns a fixed pile of input into a derived output
An e-book subscription service pays authors by pages read. Readers' apps send a page-read event for every page turned, about 100 million a day, stored as files in object storage. Each night a job counts pages per book for the previous day; at month end, another sums the days into payouts. Nobody needs the count within a second, but every event must be counted exactly once.
That shape defines batch processing. The input is bounded: yesterday's files, complete and unchanging. The output is derived: it can always be rebuilt from the input, so a wrong output is fixed by fixing the code and running again. And the job is judged by throughput, how long the whole night takes, not by the latency of any one record. Stream processing handles the opposite case, unbounded input with answers in seconds.
Aspect | Batch | Stream |
|---|---|---|
Input | A bounded set: yesterday's files | An endless flow of events |
Answers | Hours later, complete | Seconds later, updated continuously |
Fixing a bug | Rerun the job on the same input | Replay the log, if it's still kept |
Here | Nightly page counts, monthly payouts | A live "trending books" list |
2 · Map and reduce
Map, shuffle, reduce: split the work by input, then regroup it by key
MapReduce gives the basic model. The programmer writes two functions. Map reads one record and emits key-value pairs: for a page-read event, (book_id, 1). Reduce receives one key with all of its values and combines them: the sum of pages for a book. The engine does everything in between, on as many machines as the input needs:
Map tasks split the work by input file. The shuffle regroups the pairs by key, so each reduce task receives every value for its books, whichever map task produced them.
The shuffle is the step that moves data between machines: each map task writes its pairs to local disk, divided into R partitions by a hash of the key, and each reduce task fetches its partition from every map task. The shuffle is the costly step, since it copies data across executors and machines. Designing a batch job is mostly about how much crosses it.
3 · Less shuffle
Shrink what crosses the shuffle by combining early
Popularity is uneven: yesterday the top bestseller got 5 million page reads. Without help, each of those becomes a (book, 1) pair sent over the network to one reduce task. The fix is a combiner: when the reduce function is commutative and associative, as a sum is, each map task can pre-add its own pairs before sending them. Every map task then sends one partial count per book:
Records sent to the bestseller's reduce task | Count |
|---|---|
Without a combiner, one per page read | 5,000,000 |
With a combiner, one partial sum per map task | 400, 12,500 times fewer |
Spark builds the same idea into its operators. In Spark, reduceByKey or aggregateByKey beat groupByKey for sums and averages, because they combine values on the map side, before the shuffle. Joins follow the same logic: Spark SQL by default sends a copy of any table under 10 MB to every worker, so the large side of the join never has to be shuffled.
4 · Failures
Failures are handled by running the lost work again
A job of hundreds of tasks on hundreds of machines will meet failures. Batch engines don't try to make each task survive; they rerun it. MapReduce uses re-execution as its main fault-tolerance mechanism: if a worker dies, its tasks are scheduled again elsewhere. Spark does the same at a finer grain. It records how each dataset was derived, its lineage, and recomputes only the lost partitions:
The half-written output of the first attempt is ignored. Rerunning is safe because the input file is unchanged and the map function gives the same output every time.
This relies on two things. Tasks must be deterministic, so a rerun produces the same output. And output must appear all at once or not at all: MapReduce writes each reduce output to a temporary file and renames it into place, relying on the file system's atomic rename, so two attempts of the same task can't both leave data behind.
Slow machines get the same treatment. A straggler, such as a machine with a failing disk, can hold up the last few tasks of a job. Near the end, MapReduce starts backup copies of the remaining tasks and takes whichever finishes first. Without backup tasks, a large sort can take nearly half again as long.
5 · Skew
One hot key can make one task the whole job
The shuffle sends all values of a key to one reduce task. When one key dominates, that task gets far more work than the rest, and the job finishes only when it does. The combiner fixes this for sums, but not for operations that need every record of the key, such as joining each of the bestseller's 5 million events to reader data.
Spark's adaptive query execution handles common cases at runtime: it detects oversized shuffle partitions and splits them, replicating the other side of a join where needed, into tasks of roughly even size. For skew that the engine can't see, the job can split a hot key by hand, for example into 20 sub-keys processed in parallel and then combined (Hot partitions & skew).
The nightly job in PySpark
day = "2026-10-01" # the run's data interval, never today's date
events = spark.read.parquet(f"s3://reads/events/date={day}/")
pages = (events
.groupBy("book_id")
.agg(F.count("*").alias("pages"))) # pages read per book
(pages.write
.mode("overwrite")
.parquet(f"s3://reads/pages_per_book/date={day}/"))6 · Reruns
Make every run of a job safe to repeat
Whole jobs get rerun too: after a bug fix, after an outage, or when a new report needs last year's history. A scheduler such as Airflow gives each run a data interval, the period of input it covers, and starts it only after that interval has ended, so all its data has arrived. Running a job over past intervals is called a backfill.
Airflow's best practices spell out what makes a rerun safe, and they apply to any batch system:
- Read a fixed partition, such as the run's date, never "the latest data", which may change between runs.
- Write to a partition and replace it, or upsert; a plain insert on a rerun duplicates rows.
- Never use the current time in the computation, since each run would see a different one.
The nightly job above follows all three: it reads date=2026-10-01 and overwrites that output folder, so running it five times gives the same counts as running it once.
- Keep the raw input
Immutable, partitioned by date; every output can be rebuilt from it.
- Combine before the shuffle
Pre-aggregate on each machine; broadcast small join sides.
- Watch for skew
Look at the slowest task, not the average.
- Make outputs replaceable
Write a whole partition at once; overwrite on rerun.
- Schedule by data interval
Run after the interval closes, and backfill with the same code.
Check yourself
4 questions
Takeaways
Remember this
- Batch jobs process bounded input into derived output, judged by total throughput rather than per-record latency.
- Map splits work by input, the shuffle regroups by key, and reduce combines each key's values.
- The shuffle is the costly step: combine early, and broadcast small join inputs.
- Failures and stragglers are handled by rerunning tasks, which requires deterministic tasks and atomic outputs.
- Make whole runs repeatable: read a fixed data interval, overwrite its output, never use the current time.