Low-level designCore8 min

Thread pools and executors

Instead of starting a thread for every task, hand tasks to an executor that runs them on a small, fixed set of reused threads. The pool caps how many threads exist, and a queue holds the tasks waiting their turn.

Watch it evolve1 / 7
The problem

Fifty welcome emails

Each email gets its own new thread, and main waits for all of them.

80
Start
Stage 1
Final

1 · The situation

Fifty welcome emails

A newsletter service sends a welcome email to every new subscriber. Sending is slow, mostly waiting on the mail server, so the service sends in parallel. The first version starts a new thread for each email and waits for all of them. Mailer.sendWelcome() stands in for the real sending, and counts what it sent.

A thread per emailJava
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;

class Mailer {
    private final AtomicInteger sent = new AtomicInteger();

    void sendWelcome(String user) {
        // Stands in for talking to a mail server.
        String message = "Welcome aboard, " + user + "!";
        if (!message.isEmpty()) sent.incrementAndGet();
    }

    int sent() {
        return sent.get();
    }
}

public class Main {
    public static void main(String[] args) throws InterruptedException {
        Mailer mailer = new Mailer();
        List<Thread> threads = new ArrayList<>();
        for (int i = 1; i <= 50; i++) {
            String user = "user" + i;
            Thread thread = new Thread(() -> mailer.sendWelcome(user));
            thread.start();
            threads.add(thread);
        }
        for (Thread thread : threads) thread.join();
        System.out.println("Emails sent: " + mailer.sent());
        System.out.println("Threads started: " + threads.size());
    }
}
Emails sent: 50
Threads started: 50

2 · The problem

A thread for every task

Fifty threads is fine. The trouble is that the number of threads now equals the number of tasks, and nothing limits either. On a launch day with 50,000 sign-ups in a minute, the service tries to start 50,000 threads:

Cost

Why it grows with every thread

Memory

Each operating-system thread reserves its own stack

Start-up time

Creating and destroying a thread is far slower than calling a method

Switching

More runnable threads than cores means the operating system keeps switching between them

Downstream load

Every thread opens its own connection to the mail server, which has limits of its own

At some point thread creation fails, or the machine spends more time managing threads than sending email. The design has no knob: the load decides how many threads exist.

3 · The fix

A fixed set of threads, reused

A thread pool starts a fixed number of threads once. Tasks go into a queue, and each thread takes the next task when it finishes the last, so the same four threads send all fifty emails:

Mailer.sendWelcome() doesn't change. What changes is who runs it: the loop stops creating threads and hands each email to an executor as a task.

Java
class Mailer {
    private final AtomicInteger sent = new AtomicInteger();

    void sendWelcome(String user) {
        // Stands in for talking to a mail server.
        String message = "Welcome aboard, " + user + "!";
        if (!message.isEmpty()) sent.incrementAndGet();
    }

    int sent() {
        return sent.get();
    }
}
Mailer− sent: AtomicInteger~ sendWelcome(String)~ sent(): int
The whole program, with its poolJava
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.ThreadFactory;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;

class Mailer {
    private final AtomicInteger sent = new AtomicInteger();

    void sendWelcome(String user) {
        // Stands in for talking to a mail server.
        String message = "Welcome aboard, " + user + "!";
        if (!message.isEmpty()) sent.incrementAndGet();
    }

    int sent() {
        return sent.get();
    }
}

class CountingThreads implements ThreadFactory {
    private final AtomicInteger created = new AtomicInteger();

    public Thread newThread(Runnable task) {
        created.incrementAndGet();
        return new Thread(task);
    }

    int created() {
        return created.get();
    }
}

public class Main {
    public static void main(String[] args) throws InterruptedException {
        Mailer mailer = new Mailer();
        CountingThreads threads = new CountingThreads();
        ExecutorService pool = Executors.newFixedThreadPool(4, threads);
        for (int i = 1; i <= 50; i++) {
            String user = "user" + i;
            pool.submit(() -> mailer.sendWelcome(user));
        }
        pool.shutdown();
        pool.awaitTermination(1, TimeUnit.MINUTES);
        System.out.println("Emails sent: " + mailer.sent());
        String started = threads.created() <= 4 ? "no more than 4" : "" + threads.created();
        System.out.println("Threads started: " + started);
    }
}
Emails sent: 50
Threads started: no more than 4

The same fifty emails, sent by at most four threads.

A thread per email
50
A pool of four
4

Threads created for 50 emails. With 50,000 emails, the first grows to 50,000 and the pool stays at 4.

4 · The idea

Executors: tasks in, results out

An executor is anything that runs tasks you submit. A thread pool is the common kind: a fixed set of worker threads and a task queue, which is producer-consumer with the submitting code as producer and the workers as consumers. Submitting usually returns a future, a handle to the task's eventual result, as Futures and async shows.

Workers loop: take a task, run it, take the next. Threads are created once and reused.

Work

How many threads

Why

CPU-bound, such as resizing images

About the number of cores

More threads than cores only adds switching

I/O-bound, such as sending email

More than the number of cores

Threads spend most of their time waiting, so others can use the core

Mixed

Separate pools for each kind

Slow I/O tasks shouldn't block the CPU work

Python's thread pool shows the I/O reasoning in its default: the number of CPUs plus four, at most 32, chosen to keep at least five workers for I/O-bound tasks. In the default CPython build only one thread runs Python code at a time, so CPU-bound work goes to a process pool instead.

5 · When breaking it is fine

Cheap threads, and pools that bite

Pools have their own failure modes:

  • An unbounded queue. Java's fixed pool queues without limit, so a flood of tasks becomes a flood of memory: the same problem as an unbounded producer-consumer queue. Bound it, and choose what happens when it's full.
  • Tasks waiting on tasks. If every pool thread waits for a result from a task still in the queue, no thread is free to run it, and the pool stops.
  • Too small or too large. Too few threads leave cores idle; too many bring back the costs the pool was meant to avoid.

Thread-per-task comes back with virtual threads, in Java 21 and later. They're cheap enough to create one per task, and the guidance is never to pool them: Executors.newVirtualThreadPerTaskExecutor() starts a fresh virtual thread for every task. A limit on a scarce resource, such as mail server connections, is then expressed directly, for example with a semaphore, instead of through the size of a pool.

Situation

Usually

A handful of long-lived background jobs

Plain threads, started once

Many short tasks on platform threads

A fixed pool with a bounded queue

Many I/O-bound tasks in Java 21 or later

A virtual thread per task, with explicit limits on scarce resources

I/O-bound work in JavaScript or Python

Async code on one thread, covered in Futures and async

6 · Getting it right

Size it, bound it, shut it down

  • Shut pools down. A pool's threads keep a program alive until it's shut down; Java's shutdown() and awaitTermination(), Python's with block and the C++ pool's destructor all finish queued work first.
  • Bound the queue and decide what a full queue does: block the submitter, reject, or run the task in the caller.
  • Don't block pool threads on each other. A task that submits more work should not wait for it on the same pool.
  • Give pools names. A thread factory that names threads, such as mail-sender-3, makes thread dumps and logs readable.

Language

Executors

Java

Executors.newFixedThreadPool(), newCachedThreadPool(), newVirtualThreadPerTaskExecutor(); ThreadPoolExecutor for full control

C++

None in the standard library; std::async runs one task, and pools are written by hand or taken from a library

JavaScript

Async functions for I/O; worker threads, often through a pool library, for CPU-bound work

Python

concurrent.futures.ThreadPoolExecutor for I/O, ProcessPoolExecutor for CPU-bound work

7 · Look-alikes

Pools of other things

Pooling is a general idea: keep a few expensive things and lend them out.

Idea

What's pooled

Thread pool

Threads, lent to tasks

Connection pool

Database or network connections, lent to requests

Object pool

Any object that's costly to create, lent and returned

Producer-consumer

Nothing pooled: the queue between submitters and workers inside every thread pool

Bulkhead

Separate pools per kind of work, so one slow kind can't take all the threads

8 · In practice

What the libraries say

Java's newFixedThreadPool() reuses a fixed number of threads working off a shared unbounded queue; tasks submitted while every thread is busy wait in that queue. newCachedThreadPool() instead creates threads as needed and drops ones idle for sixty seconds, which suits many short tasks but has no upper limit.

Read the defaults of whatever pool you use, especially the queue: an unbounded queue in front of a fixed pool moves an overload from threads into memory.

Check yourself

4 questions

1. A service resizes images, which is pure computation, on a machine with 8 cores. About how many threads should its pool have?
2. A fixed pool of 4 threads gets 10,000 tasks at once. Where are the tasks while they wait?
3. Every task in a pool of 4 submits a second task to the same pool and waits for its result. With 4 tasks running, what can happen?
4. In Java 21, why might you start a thread per task again?

Takeaways

Remember this

  • A thread per task lets the load decide how many threads exist; a pool caps them.
  • A thread pool is a fixed set of reused workers taking tasks from a queue: producer-consumer inside.
  • Size pools by the work: about the number of cores for CPU-bound tasks, more for tasks that mostly wait.
  • Bound the task queue, shut pools down, and never let pool tasks block waiting on each other.
  • Java's virtual threads make thread-per-task cheap again; don't pool them, limit scarce resources directly.