DDoS protection & WAFs

A regional news site expects its biggest night of the year: election results. It also expects attacks, because some people would rather its results page were down. A distributed denial-of-service (DDoS) attack sends traffic from many machines at once, often a botnet of hijacked devices, so no single source can simply be blocked. Its aim is to use up some resource the site needs:

The scale is the hard part. In 2018 GitHub was hit by 1.35 terabits per second of reflected traffic. In 2023 a new HTTP/2 technique produced attacks peaking at 201 million requests a second at Cloudflare, from a botnet of only about 20,000 machines, and 398 million a second at Google. No single site's servers or links come close to absorbing that. Defense has to happen in networks built for it, layer by layer.

UDP doesn't check the source address of a packet, so an attacker can send a small request to some server on the internet while forging the victim's address as the sender. The server sends its much larger answer to the victim. This is a reflection attack, and when the answer is larger than the request, an amplification attack. The attacker spends a little bandwidth; thousands of innocent servers spend a lot of it on the victim.

Memcached is the extreme: Cloudflare observed a 15-byte request producing a 750 KB response, a factor of 50,000. That vector produced GitHub's 1.35 Tbps attack. GitHub's monitoring saw inbound bandwidth pass 100 Gbps at one facility, and the team rerouted its traffic through a DDoS mitigation provider by changing its BGP announcements. The site was unavailable for about five minutes and intermittently for four more.

The defense for volume is more capacity than the attacker, which means a provider with a network built to absorb it: A large edge network can absorb volumetric attacks that reach terabits per second. Anycast spreads a global botnet's traffic across every site, so each absorbs only its share (Anycast & global traffic routing). The news site's own job is to make sure the attack can't go around: its origin must accept traffic only from the provider.

A TCP connection starts with a three-way handshake (How data moves across a network). When a server receives a SYN, it normally stores a small record for the half-open connection and replies with a SYN-ACK, waiting for the client's final ACK. A SYN flood sends SYNs with forged sources that never complete the handshake. The server's backlog of half-open connections fills, and real users' SYNs are dropped.

SYN cookies remove the state that the flood fills. Instead of storing the half-open connection, the server encodes the essential details into the sequence number of its SYN-ACK. A real client's ACK carries that number back, and the server rebuilds the connection from it; a forged SYN never returns an ACK and costs the server nothing to remember:

Edge services do this at scale: AWS's edge network verifies incoming connections with SYN cookies before passing them on, so only completed handshakes reach the application. Other protocol attacks target TLS handshakes or firewall state tables; the common defense is the same idea, terminating connections at an edge large enough to keep the state.

Layer 7 attacks send complete, valid HTTP requests. Each one passes the network and handshake defenses; the damage is the work it causes. On election night, a botnet requesting the results search page, uncached and backed by a database query, can overload the site at a rate the network barely notices.

HTTP/2's Rapid Reset showed how cheap such requests can be. HTTP/2 limits how many streams a client may have open at once, but a stream the client cancels with a RST_STREAM frame no longer counts. Attackers opened a stream, canceled it at once, and opened another, so each connection sent requests as fast as it could while staying within the limit, and servers kept doing the work for requests already abandoned:

Defenses at this layer are about telling attackers from readers:

• Rate limits per client, counting requests per IP or other key over a window and blocking or challenging above it (Rate limiting algorithms).
• Caching, so a flood of identical requests is answered at the edge and never reaches the database (Content delivery networks (CDNs)). Election results can be cached for a few seconds and still be fresh.
• Challenges for suspicious clients: a check before the request is served, applied only to traffic that looks wrong.
• Shedding the least important work when overloaded, so the results page stays up even if search doesn't (Load shedding & backpressure).

A DDoS attack overwhelms; an application attack sneaks. A web application firewall (WAF) sits in front of the application, often at the same edge, and inspects each HTTP request for attack patterns such as SQL injection in a form field or a script in a comment. It complements input validation in the application; it doesn't replace it.

The widely used OWASP Core Rule Set shows how a WAF decides without blocking too much. Rules don't block on their own. Each matching rule adds to the request's anomaly score according to its severity, and the request is blocked only when the total reaches a threshold, by default 5, the score of one critical match. Raising the threshold lets many attacks through, so it's done only temporarily while tuning.

Each level includes the ones below it, and every step up trades false alarms for coverage. False alarms are fixed with rule exclusions for specific fields, such as allowing HTML in the news site's article editor. A practical rollout runs new rules in a detection-only mode first, reviewing what they would have blocked, before letting them block. A WAF rule is also code running on every request: a bad one can take a site down on its own (Edge computing describes one that did).

Most DDoS protection is decided before any attack, by where traffic can enter. A DDoS-resilient architecture starts from the edge: serve through services that run on a global edge network and reduce the attack surface behind them. For the news site that means:

Autoscaling is for real surges in readers (Autoscaling); attack traffic should be stopped at the edge before it reaches anything that scales, so scale for the expected readers and let the edge absorb the rest.

For the news site, the checklist:

• Serve everything through a protection provider's edge, with anycast to spread attacks.
• Lock the origin to the provider's addresses.
• Cache what can be cached, even for seconds.
• Rate-limit expensive endpoints per client, and know how to tighten limits quickly.
• Run the WAF at a paranoia level you can tune, in detection mode before blocking, with exclusions for real false alarms.
• Roll out WAF and rate-limit rules in stages, like code.
• Rehearse the attack night: who to call, which switches to flip, what to shed.