Rate Limits That Lock Out the Wrong User

The Bystander Gets Locked Out

We put a login limit on nginx, Caddy, and Traefik the way the guides show it: 5 attempts a minute per client. All three proxies sat behind a CDN. An attacker sent 20 logins and got stopped at 5, as designed. Then an honest user, one second later, tried 3 logins and got 0 through. On all three proxies. Every client behind a CDN arrives from the CDN’s address, so every client shares one bucket.

The common fix is to trust X-Forwarded-For so the proxy can see the real client. That fixed the bystander, and it also let a forger choose their own bucket. A client that skipped the CDN and sent a fresh fake header on each request got 20 of 20 through on all three proxies. Of the four setups we tested, only one passed every test on every proxy: trust exactly your CDN’s address, and read the header from the right.

This is Part 4 of a series. What Your Reverse Proxy Leaks by Default covered banners and catch-all servers. Reverse Proxy TLS: What the A+ Hides covered ciphers and HSTS. Security Headers That Break Your Apps covered the headers. The baseline here is where Part 3 stopped.

Full example: Clone the working files at github.com/KingPin/sumguy-examples/security/reverse-proxy-hardening/04-rate-limits

How We Tested

One machine, one Docker network, five curl clients with fixed addresses. A small nginx container plays the CDN. It terminates the client connection and forwards to the proxy under test, appending the client’s address to X-Forwarded-For, which is what Cloudflare does. Behind it sit nginx 1.31.8, Caddy 2.11.9, and Traefik v3.7.17, each in front of a Python backend.

The limits are the same on all three: 5 logins a minute per client with room for 5 at once, and 10 API calls a second with room for 20. Each test starts from a restarted proxy. “Got through” means the app answered: a 401 for /login (the password is always wrong) or a 200 for the API.

We ran four modes. Baseline has no limits. Trap is the guide-style config keyed on the TCP peer. Trusting is the trap plus “believe X-Forwarded-For from everyone”. Hardened is the fix. The limits are in the box at the top. The big ones: nothing ran against a real CDN, and each mode ran once.

The Scorecard

Each cell reads nginx / Caddy / Traefik, counting requests that reached the app.

Lower is better on the attacker and spoof rows. Higher is better on the bystander row. Only the hardened column is good on every row, for all three proxies. The 6 on nginx is its leaky bucket, which lets the first request plus a burst of 5 through. Traefik’s counts moved between runs: a second hardened run let 6 of 20 through on the attacker and direct-spoof rows, all inside 10 ms, so read Traefik’s 5 as about 5. nginx and Caddy repeated exactly. In baseline, the app saw the CDN’s address (172.31.40.10) as the client on all three proxies.

The Trap: One Bucket for Everyone

The trap configs key on the TCP peer. In nginx that is $binary_remote_addr. In Caddy’s plugin README example it is {remote_host}. In Traefik it is the default, the request’s remote address. Behind a CDN, the peer is the CDN.

The attacker got 5 through on all three. That looks like success. The bystander got 0 of 3, because the attacker had already spent the bucket that every user shares. The login page now locks out everyone while one person guesses passwords. The limit works as a denial of service against your own users.

nginx: burst without nodelay

The nginx trap uses burst=5 with no nodelay, which is the shape of the example in nginx’s own limit_req docs. That queues the burst instead of answering it. The attacker’s 5 queued logins came back at 12.06 s, 24.13 s, 36.19 s, and 48.25 s. The fifth hit 60 s and got a 504 from the CDN, whose proxy_read_timeout default is 60 s. The other 14 rejections were 503s, because limit_req_status defaults to 503.

On the API test, the 30-call page load got 21 through with the slowest at 2.00 s. Those are the queued requests. A page that fires 30 calls waits two seconds for the tail.

A one-off probe, not part of the kit’s scripted run: Part 3’s config has error_page 500 502 503 504 @error, so a rejected user saw “503 Service Temporarily Unavailable” with the body “Something went wrong.” That reads as an outage, not a rate limit. A user told that the site is down will retry, which is the opposite of what you want.

Caddy: not in the box

Stock Caddy 2.11.9 refuses the config with unrecognized directive: rate_limit. Caddy has no built-in rate limiting. You need the third-party mholt/caddy-ratelimit module, built in with xcaddy. The kit’s Dockerfile.caddy does that. The module’s own README example keys on {remote_host}, so the copy-paste path lands on the trap.

Caddy’s 429 in trap mode carried Retry-After: 59, an empty body, and none of the Part 3 headers. No HSTS, no nosniff. The module returns an error, and error routes skip the site’s header directives.

Traefik: the default is the trap

Traefik’s rateLimit middleware, with no sourceCriterion, keys on the request’s remote address. That was the CDN. The 429 carried Retry-After 11 to 12 and an X-Retry-In header, and the Part 3 headers were present.

The Common Fix That Isn’t

The usual next step is a snippet that tells the proxy to believe forwarded headers. We wrote the lazy version on each proxy:

• nginx: set_real_ip_from 0.0.0.0/0; with real_ip_recursive on;
• Caddy: trusted_proxies static 0.0.0.0/0 with key {client_ip}
• Traefik: forwardedHeaders.insecure=true with ipStrategy.depth: 1

The bystander problem went away: 3 of 3 on all three proxies. On nginx, the honest user’s logins came back at 12.06 s and 24.13 s, because the burst queue still delays them. Then the forger arrived.

client-c skipped the CDN and sent a new X-Forwarded-For on every request. Each fake address got its own fresh bucket. 20 of 20 got through on all three. The 0.0.0.0/0 trust list says any sender may tell the proxy who the client is. A password-guessing script needs one extra header line to turn your rate limit off.

client-d went through the CDN with a fake entry on the left of the header. The CDN appends the real address on the right, so the header arrives as fake, real. Which entry the proxy trusts decides the result:

• nginx: 20 of 20 got through.
• Caddy: 20 of 20 got through.
• Traefik: held at about 5. With depth: 1 it reads the rightmost entry, the one the CDN appended.

Traefik’s depth counts from the right, so a fake entry on the left never reaches it. The direct spoof still beat it: client-c’s request had no CDN in the path, so its fake address was the only entry in the header, and depth: 1 picked it.

The Fix: Trust Exactly Your CDN

The hardened run changes three things in each proxy: trust only the CDN’s address, read X-Forwarded-For from the right, and give the 429 a proper reply. Every test passed on all three. The attacker was limited (6 / 5 / 5 of 20), the bystander got 3 of 3, both spoofs were limited (6 / 5 / 5), and the API burst got 21 / 20 / 20 of 30 with no queueing delay. On nginx the app received only the real client, 172.31.40.102. On Caddy and Traefik it received 172.31.40.102, 172.31.40.10 in X-Forwarded-For, real client first (more on Traefik’s X-Real-Ip below).

In the configs below, 172.31.40.10 is the lab’s CDN. Replace it with your CDN’s address or ranges.

nginx

# ... Part 1 to 3 lines unchanged ...

# Believe X-Forwarded-For only when the CDN sent the request. With real_ip_recursive on, nginx walks the
# header right to left past trusted addresses and stops at the first one it does not trust: the client.
# Behind Cloudflare: list its published ranges here and use real_ip_header CF-Connecting-IP instead.
set_real_ip_from 172.31.40.10;
real_ip_header X-Forwarded-For;
real_ip_recursive on;

# $binary_remote_addr is now the client, not the CDN.
limit_req_zone $binary_remote_addr zone=login:10m rate=5r/m;
limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
# The default is 503, the same code a dead backend gets. 429 tells the client to slow down.
limit_req_status 429;
# Retry-After for the 429 page: a login slot frees every 12 s at 5r/m, an API slot every 0.1 s.
map $uri $retry_after {
    /login  12;
    default 1;
}

server {
    # ... Part 1 to 3 lines unchanged ...
    error_page 500 502 503 504 @error;
    # Without this, a 429 is nginx's stock HTML page, with "nginx" in the body (the Part 1 leak again).
    error_page 429 @ratelimited;

    location = /login {
        # nodelay: the 5 extra attempts are answered now instead of queued 12 s apart
        limit_req zone=login burst=5 nodelay;
        proxy_pass http://backend:8080;
    }

    location /api/ {
        limit_req zone=api burst=20 nodelay;
        proxy_pass http://backend:8080;
    }

    location @ratelimited {
        default_type text/plain;
        add_header Retry-After $retry_after always;
        return 429 "Too many requests.\n";
    }
}

Without the custom page, nginx’s 429 was its stock HTML page with “nginx” in the body, which is the Part 1 leak again. The hardened 429 is text/plain with Retry-After: 12 and all the Part 3 headers.

Caddy

{
  servers {
    # Believe X-Forwarded-For only when the CDN sent the request.
    # Behind Cloudflare: list its ranges (or the caddy-cloudflare-ip module) and use client_ip_headers CF-Connecting-IP.
    trusted_proxies static 172.31.40.10/32
    # Read X-Forwarded-For right to left and stop at the first untrusted address. Without this,
    # Caddy takes the leftmost entry, which the client wrote itself.
    trusted_proxies_strict
    client_ip_headers X-Forwarded-For
  }
}

# Needs a Caddy built with github.com/mholt/caddy-ratelimit (Dockerfile.caddy). Stock Caddy rejects rate_limit.
# Part 3 headers as a snippet, so the 429 error route below can import them too.
(security_headers) {
  header {
    >Strict-Transport-Security "max-age=63072000"
    >X-Content-Type-Options "nosniff"
    ?Referrer-Policy "strict-origin-when-cross-origin"
    +Content-Security-Policy "frame-ancestors 'self'"
  }
}

app.example.test {
  # ... Part 1 to 3 lines unchanged ...
  import security_headers

  rate_limit {
    zone login {
      match {
        method POST
        path /login
      }
      # {client_ip} is the address trusted_proxies resolved; {remote_host} would be the CDN
      key {client_ip}
      events 5
      window 1m
    }
    zone api {
      match {
        path /api/*
      }
      key {client_ip}
      events 20
      window 2s
    }
  }

  # ... reverse_proxy unchanged ...

  # rate_limit answers with an error, and error routes skip the site's header directives.
  # Without this block a 429 carries Retry-After and none of the Part 3 headers.
  handle_errors 429 {
    import security_headers
    header -Server
    respond "Too many requests." 429
  }
}

The hardened 429 is text/plain with Retry-After: 59 and all the Part 3 headers. Moving the headers into a snippet is the fix for the empty-body, no-headers 429 from the trap.

Traefik

http:
  middlewares:
    # ... Part 1 to 3 middlewares unchanged ...
    login-limit:
      rateLimit:
        average: 5
        period: 1m
        burst: 5
        sourceCriterion:
          ipStrategy:
            depth: 1   # rightmost X-Forwarded-For entry: the address the CDN appended
    api-limit:
      rateLimit:
        average: 10
        period: 1s
        burst: 20
        sourceCriterion:
          ipStrategy:
            depth: 1   # rightmost X-Forwarded-For entry: the address the CDN appended

The trust list lives on the entrypoint, not in the middleware:

- --entrypoints.websecure.forwardedHeaders.trustedIPs=172.31.40.10/32

The hardened 429 says “Too Many Requests” with Retry-After: 11, an X-Retry-In header, and the Part 3 headers.

One Line Off

We ran two variants of the hardened config with one line removed each. Both show what the line is for.

Caddy without trusted_proxies_strict. The CDN spoof got 20 of 20 through. Caddy’s docs say trusted_proxies parses client_ip_headers left to right by default, and the first untrusted IP becomes the client. A forger writes the left side. trusted_proxies_strict (since v2.8) parses right to left, and the docs recommend it behind CDNs that append to the right. Every other result in that variant was unchanged, so the bystander was fine, the direct spoof was limited to 5, and only the one test failed.

Traefik without forwardedHeaders.trustedIPs. The bystander went back to 0 of 3. The CDN sent X-Forwarded-For: 172.31.40.102, and the app received only 172.31.40.10, the CDN’s own address that Traefik appended. Traefik discarded the header from a sender it does not trust. Traefik’s docs say the rate limiter runs before that append, and that a depth greater than the number of IPs in the header gives an empty client IP. In our run every client shared one bucket, and nothing logged an error or a warning.

Both failures are quiet. The config loads, the limit works against a single attacker, and the first symptom is a support email from someone who could not log in.

Behind Cloudflare

We did not test against Cloudflare. The documented approach is to trust its published ranges (the lists are at cloudflare.com/ips) and read the CF-Connecting-IP header: in nginx, real_ip_header CF-Connecting-IP;, and in Caddy, client_ip_headers CF-Connecting-IP. The kit’s comments mark where those lines go. Keep the ranges current. When Cloudflare adds a range you have not listed, requests from it are keyed on the Cloudflare edge address, and those visitors share one bucket again.

One more observation: Traefik’s X-Real-Ip stayed 172.31.40.10, the CDN, even with trustedIPs set. An app that reads X-Real-Ip behind Traefik gets the CDN’s address, not the client’s. Have the app read X-Forwarded-For, or check what your app reads before you trust it.

Tune Before You Enforce

nginx has limit_req_dry_run on; (since 1.17.1). It counts requests as if the limit applied but rejects none, so you can watch the log on a live site and pick numbers before anyone gets a 429. Our tests used limits of 5 a minute and 10 a second because they make the counts easy to read, not because they suit your app. For the layered picture (fail2ban, firewall limits, CrowdSec), see DDoS Mitigation: Teaching Your Server to Say No Politely (Then Impolitely). For limits at the CDN edge itself, see Cloudflare WAF: Free Tier Firewall Rules.

Run It Against Your Own Proxy

You do not need our kit to check your own setup. You need two machines or two source addresses.

# 1. From a client that skips the CDN, send a fake client address on every request.
for i in $(seq 1 20); do
  curl -s -o /dev/null -w "%{http_code}\n" -X POST \
    -H "X-Forwarded-For: 203.0.113.$i" https://your.domain.example/login
done | sort | uniq -c

# 2. From a second address, try 3 logins right after step 1.

Read the result like this:

1. Twenty 401s in step 1 means there is no login limit, or it trusts a header any client can write. If step 1 cannot connect at all, your origin only accepts the CDN, which is the best result.
2. A mix of 401s and 429s in step 1 means the forger did not pick the bucket.
3. A 429 in step 2 means your limit shares a bucket between users.
4. A 503 means the limit still uses nginx’s default status. A 504, or a reply that takes 12 seconds or more, means the burst is queued instead of answered.

The kit’s run.sh runs the whole lab in Docker if you want the full five-test table.

The SumGuy Take

A rate limit is only as good as its key. The examples in nginx’s docs and in the caddy-ratelimit README key on the TCP peer, and the key is where all three trap configs failed. Behind a CDN, the default key is the CDN. The popular fix lets the client choose its own key. The working answer is boring: trust exactly one address, read the header from the right, and return a 429 that looks like a 429.

The practical order for a home lab:

1. Find out what your app sees as the client address. Send one request and look at the logs.
2. Set the trust list to your CDN’s address or published ranges, never 0.0.0.0/0.
3. On nginx, set real_ip_recursive on, limit_req_status 429, and nodelay on every burst.
4. On Caddy, build in the rate-limit module, then use trusted_proxies_strict and {client_ip}, and import your header snippet in handle_errors 429.
5. On Traefik, set trustedIPs on the entrypoint and ipStrategy.depth: 1 in the limiter.
6. Run the forger test from the section above. If 20 of 20 get through, your limit is a suggestion.

What Comes Next

The rest of the series:

1. Part 5: hiding admin paths, including the Traefik dashboard, the Caddy admin API, and forward auth.

Common Questions

Does Caddy have built-in rate limiting?

No. Stock Caddy 2.11.9 rejects a rate_limit directive with “unrecognized directive”. You need a third-party module such as mholt/caddy-ratelimit, built into the binary with xcaddy. Caddy’s trusted_proxies option and {client_ip} placeholder are built in, so the module can key on the real client address.

Why does my rate limit block every user at once?

Your proxy keys the limit on the TCP peer, and behind a CDN the peer is the CDN. Every visitor shares that one address and one bucket, so a single attacker spends the limit for everyone. In our run the bystander got 0 of 3 logins through. Key on the real client address instead.

Should a rate limit return 429 or 503?

Return 429. The status code 429 tells a client to slow down, and 503 reads as an outage. nginx defaults limit_req_status to 503, and Part 3’s error page then turned that into “Something went wrong.” Set limit_req_status 429 and add a Retry-After header so clients know when to retry.

How do I get the real client IP behind Cloudflare in nginx?

Use set_real_ip_from for each of Cloudflare’s published ranges, then real_ip_header CF-Connecting-IP;. Do not trust 0.0.0.0/0, because then any client can forge the header. We did not test against Cloudflare itself. The range list at cloudflare.com/ips changes, so keep the nginx copy updated.

Is trusting X-Forwarded-For safe for rate limiting?

Yes, but only from your CDN’s address, read from the right. Trusting X-Forwarded-For from everyone let a client that skipped the CDN send 20 of 20 logins through on nginx, Caddy, and Traefik, because each fake address got its own bucket. Restrict the trust list to the CDN.

Related Reading

• What Your Reverse Proxy Leaks by Default
• Security Headers That Break Your Apps
• The Header Your Reverse Proxy Keeps Dropping
• DDoS Mitigation: Teaching Your Server to Say No Politely (Then Impolitely)
• Fail2ban vs CrowdSec: Blocking the Bots Actually Smartly