Your Bandwidth Bill Is Training Someone Else’s Model
It’s 2026 and your Gitea instance is on fire again. Not from traffic that reads anything, from a scraper fleet walking every commit diff of every branch of every repo, from a different IP each time, with a user agent that says Mozilla/5.0 and means nothing at all.
You can block them in robots.txt. The polite ones will listen. The polite ones were never the problem.
Anubis is a reverse proxy that sits in front of your site and makes suspicious clients do a small amount of work before they get through. It’s written by Xe Iaso under the Techaro banner, and the current release as of this writing is v1.27.0 from August 2026. It’s the thing you have probably seen if you have ever hit a self-hosted Git forge and been greeted by an anime jackal telling you it’s checking whether you’re a bot.
One thing up front, because most writeups get this backwards. Anubis’ own design docs describe the proof-of-work as “a hack whose real purpose is to give a good enough placeholder solution so that more time can be spent on fingerprinting”. It is friction, not a paywall. Anyone quoting you a dollar cost per scraped page is making it up.
What Anubis Actually Does
Anubis is a proxy. Requests come in, Anubis evaluates them against a policy file, and one of four things happens:
| Action | Effect |
|---|---|
ALLOW | Skip all further checks, pass to the backend |
DENY | Return an error page dressed up so scrapers think they succeeded |
CHALLENGE | Serve a challenge page, or verify the client already passed one |
WEIGH | Adjust the request’s suspicion score and keep evaluating |
WEIGH is the part that makes Anubis interesting and the part every tutorial skips. Instead of a binary bot-or-not decision, rules add and subtract points from a per-request weight, and a separate list of thresholds decides what to do with the total. The default catchall adds 10 points to anything with Mozilla or Opera in its user agent, which is nearly every browser and nearly every scraper pretending to be one.
Two behaviors surprise people:
Default is allow. If a request matches no rule, it goes through. Anubis is not a default-deny firewall unless you add a catchall DENY at the bottom of your policy.
DENY and CHALLENGE both return HTTP 200 by default. This looks broken until you read the comment in the shipped config: the most aggressive scrapers really want a 200, and they stop retrying once they get one. Give them a 403 and they come back harder.
Deploying Anubis Behind Caddy
Anubis goes between Caddy and your backend. Caddy terminates TLS, hands everything to Anubis, and Anubis forwards what survives.
services: caddy: image: caddy:2 ports: - 80:80 - 443:443 - 443:443/udp volumes: - ./conf:/etc/caddy - caddy_config:/config - caddy_data:/data
anubis: image: ghcr.io/techarohq/anubis:latest pull_policy: always environment: BIND: ":3000" TARGET: "http://blog:8080" POLICY_FNAME: "/data/cfg/botPolicy.yaml" METRICS_BIND: ":9090" SERVE_ROBOTS_TXT: "true" OG_PASSTHROUGH: "true" OG_EXPIRY_TIME: "24h" volumes: - "./botPolicy.yaml:/data/cfg/botPolicy.yaml:ro" healthcheck: test: ["CMD", "anubis", "--healthcheck"] interval: 5s timeout: 30s retries: 5 start_period: 500ms
blog: image: nginx volumes: - "./www:/usr/share/nginx/html"
volumes: caddy_data: caddy_config:Those environment variable names matter. BIND is where Anubis listens, TARGET is your backend, POLICY_FNAME points at the policy file. There is no UPSTREAM_URL and no POW_DIFFICULTY, whatever a blog post generated in 2024 told you.
OG_PASSTHROUGH is worth turning on. It lets Anubis fetch and cache your Open Graph tags from the backend and serve them on the challenge page, so a link posted to Mastodon or Slack still gets a title and a preview image instead of a jackal.
The Caddyfile:
sumguy.example.com {
reverse_proxy http://anubis:3000 { header_up X-Real-Ip {remote_host} header_up X-Http-Version {http.request.proto} }}That is the entire Caddy side. X-Real-Ip is not optional: without it every request arrives at Anubis from the Caddy container’s address, IP-based rules match nothing, and rate limiting is meaningless.
Reload Caddy against its actual config file:
docker compose exec caddy caddy reload --config /etc/caddy/CaddyfileOne instance of Anubis protects one backend. If you have five services, you run five Anubis containers, which is less absurd than it sounds because each one idles under 32MB of RAM.
The Policy File Is Where the Work Happens
Bot rules live in YAML (or JSON, since v1.17.0), not in environment variables. A minimal file:
bots: - name: well-known path_regex: ^/.well-known/.*$ action: ALLOW
- name: robots-txt path_regex: ^/robots.txt$ action: ALLOW
- name: amazonbot user_agent_regex: Amazonbot action: DENY
- name: internal-network action: ALLOW remote_addresses: - 100.64.0.0/10
- name: generic-browser user_agent_regex: >- Mozilla|Opera action: WEIGH weight: adjust: 10Matchers available on a rule: user_agent_regex, path_regex, headers_regex, and remote_addresses for CIDR ranges. GeoIP and ASN matching exist but require a Thoth subscription, so treat them as out of scope for a home lab.
The shipped default policy is mostly imports, and reading it is the fastest way to understand the tool:
bots: - import: (data)/bots/_deny-pathological.yaml - import: (data)/bots/aggressive-brazilian-scrapers.yaml
# Aggressively block AI/LLM bots by default - import: (data)/meta/ai-block-aggressive.yaml # Swap for one of these if aggressive is too much: # - import: (data)/meta/ai-block-moderate.yaml # - import: (data)/meta/ai-block-permissive.yaml
# Google, Apple, Bing, DuckDuckGo, Qwant, the Internet Archive, # Kagi, Marginalia, Mojeek, Arquivo.pt - import: (data)/crawlers/_allow-good.yaml
- import: (data)/clients/x-firefox-ai.yaml - import: (data)/crawlers/xai.yaml - import: (data)/common/keep-internet-working.yaml
- name: generic-browser user_agent_regex: >- Mozilla|Opera action: WEIGH weight: adjust: 10Note what that does out of the box: AI crawlers are denied by name, well-behaved search engines including the Internet Archive are allowed by name, and everything left over that looks like a browser gets 10 points of suspicion. Google is on the allow list for a specific reason spelled out in the config comments, which is that if you challenge Googlebot it starts trying to work around you.
If you already maintain a robots.txt full of AI crawler blocks, the project ships a robots2policy converter that turns it into policy rules so you’re not maintaining the same list twice.
Weights and Thresholds
Thresholds turn the accumulated weight into an action. These are the defaults Anubis ships:
thresholds: - name: minimal-suspicion expression: weight <= 0 action: ALLOW
- name: mild-suspicion expression: all: - weight > 0 - weight < 10 action: CHALLENGE challenge: algorithm: metarefresh difficulty: 1
- name: moderate-suspicion expression: all: - weight >= 10 - weight < 20 action: CHALLENGE challenge: algorithm: fast difficulty: 2
- name: mild-proof-of-work expression: all: - weight >= 20 - weight < 30 action: CHALLENGE challenge: algorithm: fast difficulty: 4
- name: extreme-suspicion expression: weight >= 30 action: CHALLENGE challenge: algorithm: fast difficulty: 6expression is a CEL expression over the variable weight. difficulty for the proof-of-work algorithms is the number of leading zero bits required in the SHA-256 output, so it scales exponentially. The defaults top out at 6. The env var DIFFICULTY defaults to 4.
Do not copy a difficulty of 16 out of somebody’s example. The Anubis docs use 16 in exactly one place, in a commented-out punishment rule, annotated # impossible. That is a joke in the docs. Read the comment before you copy the number.
One more thing about thresholds, and it catches everyone: they only apply to requests that did not match a terminal rule. A rule with action: CHALLENGE short-circuits the whole threshold ladder. If you want graduated difficulty, use WEIGH.
Adding your own weight rules is where the tuning happens:
bots: # Signed-in users carry a session cookie. Trust them a little. - name: has-session headers_regex: Cookie: "session=" action: WEIGH weight: adjust: -15
# Datacenter ranges you know are not readers - name: known-scraper-range action: WEIGH remote_addresses: - 203.0.113.0/24 weight: adjust: 20False Positives, and the JavaScript Problem
The proof-of-work challenges need JavaScript. That is the real cost of running Anubis, and it is bigger than the CPU time.
Things that break:
- Feed readers. Feedly, Inoreader, and every self-hosted FreshRSS instance fetch your feed with no JS engine. Allow the feed path explicitly.
- Link preview fetchers. Slack, Discord, Mastodon, and Signal all unfurl links.
OG_PASSTHROUGHhandles the common case; anything past that needs an allow rule. - Uptime monitors. Uptime Kuma, Pingdom, and friends will report your site as down. Allow their source IPs.
- Text-mode and accessibility tooling. No JS means no challenge solved.
The fix for most of it is path and IP allow rules ahead of the catchall:
bots: - name: feeds path_regex: ^/(rss|atom|feed)\.xml$ action: ALLOW
- name: monitoring action: ALLOW remote_addresses: - 10.0.0.5/32 - 192.168.1.100/32For readers without JavaScript, the metarefresh algorithm is the escape hatch. It sends a page that reloads itself after a delay rather than hashing anything, so difficulty there means seconds to wait, not zero bits. It ships as the lightest threshold tier. The tradeoff is honest: plenty of modern scrapers drive headless Chrome, and headless Chrome passes a meta refresh without noticing.
Metrics
Anubis exposes Prometheus metrics on a separate listener, which is what METRICS_BIND: ":9090" above turns on. You can also configure it in the policy file:
metrics: bind: ":9090" network: "tcp" basicAuth: username: metrics password: "generate-a-real-one"Rule names show up as metric labels, which is the reason the docs tell you to name rules in lower kebab-case. Once it’s scraping, you can see which rules fire and how much, and that is how you find out your generic-bot-catchall is eating your RSS subscribers.
pprof debug routes on the metrics server default to disabled now. Earlier versions exposed them on every bind address, which leaked command line arguments to anyone who found the port.
Gotchas
Cookie settings changed in v1.27.0. Anubis now derives cookie names from cookie settings, producing names like techaro.lol-anubis-auth-347ddb4a. Before this change, altering any cookie option without every client clearing their cookies caused an infinite challenge loop that looked exactly like Anubis blocking real users. If you have HAProxy rules keyed on the old cookie name, they need updating.
Storage is in-memory by default. store.backend: memory means every restart invalidates every issued pass and everyone gets challenged again. Fine for a blog, less fine if you redeploy hourly.
Get X-Forwarded-For right. Anubis has a dedicated caveats page for this, and getting it wrong means either every request looks like it came from your proxy or clients can spoof their source address into your allow rules. Do not accept the header from anything but your own reverse proxy.
CDN caching in front of Anubis. If Cloudflare caches a challenge page, everyone behind that cache gets a challenge they cannot pass. Either put Anubis in front of nothing cacheable, or make sure your CDN respects the no-store headers on challenge responses.
A honeypot is on by default. The shipped config enables the naive honeypot implementation. Set ip_log_file if you want the addresses that trip it written somewhere you can read.
Is Anubis Right for You?
Run Anubis if you are hosting something with an enormous crawlable surface: a Git forge, a wiki, a bug tracker, an archive. That is the case it was built for, and the case where scrapers take the site down.
Skip it if you run a static blog behind a CDN. A few hundred markdown pages served from cache costs you nothing to have scraped, and you would be trading zero bandwidth savings for a JavaScript requirement and a support burden. robots.txt plus a CDN bot rule covers it.
The honest summary: Anubis will not stop AI training, and nobody serious claims it does. What it changes is which sites are cheap to crawl, and yours stops being one of them. The scrapers that keep working are the ones running real browsers, and Techaro’s own roadmap says the future of this is fingerprinting rather than hashing. What you get today is a working tool that keeps your forge online, and that is worth more than a theory about arithmetic.
Deploy it, watch the metrics for a week, and tune the allow rules before your RSS readers complain. Your 2 AM self will appreciate it.
Common Questions
Does Anubis block AI crawlers by default?
Yes. The shipped policy imports an aggressive AI blocklist that denies known LLM crawlers by user agent, and moderate and permissive variants are included as commented alternatives. Well-behaved search crawlers including Google, Bing, DuckDuckGo, Kagi and the Internet Archive are explicitly allowed so they keep indexing you normally.
Do visitors need JavaScript to use a site behind Anubis?
For the proof-of-work challenges, yes. The metarefresh algorithm is the no-JavaScript alternative and reloads the page after a delay instead of hashing. It ships as the lightest threshold tier. Headless Chrome scrapers pass meta refresh easily, so it buys accessibility rather than protection.
What proof-of-work difficulty should I set in Anubis?
Leave it alone. The DIFFICULTY environment variable defaults to 4, and the default thresholds run from 2 up to 6 leading zero bits. Difficulty is exponential, so 16 is not four times harder than 4, it is astronomically harder. The Anubis docs label difficulty 16 as impossible.
Why does Anubis return HTTP 200 when it blocks a request?
Deliberate design. The shipped status_codes block sets both DENY and CHALLENGE to 200 because the most aggressive scrapers keep retrying until they receive a success code. Returning a 403 makes them hammer harder. The body is an error page that a scraper reads as content.
Can Anubis protect more than one backend service?
Not per instance. One Anubis container maps to one TARGET. For several services, run several Anubis containers, or terminate TLS in one proxy and route to per-service Anubis instances behind it. Each instance uses under 32MB of RAM, so the overhead is small.
