docs

A user-agent is a name tag, not an ID

Every bot rule eventually needs an exception for search engines — nobody wants to block Googlebot. The obvious way to write that exception is to match the user-agent string, and it is the wrong way, because the client chooses its own user-agent. One curl -A flag is the whole attack:

curl -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' https://example.com/

A UA-string allowlist is a door with the password written on it. And the incentive runs the wrong way: the more trusted a crawler is, the more worthwhile its name is to borrow, so the UAs an operator most wants to trust are exactly the ones under the most pressure.

The useful part is that this pressure produced a fix. The vendors whose crawlers are worth impersonating — Google, Microsoft, OpenAI, Apple and others — publish the IP ranges their crawlers actually come from, as machine-readable JSON they keep current. That turns identity into something the client cannot choose.

How unmask decides

unmask ships those published ranges as presets and matches the address, not the name. Two independent settings are involved, and the useful mental model is that each one controls its own rescue path:

With both in their shipped state, Googlebot/2.1 arriving from a rented VPS is not a search engine; it is an ordinary unknown client and it meets the ordinary challenge flow. The real Googlebot, arriving from Google's published range, is rescued as before — and it does not have to solve anything, so your rankings are unaffected.

A crawler whose vendor publishes no range is passed on its user-agent string alone, and that string is free to claim. So the name buys deliberately little: it clears the challenge a browser would get, and nothing else. A trap it trips, a community listing, a country, network or custom rule, a protected path, a deny and every rate limit apply to it as to anyone; the name counts only for a GET or HEAD; and a per-address budget (settings → UA filter, 30 requests a minute by default) caps how much one address gets through on the name before it meets a CAPTCHA. Each crawler's Detail dialog shows what the last day let through on its name and from how many addresses, so a name that is being abused can be set to normal checks or stopped.

A link-preview fetcher is a named crawler too, and it calls from abroad, so a country rule written against the world's scrapers would stop it with them — a day of posts without their cards. The fetchers of X, Facebook and LINE are therefore checked a second way: the name together with the network the request comes from (AS13414, AS32934, AS38631 — a vendor's own system, which nobody else can send from). A fetch that passes both is exempt from the country and ASN rules only; a custom rule, a protected path, a trap and the rate limits still apply. This needs an ASN mmdb (settings → Network). Without one, or for a fetcher calling from a cloud (Slack, Discord), the country tab shows as you edit a rule which fetchers the last day's traffic says it would stop, and the notification list says so when it starts happening.

Which crawlers can be verified this way

These vendors publish a machine-readable range list, so their crawlers are identified by address:

Retired crawler names are covered on purpose. Google Web Preview and similar products no longer generate genuine traffic, so leaving them on UA-only rescue would only ever pass a spoof — “Googlebot is verified, but Google Favicon walks through” is not a defence. One vendor-branded pattern left open is the whole hole.

Crawlers with no published range — smaller search engines, feed readers, link unfurlers, in-house monitors — can only be recognised by their UA string, so that remains their rescue path. There is no way around this: if a vendor does not publish where it comes from, nobody can tell its crawler from an impersonation. Judge them by what they are worth to you, and set the ones that are not to Normal checks or Stop in their group's Detail under settings → UA filter.

The one thing worth checking

The two settings have to agree, and there is a deliberate asymmetry between them that is worth understanding before you change either.

Turning a range preset off re-opens the UA path for that vendor. This is the safe direction, not an oversight: if disabling a preset also dropped the UA rescue, the genuine crawler would have no way left to pass and you would be blocking Googlebot. So unmask never lands in “UA refused and range off”. The consequence is that a preset switched off quietly restores exactly the spoofable behaviour the policy exists to remove.

Both axes work the same with the nginx module on your host and in the gateway container, so the answer does not depend on how you deploy.

Checking your own install

Send yourself a spoofed crawler. From any machine outside the vendor's ranges — your laptop, a VPS, anything — ask for a page that unmask protects:

curl -sS -o /dev/null -w '%{http_code}\n' \
  -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' \
  https://your-site/<a protected path>

A 403 carrying the challenge page is the correct answer: the name was ignored and the address decided. A 200 means the UA string still rescues that pattern — check the two settings above.

You cannot test the other half from outside, and that is the point: producing a request from inside Google's published range is precisely what an impersonator cannot do. What you can confirm is that genuine crawlers still arrive — the admin's AI / crawler traffic card (on the Dashboard and Stats pages) breaks crawler traffic down by category, a category opens into the individual crawlers with 🛡 on the ones verified by address, and Search Console's live URL inspection fetches your page as the real Googlebot, from the real range, on demand.