docs

A user-agent is a name tag, not an ID

Every bot rule eventually needs an exception for search engines — nobody wants to block Googlebot. The obvious way to write that exception is to match the user-agent string, and it is the wrong way, because the client chooses its own user-agent. One curl -A flag is the whole attack:

curl -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' https://example.com/

A UA-string allowlist is a door with the password written on it. And the incentive runs the wrong way: the more trusted a crawler is, the more worthwhile its name is to borrow, so the UAs an operator most wants to trust are exactly the ones under the most pressure.

The useful part is that this pressure produced a fix. The vendors whose crawlers are worth impersonating — Google, Microsoft, OpenAI, Apple and others — publish the IP ranges their crawlers actually come from, as machine-readable JSON they keep current. That turns identity into something the client cannot choose.

How unmask decides

unmask ships those published ranges as presets and matches the address, not the name. Two independent settings are involved, and the useful mental model is that each one controls its own rescue path:

With both in their shipped state, Googlebot/2.1 arriving from a rented VPS is not a search engine; it is an ordinary unknown client and it meets the ordinary challenge flow. The real Googlebot, arriving from Google's published range, is rescued as before — and it does not have to solve anything, so your rankings are unaffected.

Which crawlers can be verified this way

These vendors publish a machine-readable range list, so their crawlers are identified by address:

Retired crawler names are covered on purpose. Google Web Preview and similar products no longer generate genuine traffic, so leaving them on UA-only rescue would only ever pass a spoof — “Googlebot is verified, but Google Favicon walks through” is not a defence. One vendor-branded pattern left open is the whole hole.

Crawlers with no published range — smaller search engines, feed readers, link unfurlers, in-house monitors — can only be recognised by their UA string, so that remains their rescue path. There is no way around this: if a vendor does not publish where it comes from, nobody can tell its crawler from an impersonation. Judge them by what they are worth to you, and disable individual patterns under settings → UA filter when they are not.

The one thing worth checking

The two settings have to agree, and there is a deliberate asymmetry between them that is worth understanding before you change either.

Turning a range preset off re-opens the UA path for that vendor. This is the safe direction, not an oversight: if disabling a preset also dropped the UA rescue, the genuine crawler would have no way left to pass and you would be blocking Googlebot. So unmask never lands in “UA refused and range off”. The consequence is that a preset switched off quietly restores exactly the spoofable behaviour the policy exists to remove.

Both axes are wired identically whether you run the nginx module or forward-auth, so the answer does not depend on your deployment mode.

Checking your own install

Send yourself a spoofed crawler. From any machine outside the vendor's ranges — your laptop, a VPS, anything — ask for a page that unmask protects:

curl -sS -o /dev/null -w '%{http_code}\n' \
  -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' \
  https://your-site/<a protected path>

A 403 carrying the challenge page is the correct answer: the name was ignored and the address decided. A 200 means the UA string still rescues that pattern — check the two settings above.

You cannot test the other half from outside, and that is the point: producing a request from inside Google's published range is precisely what an impersonator cannot do. What you can confirm is that genuine crawlers still arrive — the admin's Crawlers view breaks traffic down by crawler, and Search Console's live URL inspection fetches your page as the real Googlebot, from the real range, on demand.