← CloudScale Plugin Help/CloudScale Cyber and Devtools: Free WordPress Security, AI Penetration Testing & Developer Toolkit
Fake Crawler Check
Scrapers and vulnerability scanners call themselves Googlebot because sites wave Googlebot through. A visitor claiming to be Googlebot, Bingbot, Applebot, YandexBot, Baiduspider or Yahoo Slurp is checked with forward-confirmed reverse DNS, the method those search engines publish. No outside service is involved, no visitor waits on it, and it starts in Log only.


🕷️ Anyone Can Say They Are Googlebot
The user agent is one line of text the visitor writes themselves. Scrapers and scanners claim to be a search engine because sites, CDNs and rate limiters let search engines through. The address cannot be faked so easily, and the search engines publish how to check it.
How the Check Works
- Reverse lookup. The visitor’s address is resolved to a host name, for example
crawl-66-249-66-1.googlebot.com. - Domain match. The name must end in that operator’s published domain:
googlebot.comorgoogle.comfor Google,search.msn.comfor Bing,applebot.apple.comfor Apple, and so on.evilgooglebot.comandgooglebot.com.evil.netdo not match. - Forward lookup. That name must resolve back to the same address. Anyone who controls their own reverse DNS can make an address claim to be
googlebot.com; only Google can makegooglebot.compoint back at it.
Only the six crawlers that publish a way to verify them are judged. A bot that publishes nothing, such as GPTBot, is not called fake for lacking a proof that does not exist.
Built so it never costs you a search ranking
- No visitor waits on DNS. The lookup runs after the response has been sent, and the verdict is cached: a week for a real crawler, a day for a fake. The first visit from a new address is always allowed.
- A DNS outage is not a verdict. If the lookup fails, a known name is resolved as a control. If that fails too, the resolver is down, nothing is stored, and the crawler is checked again later.
- Bounded. At most twenty new addresses are verified a minute, so a flood of fake claims cannot turn your server into a DNS cannon.
- Log first. Log only records each fake in Recent Matches as
fake_crawlerand refuses nothing. Block refuses a proven fake on its next request. - Block is refused when it would hurt you. If every claim so far has failed, the site is almost certainly reading a proxy’s address instead of the visitor’s, and Block would refuse Google itself. Fix the Visitor IP Source first.