Most Googlebots are fake

Summary

On the internet, no one knows you’re a dog… and the flip side is that I can claim to be a dog (or a Googlebot) whenever I feel like it. These liars impersonate Googlebot in hopes that it will help them bypass bot controls; indeed, if you look up that IP in my log above, it comes from Virtual Machine Solutions LLC. This is a hosting provider, so someone (not Google) rented a server from them to send this traffic. People do this, all the time, and it seems to have hit a new scale recently. Chris Siebenmann theorizes that “it’s a large scale campaign by a single abusive crawler by people who can afford to obtain a lot of servers at a lot of different hosting providers”, rather than a widespread rise in popularity for an old trick.Lightning Q&AQ: How do I verify that a request came from the real Googlebot?A: Here are Google’s docs for verifying Googlebot; in short, you can spot-check with a couple of `host` commands or check against Google’s published IP ranges. Most major crawlers publish some kind of IP list, but the details vary; the JAFAR proposal aims to standardize these lists, though it’s still working its way through the standard-making process.Q: What if I looked up the IP address and saw it was registered to Google LLC?A: I’d still run through the verification process, since “Google LLC” is also the registration for Google Cloud hosting IP addresses. Though personally, I wasn’t able to find any recent examples of this - I wouldn’t be surprised if the Google Cloud anti-abuse team had some kind of controls to crack down on this.Q: I checked and it really is real Googlebot. What should I do?A: See Google’s docs on how to handle overcrawling of your site.Q: You mean to tell me people can lie on the internet?A: Yes :(Q: What if we added another field to stop them from lying?A: Please refer to RFC 3514, the “evil bit” standard. Unfortunately, bad actors do not comply with this standard. On a more serious note, I’ve written at length about the Web Bot Auth HTTP Signa...