Here are the most common methods used by Googlebot impersonators and how you can protect your Web site: Method #1: Not validating Google IPs It uses a specific user agent, it arrives from IP addresses that belong to Google and always adheres to the robots.txt (the crawling instructions that website owner provide to such bots). Googlebot has a very distinct way of identifying itself. The vast majority of such impersonators post comment spam and also steal website content.įake Googlebot: How do the bad guys do it?.Among those targeted sites, 21% of those claiming to be Googlebot, were impersonators.16.3% of sites suffer from Googlebot Impersonation attacks of some kind.In a recent study of 1,000 customer websites that we performed at Incapsula, we discovered the following:

The problem? Hackers emulating the Googlebot have turned it into a target for impersonation. In fact, you want to give Googlebot the real VIP treatment. The more Googlebot visits your site, the faster your site’s content updates will appear in Google’s search results.Ĭonsequently, it’s of the utmost importance to allow Googlebot to crawl your website without blocking or disturbing it. Google uses a crawler called ‘Googlebot’ that crawls millions of sites simultaneously and indexes their content in Google’s databases. By doing so, Google learns what is new on our site and can immediately share our updated content with anyone searching online. We all want Google to visit our site and index our content as often as possible.

Google bot ip