New Googlebot spidering architecturePosted under search engine optimisation
Most webmasters are familiar with Googlebot's continual spidering of a web site. Sometimes googlebot traffic can be greater than user traffic, making some web sites like Slashdot to ban googlebot from their servers.
Google has recently addressed the problem. In a move aimed at reducing bandwidth consumption, the adsense spider is now also refreshing the main googlebot cache. The change is shown in the diagram below.
One consequence is that including adsense on your site might get your pages indexed quicker since the Adsense robot is extremely quick.
Matt Cutts, who is provides politic feedback to many a frustrated webmaster and marketing proffessional, confirms that the AdSense Bot is Helping Googlebot With Indexing
Matt warns that you that the Google algorithm will probably penalize those that serve different content to the MediaBot and Googlebot
Technorati Tags: google
- Matt Cutts Keynotes London Search Engine Strategies - Feb 15, 2007
- Search Engine Strategies Conference: London 2006 - Jun 03, 2006
- Where things go wrong with Alexa rankings - Jun 01, 2006


