Googlebot Behaviour

Ever had trouble understanding Googlebot’s behaviour or how to best implement a robots.txt file? A week ago, Vanessa Fox answered questions about Googlebot’s behaviour on Webmaster Central. Here are a few of the things I learnt. If your site is down for maintenance you may be worried that Googlebot will index your “Down for Maintenance” page. Let Googlebot know to rather try the pages later by simply configuring your server to return a status of 503 (network unavailable) instead of 200 status (successful). While some have trouble getting Googlebot to crawl their site at all others may suffer from their site being crawled too much. If you are the later, Vanessa Fox says you just need to contact Google and they will make sure they don’t overwhelm your server. She mentioned that Google are looking on bringing out a new feature on webmaster tools that’ll allow you to provide input on your crawl rate. Should you want to prevent Googlebot from crawling a number of pages, place those pages in a single directory (if possible) and use a robots.txt file to block those pages. You can also use META tags but those are for single pages. Any questions on Google’s behaviour? Join our SEO forum. You can read the full Q & A post by Vanessa Fox on Webmaster Central.