Baidu Rocks!
We recently sent an email to Baidu about crawl activity being particularly aggressive given that we are an english language site and are seeing no benefit of that crawl activity translating into traffic.
The response is below and well, we think it's absolutely unique from a search engine in the level of detail and the crafting of the response, (Ever tried emailing Google or Bing? Good luck!).
Firstly, our email:
Hi Baidu,
We've tried to signal that we don't wish to be crawled by Baidu, (Aggressive crawling seen but not benefit, (No traffic down-streaming)).
Unfortunately, you appear to ignore robots.txt?
See: http://www.explore.to/robots.txt
# Baidu [zai hui] User-agent: Baiduspider Disallow: /
Anyway, we feel forced to direct the Baidu crawler to our 404 pages as the only way we know how to potentially halt the crawl.
Any further guidance would be appreciated.
The response from Baidu:
Dear Simon,
Thank you for your email.
Our spider do honor robots file. However, the DNS setting updates in a certain rate that your robots.txt might not be activated immediately. And we are sorry about the delay.
During the lag, it seems that you have redirected our spider to crawl 404 pages, making our spider unable to report back your robots file and explaining why our spider has yet stopped, (Dope! My fault)
The team has discussed your case. Baidu cherishes the opportunity to present every best website and endeavors to offer excellent user experience to hundreds of millions of users worldwide. Therefore, we would like to have your website and hereby ask your permission on setting a proper crawling frequency.
According to our crawling log, we maintain about X thousand hits per day, would it be ok if we scale it back to about X thousand daily?
Or could you inform us a proper crawling frequency so that we could make special adjustment for your site?
Please write to tell us your opinion on the proposal. If you still not happy about the access of baidu spider, we would set our spider DISFOLLOW your website then, as requested.
We are looking forward to hearing from you.
Best Regards,
X, X
Baidu Spider Team









