Allow search engines, nice feature, thanks!
-
Nice job on the option to whitelist search engines.
I’d tend to whitelist all search engines, but it should be noted that Yandex is poorly behaved and can use an amazing amount of bandwidth. At one time, they crawled my site so much is was almost like a DDOS attack and I had to block the Yandex crawler. My hope is they’re better behaved now so I thought I’d give Yandex a go, and I whitelisted Yandex as well as removing the blocked Yandex IPs in my .htaccess. They probably ignore robots.txt but I took the blockage out of there as well.
Funny how all the media focus is on Google, while we have these other search engines that are all over the map in terms of behavior. Indeed, as the article linked below states, is it really that important to have a search engine from Russia slamming your website with spider hits that cost you bandwidth you have to pay for? At some point, they become a parasite. And who knows what spam gangs and scraper operations some search engines are associated with?
More here: https://searchenginewatch.com/article/2067357/Bye-bye-Crawler-Blocking-the-Parasites
MTN
MTN
- The topic ‘Allow search engines, nice feature, thanks!’ is closed to new replies.
