Amazon Bot Infinite pages

Hi @web56932

You can create or edit a robots.txt file in your WordPress root directory. If it doesn’t exist, you can create one. And add instructions there so Amazonbot won’t crawl on not needed pages. Also, you can implement a firewall rule that blocks the bot by the user-agent string, as far as we know it is the most effective solution. Because the Amazonbot usually ignores mostly everything.

Best Regards – Victor

I’m also seeing this. Its shutting down my website, as the bot just crawls. Whats more, it is compounded by WP-rockets Critical CSS and Preloading. The number of counts on the taxonomy tables start increasing, and the queries start delaying. Then the memory goes to upto 8gb. And eventually we get 20 second page loads.

This is a massive problem. And I had to disable your plugin entirely, as this just causes a huge issue on my website.

Also technically, when we filter based on one taxonomy page, we are duplicating that taxonomy page. So on content type with multiple taxonomies, we start getting duplicate content. Your filtered content, and the actual taxonomy. It makes it very hard to use this product.

I suspect nofollow is going to help.

Hi @mgparisi

As we mentioned, if you’re having issues with Amazon’s bot crawler ignoring the nofollow/noindex directives in robot metatags, it might be helpful to reach out to Amazon’s technical support or developer forums for assistance. It’s possible that there could be a technical glitch or misunderstanding regarding how Amazon’s bot interprets these directives. As there are no such issues with other crawlers, as far as we know the only issue that happens is only with this bot.

Best Regards – Victor

You can use cloudflare to block the amazonbot. I had same problem.

Rule under security WAF.

WHen userAgent contains Amazonbot

Then block