Issue with bots

Please take note that my hosting service recently took down my site due to the influx of bots. However, the biggest issue was Wordfence trying to store every visit in my site’s sql database and this was slowing down the whole server. We resolved the issue by turning off the live traffic option in Wordfence. I am not sure if this helps you or not.

In addition, I went into Google’s Webmaster Tools and set Google’s crawl rate to the lowest number.

As for the bots, I was told to block the non-Google bots using the following code added to the robots.txt file:

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^bot\*$
RewriteRule .* – [F,L]

I was also given the following additional code to block empty user-agents (also added to robots.txt file):

RewriteCond %{HTTP_USER_AGENT} ^$
RewriteRule ^(.*)$ – [F,L]

Please let me know if the advice above helps resolve your issues as I am in a similar position.

Hi @bimcreate
If you are sure this was because of bad bots crawling your website, then you can set the options in “Rate Limiting Rules” to be more aggressive, also you can use the “Live Traffic” log to detect some of these bots and block their IPs if needed.

Thanks.

thanks!