GoogleBot Issues

Hello @sumitmalhotra and thanks for reaching out to us!

If you check Tools > Live Traffic, are you see the blocks here? If so, you can whitelist them if you wish to directly from here.

You will want to review your Rate Limiting options to see why they are being blocked. Navigate to All Options > Firewall Options > Rate Limiting.

I generally set my Rate Limiting Rules to these values to start with:

• If anyone’s requests exceed – 240 per minute
• If a crawler’s page views exceed – 120 per minute
• If a crawler’s pages not found (404s) exceed – 60 per minute
• If a human’s page views exceed – 120 per minute
• If a human’s pages not found (404s) exceed – 60 per minute
• How long is an IP address blocked when it breaks a rule – 30 minutes

I also always set the rule to Throttle instead of Block. Throttling is generally better than blocking because any good search engine understands what happened if it is mistakenly blocked and your site isn’t penalized because of it. Make sure and set your Rate Limiting Rules realistically and set the value for how long an IP is blocked to 30 minutes or so.

Remember there is no hard and fast, one size fits all set of rules for every site. This is just a good place to start. During an attack, you may want to make those rules stricter. If you see visitors, like search engine crawlers getting blocked too often, you might want to loosen them up a little.

Let me know if this helps!

Thanks!

Thanks for your suggestion WFAdam … I made the changes as suggested by you but it didn’t help at all.

Hello again @sumitmalhotra

If you check Tools > Live Traffic, are you see the blocks here? If so, you can whitelist them if you wish to directly from here.

Let me know what you find!

Thanks!

Hi WFadam, I see blocked Bots under Tools > Live traffic

They are not googlebot. The problem persists

• This reply was modified 4 years, 2 months ago by sumitmalhotra.

Hello again @sumitmalhotra

Best thing to try here is, open your Tools > Live Traffic page in one tab and then open the Google Mobile test in another. Run the test until it doesn’t work, switch over to your Live Traffic tab and see if you see the attempt to connect from here. If you do, you can click on it and whitelist it. If not, it might not be Wordfence that’s blocking it.

You can also try to disable Wordfence and run the Google test to see if that’s the cause.

Let me know what you find!

Thanks!