Can you overuse Immediately block IPs that access these URLs?

Hi @mesmer7, thanks for your question.

My answer is, “sort-of”. In many cases where visits to non-existent paths/URLs are done, they happen with no prior knowledge of the plugins or platform you’re running and are done in a hit-and-hope manner by bots. Making sure your plugins, themes, and WordPress itself are always the latest versions should limit the chance of one of these ever succeeding.

Google and other legitimate bots should mostly only be hitting paths that exist or have existed in the past, but keeping an eye on your Live Traffic blocks should reveal whether any of your manually input URLs are problematic for them. If you see a number of Google (or other search) bots blocked at paths you’ve chosen, it could be worth removing the paths from that setting.

Wordfence itself handles its blocks by looking at the intent of an IP’s visit rather than just the page they tried to visit. The plugin does all of the important blocking for you so it’s our general recommendation to not implement a manual blocking regime – which can be time consuming to keep up with current URLs and IP ranges etc. However, if your site is being hit many times from one specific attack on the same path over and over, I can see why you’d want to stem the flow yourself.

You can read more here: https://www.wordfence.com/help/firewall/options#immediately-block-urls

Thanks,
Peter.

@wfpeter Thanks for the response.

I have not seen googlebot hit any of the non-existent URLs. But Googlebot reduced it’s crawl to my product pages and hasn’t crawled any of my articles for months. That’s the reason for my question.

Although there was one bot yesterday that claimed to be googlebot, but didn’t behave like it.

Based on your answer, I think I’ll remove 90% of the ‘immediately block’ url list, and just leave the few that I see a lot of bots scanning for.

Hi @mesmer7, thanks for your response.

It’s best to only manually block IPs if you’re firefighting an attack that just keeps retrying, even after the times set in Wordfence > All Options > Brute Force > Amount of time a user is locked out and Wordfence > All Options > Rate Limiting > How long is an IP address blocked when it breaks a rule? have expired.

I generally set my Rate Limiting Rules to these values to start with:
Rate Limiting Screenshot

• If anyone’s requests exceed – 240 per minute
• If a crawler’s page views exceed – 120 per minute
• If a crawler’s pages not found (404s) exceed – 60 per minute
• If a human’s page views exceed – 120 per minute
• If a human’s pages not found (404s) exceed – 60 per minute
• How long is an IP address blocked when it breaks a rule – 30 minutes

With Brute Force, I recommend trying 3-5 for attempts and password resets, counted over 4 hours, with a 30 minute (or longer) lockout time period.

I also always set the rule to Throttle instead of Block. Throttling is generally better than blocking because any good search engine understands what happened if it is mistakenly blocked and your site isn’t penalized because of it. Make sure and set your Rate Limiting Rules realistically and set the value for how long an IP is blocked to 30 minutes or so. This could assist in any issues you’re seeing with Googlebot.

Thanks again,
Peter.

Yes, I have those values for the rate limiting.

Additionally, I’d seen dozens of bots try to access non-existent themes and plugin files. So I started adding those to the ‘immediately block’ list.

Based on your first response to this thread, I think I went overboard. This weekend I cleared that list except for two values.

I also changed the rate limiting for Google from “Verified Google Crawlers” to “Anyone Claiming to be Google”. That’s when I noticed Google’s crawlers becoming more active. So I think “Verified Google Crawlers” was too restrictive and the reason for my problem.

I’d keep an eye on that just in case too many of those turn out to be false, but those may be spotted by other rules and settings.

Start up a new topic any time if you have further Wordfence questions and we’ll be glad to help.

Thanks,
Peter.