Bing Bot

Hi @polina92, thanks for getting in touch.

Do you have any Live Traffic logs of these issues? They should look like the below entry:

Washington, United States was blocked by real-time IP blacklist at https://********.com/section/page-name
11/1/2021 10:40:27 AM (39 minutes ago)
IP: 13.66.139.2 Hostname: msnbot-13-66-139-2.search.msn.com
Human/Bot: Bot
Mozilla/5.0 (compatible; bingbot/2.0; +https://www.bing.com/bingbot.htm)

The reason I ask is because I’d like to verify the bot with bing, and also check our internal block lists for the presence of the IP used.

Thanks,

Peter.

Hi,
I can see the number of IP’s that your plugin declares to have blocked, but not which ones -till I am not mistaken-.
For more info, I can add 2 elements
1. Bing bot is very greedy, resource wise.
2. I have had to reduce the number of links per sitemap for Bing, to enable their crawling. And submit them 1 by 1. What is time consuming.
Kind regards

Hi @polina92,

To filter the results to be less time-consuming, you could navigate to Wordfence > Live Traffic and select the “Advanced Filters” checkbox. Set up a filter in the 3 input boxes to be:

User Agent > contains > bing

This will filter any results that include the term bing as my example above. You should get examples of blocked and allowed cases of the bingbot visiting your site.

Thanks,

Peter.

Hi There,
I did not find any record. There are in wpcerber, not in yours
How to interprete?
Kind regards

Hi @polina92,

If there are no records of Bingbot being blocked or tracked in your Wordfence Live Traffic, but you are seeing that elsewhere you will need to consult with the providers of that plugin to see whether anything is incorrect. We recommend against blocking Bingbot – you can check whether these IPs are genuine bots as these services will publicly state, or at least provide a search for, the IPs these use. There can be issues with SEO rankings amongst other things if you flat-out decide to block a crawler using robots.txt, but of course this is at your discretion.

Treatment of Google crawlers, along with other crawlers can be set in the Rate Limiting section of Wordfence > All Options. Whilst Bingbot and DuckDuckGo are not specifically allowlisted by default, they can be controlled through the crawler options on this page if you wish.

I generally set my Rate Limiting Rules to these values to start with:
Rate Limiting Screenshot

• If anyone’s requests exceed – 240 per minute
• If a crawler’s page views exceed – 120 per minute
• If a crawler’s pages not found (404s) exceed – 60 per minute
• If a human’s page views exceed – 120 per minute
• If a human’s pages not found (404s) exceed – 60 per minute
• How long is an IP address blocked when it breaks a rule – 30 minutes

I also always set the rule to Throttle instead of Block. Throttling is generally better than blocking because any good search engine understands what happened if it is mistakenly blocked and your site isn’t penalized because of it. Make sure and set your Rate Limiting Rules realistically and set the value for how long an IP is blocked to 30 minutes or so.

Just as a note, there may be overlapping features when running two security plugins concurrently that can cause performance issues if both are scanning for the same vulnerabilities or checking the same files concurrently, so it would be advised to choose your preferred security plugin, or disable overlapping features from one where possible, and stick with that going forward.

Thanks,

Peter.