No broken links reported with new BLC

Hi @jamminjames

Sorry to hear you are having this issue.

I checked the logs of your site and it seems the scan is timing out, we pinged the plugin team to verify why it is happening.

We will keep you posted.
Best Regards
Patrick Freitas

@wpmudevsupport12, any word from the plugin team?

@wpmudevsupport12, still nothing? You said you pinged the plugin team, have they looked at it? Every report I get says no broken links, but I know that can’t be. We have thousands of posts with links, and with the previous plugin version, there would always be broken links found.

If they’re not going to do anything, how do I go back to the plugin version that actually worked? Thanks.

Hi @jamminjames

when it comes to your timing out, we have some limitations. Scan will time out and stop further scanning after one hour (that’s the “scan timeout” error). Also, we will stop further scanning after we encounter 20 hard errors (server errors and/or connection timeouts) – that’s because there’s something obviously not going well on your site -> possibly a traffic peak or a performance issue with your site, or something else of the sort – and we don’t want to contribute to the problem. That’s the “too many server errors” thing.

https://wpmudev.com/docs/wpmu-dev-plugins/broken-link-checker/#broken-link-status-codes

19 Too Many Errors – This indicates that BLC encountered too many server errors on your site and aborted the scan so it doesn’t put additional unwanted strain on your server.

With BLC 2.0 you can still use old BLC as it is still part of the core. Please visit this URL in your admin:
/wp-admin/admin.php?page=view-broken-links

and follow guide from modal popup to switch to old BLC engine.

Kind Regards,
Kris

• This reply was modified 1 year, 7 months ago by Kris - WPMU DEV Support.

Okay, thanks. We’ll try to find the cause of the server errors. As far as the scan timeout goes, is that a BLC setting or a php setting, or what?

Hi @jamminjames

Sorry to hear you had this inconvenience with the new version of BLC.

The timeout could be also related to access restriction in the server or network. The plugin requires a new API host IP and User Agent to be whitelisted. Could you please verify the new IP is whitelisted in any security plugin, hosting protection firewall or service like Cloudflare (in case you are using any)? The IP is the following:

165.227.127.103

You can also remove any restriction to the User Agent:

WPMU DEV Broken Link Checker Spider

Please check if whitelisting/allowing the IP and user agent helps to complete the scanning and report the proper number of links.

Kind regards

Luis

I added the IP to the exceptions list in our WordPress Wordfence security plugin and ran a manual scan. Got a “Something went wrong with request” error after about a minute or so in the WordPress BLC window. However, the spinner kept going, and hovering over it, it says a scan is still in progress, even after refreshing the page. After refreshing, the error went away, but came back after another minute or so. In the “The Hub” tab, no errors show, and it keeps scanning.

We do use the free version of Cloudflare, but I don’t see any Firewall settings there.

I added this line to robots.txt:
User-agent: WPMU DEV Broken Link Checker Spider

Is that right?

• This reply was modified 1 year, 7 months ago by jamminjames.
• This reply was modified 1 year, 7 months ago by jamminjames.

Update: I changed the robots.txt lines to:

User-agent: WPMU DEV Broken Link Checker Spider
Disallow:

Also, after a half hour or so, the Hub tab stopped showing the scan info, but did not show any results. (During the scan, it did show it was detecting over 100k links.) In the WordPress tab, it said 0 broken links, and “Duration 0”, blanks for start and end times. Clicking view full report opened a new tab which was blank other than the Run Scan button and some graphics.

• This reply was modified 1 year, 7 months ago by jamminjames.

Another update:

Found the Firewall rules in Cloudflare (hidden under “WAF”) and added an “allow” rule for the ip, under “Tools/IP Access Rules” … does that sound right?

Hi @jamminjames

I hope you are doing well.

Sorry for the delay here.

“(During the scan, it did show it was detecting over 100k links.)”

I double-checked our logs and it doesn’t show any server error anymore, which indicates the scan is working well.

But unfortunately, it is timing out.

At this initial release, we have a limit if one hour scan, due to the number of links it will timeout on your site so using the V1 engine would be necessary for now.

We are already working to improve this situation and looking for alternatives like increasing the timeout or including a resume scan button.

We can’t give an estimate time yet, unfortunately.

Best Regards
Patrick Freitas

Regarding your comment about improving the timeout situation, has this been done yet in the latest release?

Hi @jamminjames

The timeout limit – in terms of scan time – has already been increased on our end (in the cloud engine) form 1 hour to 3 hours so it that was the only reason for the issue, it should be better now.

However, that didn’t change the limit overall limit of errors that can happen during scan that would cause the scan to break and timeout limit on single URL request (as this could/would cause additional stress on site’s resource usage).

Do I correctly understand that the same issue is still happening for you? Did at least “something” improved for you since your initial issue report or it’s still the same?

Kind regards,
Adam

I haven’t tried it yet, we switched back to the old method a while ago. But you said earlier, “I double-checked our logs and it doesn’t show any server error anymore, which indicates the scan is working well.” … So that sounds like the only problem was the scan time, right?

Hi @jamminjames

It does look on our end like this was the only issue, yes. But the only way to confirm that for sure would be to actually try running scan again and see what happens.

I believe it should be fine but if not – we would then be able to check new logs and see if anything else/new came up.

No rush though ?? We are working all year round and can get back to it at any time so just test it whenever it suits you best and let us know then.

Best regards,
Adam

Okay, I’m running a scan now. We have thousands of posts on our site, and the scan does take a very long time. One question: is it just the initial scan that takes so long, or will it do this every time it scans?