BLC reporting working links as broken

Hello @scarne

Trust you are doing great today.

I reviewed your shared links and I believe the mentioned errors are legit, let me explain
401 Unauthorised: The first links show that error and it means the BLC bot could not access the page as it was asking for authentication, if you open the link on your end you will see it asking for consent before showing the page contents.

21 Int Request Error: The second link error happens because that page is not accessible I tested on my end and used a tool but it showed an error when tried to access it multiple times.

500 Internal Server Error: I can’t see any issues with the amazon link but looks like when the BLC bot trying access it it shows a server error, it could be a temporary issue, you can add to ignore list or try to scan the site again.

403 Forbidden: The last link is not working for me either it seems their firewall is blocking all requests from different countries except Germany, I’m afraid we can’t do much about this issue on our end you need to add to exclude list as well.

I’m suggesting to take a look at this document to learn more about each type of error showing in broken link checker reports:
https://wpmudev.0om/docs/wpmu-dev-plugins/broken-link-checker/#broken-link-status-codes

Please let us know if you have any other questions or concerns.

Kind Regards,
Amin

I have reviewed your response which I find very puzzling indeed. I have looked again and I still find that none of the links are broken.
* 401 Unauthorised: The link is behind a paywall, but it isn’t broken.
* 21 Internal Request Error: This is a technology explanation site. I don’t know why you can’t access it, but I have had no difficulty.
* 500 Internal Server Error: You have acknowledged this one isn’t broken.
* 403 Forbiddeen: This site is not restricted to Germany. In fact, it’s a Brtish political site and I am able to access it from the UK.
It seems that there is a problem with your systems being able to access these sites, not the sites themselves.

Hi @scarne

I placed all 4 links on my site and I was able to replicate only this:

https://www.fnlondon.com/articles/bankers-take-lessons-british-gas-20110627 (401 Unauthorised)

https://www.makeuseof.com/tag/feedreaders-to-become-most-popular-tools/ (21 Int Request Error) -> randomly

https://www.libdemvoice.org/leveson-report-open-thread-31860.html (403 Forbidden)

I have pinged our BLC Team in this matter:

1. Do we treat consent also as 401 Unauthorised, same as username or password

=====

When it comes to:
https://www.makeuseof.com/tag/feedreaders-to-become-most-popular-tools/ (21 Int Request Error)
There is a high possibility that this site have some issues, as based on my above tests (not only via BLC, but visit in the browser) I was able and not able to access site randomly (same with BLC scan)

=====

As for:
https://www.libdemvoice.org/leveson-report-open-thread-31860.html (403 Forbidden)
when I accessed this URL in the browser, it seems it is under CF(cloudflare) and it blocks our IPs or UA.
You can check this part: https://wpmudev.com/docs/wpmu-dev-plugins/broken-link-checker/#troubleshooting-notes

For issue with link no.1 we will post an update here as soon as more information is available from BLC Team.

Kind Regards,
Kris

I think you are confirming that three of the four links are NOT BROKEN (and the fourth one is only broken sometimes). If BLC can’t get through to a Clouflare site or to one which requires “consent”, that means that BLC isn’t doing what it promises to do. It may be reassuring to you and your colleagues to know that there is an explanation for the wrong report, but from the customer’s perspective, it is still a wrong report.

Hi @scarne,

We do understand you, however, in this case, the primary issue with most URLs is with the user agent (UA) or IP being blocked by the servers hosting the content. This is a common challenge with any automated tools, as many websites implement security measures that can inadvertently block legitimate crawlers.

The BLC tool operates within the constraints of how web servers interpret and respond to automated requests.

You should also find similar if you try to run a cURL request from your PC, for example:

curl -IL https://www.amazon.co.uk/dp/B07BN6BWRV/ref=dp-kindle-redirect

Due to these aspects, the solutions or workarounds are limited and will require Whitelisting BLC’s UA/IP etc, however, these can not always be enforced as it entirely depends upon the destination server where it’s crawled.

Understanding the technical nuances behind each type of server response and considering the context of your site’s audience and the nature of the linked content can help in deciding how to address these reports.

You can always check our documentation on what each server response error would point:

https://wpmudev.com/docs/wpmu-dev-plugins/broken-link-checker/#broken-link-status-codes

Kind Regards,

Nithin

I was trying (unsuccessfully, it seems) to explain how this looks from the perspective of a lay user like myself. There may be a very good reason why your tools cannot access a site that, for example, requires authorisation. It would be a lot more helpful if BLC identified those as needing investigation by the customer (ie me) rather than reporting them as “broken”.

I write news and comment pages with links to primary sources (eg national newspapers), some of which are behind a paywall. Those of my readers who pay to read those other sites can click through. Those who don’t can at least have the satisfaction of knowing that my research is properly sourced and other readers will be able to check up on me.

Kind regards

Hi @scarne,

It would be a lot more helpful if BLC identified those as needing investigation by the customer (ie me) rather than reporting them as “broken”

I believe that’s what BLC is doing, our crawlers didn’t receive a 200 response from the URL but instead received a 401 error. We report that in our scan result so that the end user can investigate and mark the URL to be ignored.

We have detailed documentation about the error status codes in our documentation for your reference, please find the same here: https://wpmudev.com/docs/wpmu-dev-plugins/broken-link-checker/#broken-link-status-codes

I am afraid, there is no way BLC can identify the URL is under a paywall.

Kind Regards,
Nebu John

This simply isn’t correct. I have switched back to the “local” version of BLC and, sure enough, the link we have been focussing on is NOT shown as broken. In the list of All links, it is marked as “401 OK”. In the list of Broken links, it is (corrctly) not showing (becaus the link works). I don’t know why you are insisting that the cloud version of BLC is doing its job by reporting this perefectly good link as broken.

It is worth reminding ourselves that, although this exchange has become focussed on one or two examples, my original point was that the cloud version of BLC was giving me a LOT of false negatives. That remains the case.

• This reply was modified 1 month, 3 weeks ago by Simon Carne.

Hello @scarne

I can understand your point and concern here.

The cloud version scan engine is completely different than the local version, the cloud version is limited to crawling what is publicly accessible on the site, When comparing it with the local version, there are both advantages and disadvantages.

The cloud version uses a more advanced and modern method to check the links, similar to what Ahrefs, Semrush, and other online tools do while the local version may work better on some types of links it can’t detect all the links on site.

I will bring your request to our Broken Link Checker development team’s attention for review, I believe we can improve how BLC shows the link in the cloud version.

Kind Regards,
Amin

Hi @scarne

An update here

As Amin let you know, we reported it to our developers and product team and we will be working on improvements in both engines. We can’t give any estimated time yet but please keep the plugin updated and eyes in the changelogs.

I’ve set this thread as resolved, but feel free to let us know whenever you need any help with the plugin.
Best Regards
Patrick Freitas