Problem with broken link checker

Any thoughts on this problem (which is still happening)?

I’ve raised a ticket with my host provider to see if they can track down the appropriate error log but, so far, no response.

Since it works on my Test site where there are fewer links, it would seem to be something to do with the volume of links to be tested. Is there some way the checking activity could be broken up into smaller segments of, say, 30 links at a time?

Here is the reply I just got from my host provider (Namesco):

If you are receiving a 500 error when running a plugin through WordPress it suggests that the plugin is timing out because the PHP resources it requires to complete the script are being exhausted. We cannot raise PHP features but I can suggest some things which may help you run the script.

In the Dashboard please try disabling all running plugins apart from the plugin you are referring to, and any sidebar widgets temporarily, and make sure you are currently using a default WordPress theme – then retest.

If that still doesn’t run then, really, that plugin is more suited to a VPS environment where the php memory can be raised by accessing the php.ini file. We do not provide access to that file on the shared platform I’m afraid.

Their suggestion that it is timing out sounds plausible. I had a timeout problem with another plugin (Wordfence) until I adjusted a parameter that controls the size of chunks in which they do their checks.

Can you let me know whether you think you may be able to solve this problem by doing the link checks in batches? Otherwise I will have to look for an alternative method of checking the links.

Yes, it is most likely possible to do this in batches. I just need to find time to figure this out between my day job and paid WordPress development contracts. I will try to get to this by the end of the week.

Hi Yannick. Thanks for that. There’s no urgency. I just wanted reassurance that it was on your ToDo list so that I don’t need to put finding an alternative onto mine.

Please try upgrading to the latest version I just released. I have changed the code for the link checker so it outputs data bit by bit to browser instead of processing entire database at once and only sending output at the end.

Let me know if this does not help and i will research further.

Please consider donating to support this plugin’s development.

Alas! I still get the same error.
No output appears before the error message.
I guess it’s difficult for you to pin down the problem unless you can reproduce the fault. If you tell me which source file(s) to look into I could attempt some debugging at my end.

I have some other ideas. Would you be okay if I make some changes tonight and send to you as a github snippet to upload to your site?

Yes. Happy to give that a try tomorrow evening. I haven’t used github before but I’m sure I can manage.

Here is a link to the replacement file:

https://gist.github.com/ylefebvre/739330017fd83675f12e

You can use the download zip button to get a zip file containing the replacement link-library-admin.php. Then upload that file to the /wp-content/plugins/link-library directory of your web site via FTP. Or replace the code for that specific file in the plugins editor in the WordPress admin.

The change that I made here is in the function called ReciprocalLinkChecker, where I added a call to set_time_limit(0), to let it run.

Let me know if this works.

Success! Many thanks.

Not sure if there’s anything further you can do on this but, just in case.

Three of my links failed the broken link check, two with an Error 403 and the third ‘Website Unreachable’. All three links are valid links, however. I suspect this might be the checker timing out before the web site has responded. Here are the links that failed:

https://www.scottishreviewofbooks.org/
https://www.theinterpretershouse.com/submissions
https://www.shanestrachan.com/

The first two are the Error 403 ones and the third is the unreachable one.

If there’s not an easy answer to this, don’t bother with it. I will always do a double check on any links that fail the broken link checker.

Oops, got one of the links above wrong.
Should be:
https://www.theinterpretershouse.com/submissions

Try changing line 502 in link-library.php.

First, try this:
$response = wp_remote_get( $external_link, array( 'sslverify' => false ) );

If that does not help, try this:

$response = wp_remote_get( $external_link, array( 'timeout' => 10 ) );

You can try increasing the timeout value until you get a valid response. If these help, I can include them in the plugin.

Please consider donating to support this plugin’s development.

Thanks for the suggestions.

Your first option didn’t help. The second did but, as I increased the timeout I started getting the Internal Server errors again.

As I investigated things further, it became apparent that your analysis of the response codes from wp_remote_get isn’t sufficient for my purposes. Some of the links that were passing your checks as OK, were actually returning codes 301 (Moved permanently) or 302 (Moved temporarily) both of which require further investigation on my part.
One of the code 301 sites was actually re-directing to a page that said ‘This web site has been closed’, one of the code 302 sites was re-directing to the wrong page and others were re-directing correctly but, having this information, enables me to update the link to the new URL.

Also, when the wp_remote_get timed out, your code returned error 403, rather than the fact that it had timed out.

So, in file link-library-admin.php, when the ReciprocalLinkChecker function is called with $check_type = ‘broken’, I have replaced your call to the CheckReciprocalLink function with a call to a custom function of my own (using curl_exec and curl_getinfo and a CURLOPT_TIMEOUT to prevent Internal Server error when accessing a link takes too long). This function returns both the HTTP code and its meaning and, whenever the code is not equal to 200, I output both code and meaning in the Broken Link Checker Report. This more detailed report alerts me to all the links that need to be checked and then removed or updated.

So far, this seems to be working well for me. Should you wish, I can provide you with the code of my function and details of the edits I have made to the ReciprocalLinkChecker function.

The thing about CURL is that I don’t think it is supported on all PHP installations, so I am not sure it would make sense to add this as the primary means to check for broken links.

Still, it might be interesting to look at your code, if you can send it my way.