Scan initiation failed

Hi @joyryde

I hope you’re well today!

As for the typo – thank you, I’ll pass that over to Broken Link Checker team to get it fixed.

As for the main issue:

From the description I see it’s about the new cloud engine. This acts much like regular crawler (e.g. those from search engines) so it scans the site “from outside” and must be able to pass through any possible lockouts/blocks/filters.

I understand that you have already made sure that CloudFlare is no longer blocking it but since it still doesn’t start, there certainly are some connection issues.

It would be best now if we could check Broken Link Checker logs on our end to see what exactly our engine is seeing/getting from the site.

For this, I would need to know what is the site in question – would you let us know what site is it, please?

Best regards,
Adam

Hi Adam!

https://nomadicsupply.com

Hi @joyryde

Hope you are doing fine and thanks for sharing additional info!

I reviewed the logs in the HUB and they are showing as if the process is started successfully, no errors are listed on our end and it seems to communicate with our servers/API. However the process doesn’t seem to complete. Could you try the following steps to verify the communication with the site is effective?

1. Log out from the WPMU DEV Dashboard.
2. Clear your site and server cache.
3. Log back into the WPMU DEV Dashboard.
4. Run the Scan from the Broken Link Checker one more time.

If you still have issues, could you share a screenshot of the User Agen and the IP address whitelisting done in Cloudflare in case we can provide some additional recommendations?

Looking forward to hearing from you.

Kind regards

Luis

1. Log out from the WPMU DEV Dashboard. Done.
2. Clear your site and server cache. Done.
3. Log back into the WPMU DEV Dashboard. Done.
4. Run the Scan from the Broken Link Checker one more time. Done.

Cloudflare:

https://tinyurl.com/2y9ptof3

https://tinyurl.com/2cumhjt9

Hi @joyryde

Thank you for response!

I understand that you are still getting the very same error, right?

I check logs again on our end and I noticed that, while the scan does start (like Luis mentioned) and doesn’t seem to continue, there’s actually another message: 403 Forbidden.

This would mean that scan is, indeed, initiated and communication between site and us starts but when scanner engine tries to request the site to detect links, it gets “403 forbidden” HTTP status from it.

CloudFlare settings look fine so unless there some other rule set that overrides it, it shouldn’t be an issue. But there still may be something server-side related.

Do you have any additional security plugins installed on site directly (aside form using CloudFlare)? If yes, what are they?

Do you know if there is any firewall right on your server? If there is or you are not sure, could you get in touch with your host support and make them confirm that the same IP and user-agent string is not only “not blocked” but also additionally included in all allow/bypass lists?

Kind regards,
Adam

We use Cloudways Hosting and Wordfence.

Hi @joyryde,

Please add the User-Agent and IP mentioned on https://wpmudev.com/docs/wpmu-dev-plugins/broken-link-checker/#troubleshooting-notes to the allowlist on your Wordfence firewall settings and let us know the results accordingly.

User Agent: WPMU DEV Broken Link Checker?Spider
IP : 165.227.127.103

Kind regards,
Zafer

I am able to whitelist an IP with Wordfence, but not a Spider. At least not that I can see in their Firewall settings.

We whitelisted you and your plugins scan failed again.

Hi @joyryde

We still see error in our logs:
Request Error: Forbiden

Can you manually add this code to robots.txt on the root of your site?
WPMU DEV Broken Link Checker?Spider

Kind Regards,
Kris

There is nothing in our robots.txt file except this:

User-agent: Googlebot
Crawl-Delay: 5

Not sure where you’d add “WPMU DEV Broken Link Checker?Spider” to that.

Hi @joyryde

Right below the current content inside robots.txt file try adding these lines exactly:

User-agent: WPMU DEV Broken Link Checker Spider
Allow: /

if that doesn’t help, we are back to the Wordfence or server configuration, I’m afraid, but please give it a shot first.

Kind regards,
Adam

Hello!

We’ve added it. It says SCAN INITIATION FAILED.

Hi @joyryde,

I’m afraid, the issue is still related to a Firewall or the server blocking the request from BLC.

The BLC is getting the following response when I try to initiate a new scan:

{"code":403,"msg":"Request Error: Forbidden"},

Do you have Bot Fight mode enabled in Cloudflare?
https://developers.cloudflare.com/bots/get-started/free/

Could you please temporarily disable it and then run a new scan to confirm whether it works fine?

If it’s still the same issue, then most probably a configuration on the Hosting side is blocking it.

Would recommend bringing it to the Hosting provider’s attention and checking whether the mentioned IP or the UA is blocked in any way or not.

Please do let us know how that goes.

Kind Regards,

Nithin

Hi,

We disabled Bot Fight Mode in Cloudflare and then ran a scan, the result is:

Scan initiation failed

Hi @joyryde

Thanks for response!

We are also still getting only “403 forbidden” response. Unfortunately, I don’t have any more information on that on our end as we only store the HTTP response code – there is no full response headers stored, any traceroute data and similar. In other words, from our end there is no way of telling what exactly (was it CF or the server/site itself) responded with that “403 forbidden” code.

So that means we are somewhat limited to some “trial and error” procedure but I think at this point we should try to establish for sure whether the CF is actually still involved in it or not so let’s try to do this:

Could you check your hosting logs (I think you should have access to access.log via the hosting panel) and look for entries in access.log that come from any of these IPs

18.204.159.253
54.227.51.40

and this IP specifically

165.227.127.103

Also check logs for the user-agent string of “WPMU DEV Broken Link Checker Spider”.

If there are entries in logs, please let share those exact entries (from around the date/time when you last time tried to initiate scan).

If there are no such entries it would mean that they didn’t even reach the site/server – so that would point back to CloudFlare or, unlikely but still possible, to some DNS issues.

That’s not much but at least we could determine for good this way what to focus on – CF/DNS or site/server.

Kind regards
Adam