Zendesk Cloudflare link check error

  • Resolved Joe Westcott

    (@redredweb)


    3 years, 3 months ago

    Hi folks, it seems that Zendesk links report back as broken by default these days.

    Zendesk uses Cloudflare DDoS protection and reCAPTCHA, and this results in a “403 forbidden” error when bots scan Zendesk URLs.

    Is there a workaround for Google’s reCAPTCHA service, similar to the way that Broken Link Checker can enable successful YouTube link scanning via API, which is also a Google service?

    I’m guessing that maybe this is just a known bug with the Broken Link Checker plugin, where it fails with a 403 forbidden error for sites that use Cloudflare DDoS protection and reCAPTCHA, but I’d love to be wrong about this.

    Edit: It seems that there’s a related support issue that I didn’t find previously because I searched this support forum for “Zendesk”, and found no matches.

    In that forum, Adam (@wpmudev-support8) mentioned that the team was looking into possibly getting whitelisted as a “good bot”, about 1.5 years ago.

    That topic closed with a suggestion to whitelist the affected domain, but this doesn’t solve the problem because whitelisting a domain means that the Broken Link Checker plugin will fail to scan the domain for broken links.

    • This topic was modified 3 years, 3 months ago by Joe Westcott. Reason: typo
    • This topic was modified 3 years, 3 months ago by Joe Westcott. Reason: linked to related issue, explained why the proposed solution does not solve the problem

    The page I need help with: [log in to see the link]

Viewing 2 replies - 1 through 2 (of 2 total)
  • Plugin Support Williams – WPMU DEV Support

    (@wpmudev-support8)

    Hi @redredweb

    I hope you’re well today!

    Yes, this is a known issue but I’m afraid I still don’t have any “fool proof” solution for this.

    IT was some other person who suggested “whitelisting” with CloudFlare but it’s not as simple as it seems. As I mentioned there, I fully agree that it’s a “good bot” but, well, it’s still a kind of a bot and the “get it to be allowed by CF” idea sounds great but there’s a major obstacle:

    – all the checks are made directly from the site/server where the plugin is installed
    – so that means it’s not from our servers/API but from thousands of different and “unrelated” services
    – the only way to possibly identify it would be e.g. user-agent string and that is super-easy to spoof (fake)

    So if CF would allow it fully out of the box based solely on user-agent string it would render their bot protection essentially useless because everyone could “pretend to be a Broken Link Checker bot” and use that to bypass that protection.

    The two options to possibly solve it “for good” would be to either get all the checks done from us and make sure that our services are “officially allowed” by CF to do these checks or to make it possible to integrate plugin with CloudFlare API.

    I’m aware that this is an ongoing issue that is already known for quite some time but I’m afraid I still can’t give any ETA. Our developers are aware of it and are working on this but also on a huge number of other improvements. The plugin is essentially being rewritten from scratch so it takes time. I hope after the “brand new incarnation” of it is published, we’ll be able to address all such issues much faster but for now the only way to overcome is to either contact the “target” service and ask them to fully allow all the requests from your site’s IP or to set the links to be ignored and not checked.

    Kind regards,
    Adam

    Thread Starter Joe Westcott

    (@redredweb)

    Got it, thank you, Adam.

    I’ll mark this as resolved because based on your reply, it seems that we’ve hit the limits of what this plugin can do at this time, and our only feasible options at this time are to either:

    1. Stop checking for broken links on certain domains like Zendesk.com and other domains that use Cloudflare DDoS protection.

    2. Or find a new way to check for broken links without this plugin.

    As you mentioned, Cloudflare has an API in place, so the only feasible way to run broken link checks that are allowed through Cloudflare’s DDoS protection is to somehow use their API, either through a plugin that we write ourselves or a similar method, using API authorization that we set up ourselves.

    Which would basically mean that we’d have to write our own plugin or our own service to handle broken link checks.

    Oh well. We’ll consider the two options listed above.

    Thanks for your reply and your help, Adam.

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Zendesk Cloudflare link check error’ is closed to new replies.