403 Forbidden all over

  • Resolved christel235

    (@christel235)


    5 years, 1 month ago

    3 external links on my site (e.g. https://www.socialmediaduo.nl/visuele-content/ linking to https://pixabay.com and https://www.pexels.com/) were marked as “403 forbidden”. But when I click on them everything works fine.
    I have tried changing the URL’s in those 3 links from https to http (also valid URL’s) but status stayed the same.
    Plus I have plenty (all?) internal links having ‘403 Forbidden’ as Warning.
    I don’t want to disable my security plugin, as was suggested as possible solution in another thread.
    For now I disabled Broken Link Checker. I hope this will be fixed soon in the next update. But I also welcome other advice.

    The page I need help with: [log in to see the link]

Viewing 15 replies - 1 through 15 (of 16 total)
  • Plugin Support Williams – WPMU DEV Support

    (@wpmudev-support8)

    Hello @christel235

    I hope you’re well today!

    I’ve tested it on my end and I can confirm that both Pixabay and Pexels URLs are giving “403 Forbidden” status. A view on details shows that it’s actually not even any of these services but CloudFlare they are behind.

    In other words, when your site, through Broken Link Checker, connects to these URLS, the request doesn’t even reach actual Pixabay or Pexels but “hits the CloudFlare wall” – literally. CloudFlare apparently is not able to set cookies so instead of content it returns 403 Forbidden page with a captcha (you can sometimes see in in browser too on some services) which is something plugin can’t override.

    Taking that into account, I wouldn’t say it’s a bug or a glitch in the plugin as it’s actually what CloudFlare is (among other things) expected to do – prevent “bot(ish)” automated traffic to sites.

    However, while I’m not sure if there’s a workaround for this, I fully agree it’s definitely something we should look into as it might be more and more common over time due to wider use of CloudFlare and similar services.

    I’ve already reported it to our developers so they’ll look into it to see how this could be improved in future releases.

    As for the internal links on your site. Are there any specific links that I could test on my end? If you click on “Details” link under the “403 Forbidden” status for these links – what does it say?

    Kind regards,
    Adam

    Thread Starter christel235

    (@christel235)

    Hello Adam,

    Thanks for your explanation. Now I understand why as a human the external links are perfectly fine but the plugin encounters them as not functioning.
    I still don’t understand how internal links have the same problem but the cause is probably my security plugin being not compatible with Broken Link Checker.
    To help you trying to solve this an example of one of the many internal links getting warning “403 Forbidden” plus the details. It’s on page https://www.socialmediaduo.nl/veelgestelde-vragen-online-communicatie/

    1. Link laatst gecontroleerd op: 28 januari 2020
    2. HTTP code: 403
    3. Responstijd: 0.012 seconden
    4. Uiteindelijke URL: https://www.socialmediaduo.nl/wanneer-fotos-van-internet-toegestaan/
    5. Aantal omleidingen: 0
    6. Aantal: 1
    7.
    Deze link werkte 3 keer niet.
    Deze link is al 15 dagen niet-werkend.
    1. Logboek: === HTTP-code: 403 ===

    Response headers
    ================
    HTTP/1.1 403 Forbidden
    Date: Tue, 28 Jan 2020 09:16:28 GMT
    Server: Apache
    Content-Length: 247
    Content-Type: text/html; charset=iso-8859-1
    X-NoCache: 1
    X-Varnish-Host: ip-172-16-1-103
    X-Varnish: 6619615
    Age: 0
    Via: 1.1 varnish (Varnish/5.0)
    Connection: close

    Request headers
    ================
    GET /wanneer-fotos-van-internet-toegestaan/ HTTP/1.1
    Host: https://www.socialmediaduo.nl
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36
    Accept: */*
    Referer: https://www.socialmediaduo.nl
    Connection: close
    Range: bytes=0-2048

    Response HTML
    ================
    <!DOCTYPE HTML PUBLIC “-//IETF//DTD HTML 2.0//EN”>
    <html><head>
    <title>403 Forbidden</title>
    </head><body>
    <h1>Forbidden</h1>
    <p>You don’t have permission to access /wanneer-fotos-van-internet-toegestaan/
    on this server.<br />
    </p>
    </body></html>

    ==========
    Severity: Warning
    Reason: This might be a false positive. Make sure the link is not password-protected, and that your server is not set up to block automated requests.
    ==========

    Link werkt niet meer.

    Plugin Support Dimitris – WPMU DEV Support

    (@wpmudev-support6)

    Hello @christel235

    I can’t see anything wrong in that link on the page you shared, so it should be the security plugin (or some other security tweak in the server level) doing that. Which plugin are you using? Just in case we have a quick reply here. If not, you should first confirm that via a conflict test (deactivate the security plugin and scan links again) and if it’s the root cause indeed, contact their support channel.

    Warm regards,
    Dimitris

    Thread Starter christel235

    (@christel235)

    The security plugin I use is ‘All In One WP Security’.
    But I think I’m gonna activate Broken link checker once in a while and see if new broken links pop-up or anything with another status than “403 Forbidden”.

    Plugin Support Dimitris – WPMU DEV Support

    (@wpmudev-support6)

    I’d appreciate any update on this @christel235 ??

    Warm regards,
    Dimitris

    I have that too. It started a couple of weeks ago. I reached out to pixabay about it and they said they would report it to their technical people to fix it. It doesn’t look like that is going to happen though.

    Would be great if broken link checker could do something to stop these links from being broken. Maybe have a setting where you can indicate that 403 should not report as broken?

    @jpnl – I suggested adding a setting for my issue with 503 codes. I decided to “patch” the code until the setting is implemented. Here’s what I did (change 503 to 403):

    In the file modules/checkers/http.php
    line 120

    $good_code = ( ( $http_code >= 200 ) && ( $http_code < 400 ) ) || ( 401 === $http_code ) || (503 === $http_code);
    return ! $good_code;

    Plugin Support Imran – WPMU DEV Support

    (@wpmudev-support9)

    Hello @jpnl

    I trust you are doing well!

    You can exclude the Pixabay domain from being checked. Please go to Settings > Link Checker > Which Links To Check. Into the Exclusion list field add the Pixabay domain. If you have any questions, please open a new thread in our forums, as this thread was originally opened by another author.

    @techforaging thank you for chiming in, although the issue is in this thread is different from the thread that you have created on this page.

    Hope this helps!

    Cheers,
    Nastia

    I too am starting to get 403s eg
    Broken Link Checker has detected 2 new broken links on your site.
    Here’s a list of the first 5 broken links:

    Link text : Website
    Link URL : https://www.bavariangrillhaus.com.au/
    Source : Bavarian Grill Haus

    Link text : BavarianGrillHaus.com.au
    Link URL : https://www.bavariangrillhaus.com.au/
    Source : Bavarian Grill Haus

    Link text : Website
    Link URL : https://www.bavariangrillhaus.com.au/
    Source : Red Baron Craft Brewery

    Link text : BavarianGrillHaus.com.au
    Link URL : https://www.bavariangrillhaus.com.au/
    Source : Red Baron Craft Brewery

    Link text : Website
    Link URL : https://www.tamborineweddingchapel.com.au/
    Source : Tamborine Wedding Chapel

    You can see all broken links here:
    https://www.discovertamborine.com.au/wp-admin/tools.php?page=view-broken-links
    all 403 Forbidden

    Plugin Support Williams – WPMU DEV Support

    (@wpmudev-support8)

    Hi @benwalsh

    I checked those links from my end and then were all detected as fine (“200 OK” status). Please note though: plugin runs a check from your site directly – so it’s using your server and connecting from there. A check from my end or from some other site (so from a completely different location/IP) might sometimes be giving different results.

    Are these all your sites or they are “3rd party” sites?

    Also, in the “Status” column if you hover over the status (e.g. over “403 forbidden”) you should see a “Details” link for each of them. Once you click on it, it should show you some more information about the communication with that link target. The “log:” part there can be helpful in understanding why it’s responding with “forbidden” status. Did you check that?

    Kind regards,
    Adam

    Thanks Adam

    these are all third parties and yes they are fine in browser

    Broken link checker status details are
    Link last checked: February 20, 2020
    HTTP code: 403
    Response time: 0.103 seconds
    Final URL: https://www.bavariangrillhaus.com.au/
    Redirect count: 0
    Instance count: 4

    This link has failed 6 times.
    This link has been broken for 5 days.
    Log: === HTTP code : 403 ===

    Response headers
    ================
    HTTP/1.1 403 Forbidden
    Connection: close
    Cache-Control: no-cache, no-store, must-revalidate
    Pragma: no-cache
    Expires: 0
    Server: Apache/2.2.16 (Debian)
    Content-Length: 13815

    Request headers
    ================
    GET / HTTP/1.1
    Host: https://www.bavariangrillhaus.com.au
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36
    Accept: */*
    Referer: https://www.discovertamborine.com.au
    Connection: close
    Range: bytes=0-2048

    Response HTML
    ================
    <!DOCTYPE HTML>
    <html lang=”en-US”>
    <head>
    <meta charset=”UTF-8″ />
    <meta http-equiv=”content-type” content=”text/html; charset=utf-8″ />
    <meta name=”robots” content=”noindex, nofollow” />
    <meta name=”keywords”
    content=”joomla, Joomla, joomla 1.5, wordpress 2.5, Drupal” />
    <meta name=”description” content=”Joomla!” />
    <meta name=”generator”
    content=”Joomla! 1.5 – Open Source Content Management” />
    <meta name=”generator” content=”WordPress 2.5″ />

    <meta http-equiv=”Content-Type” content=”text/html;charset=UTF-8″ />
    <meta name=”viewport” content=”width=device-width, initial-scale=1, maximum-scale=1″ />
    <title>Waiting for the redirectiron…</title>
    <style type=”text/css”>
    body {background-color: #ffffff; font-family: “Helvetica Neue”, Helvetica,Arial,sans-serif;}
    html, body {width: 100%; height: 100%; margin: 0; padding: 0;}
    span {color: #878787; font-size: 12pt; text-align: center;}
    h1 {color: #878787; font-size: 18pt; text-align: center;}
    .link {margin-top: 40px;}
    .sk-circle {margin: 80px auto;width: 100px;height: 100px;position: relative;}
    .sk-circle .sk-child {width: 100%;height: 100%;position: absolute;left: 0;top: 0;}
    .sk-circle .sk-child:before {content: ”;display: block;margin: 0 auto;width: 15%;height: 15%;background-color: #666666;border-radius: 100%;-webkit-animation: sk-circleBounceDelay 1.2s infinite ease-in-out both;animation: sk-circleBounceDelay 1.2s infinite ease-in-out both;}
    .sk-circle .sk-circle2 {-webkit-transform: rotate(30deg);-ms-transform: rotate(30deg);transform: rotate(30deg); }
    .sk-circle .sk-circle3 {-webkit-transform: rotate(60deg);-ms-transform: rotate(60deg);transform: rotate(60deg); }
    .sk-circle .sk-circle4 {-webkit-transform: rotate(90deg);-ms-transform: rotate(90deg);transform: rotate(90deg); }
    .sk-circle .sk-circle5 {-webkit-transform: rotate(120deg);-ms-transform: rotate(120
    Link is broken.

    Plugin Support Williams – WPMU DEV Support

    (@wpmudev-support8)

    Hello @benwalsh

    Thanks for response and sharing this information!

    I’m afraid though that this isn’t really an issue or a bug in the plugin. The “log” confirms that the target server rejects/denies those connections and that usually means that either it’s rejecting it by IP of your server – so e.g. based on geolocation or similar rules – or for some reasons it considers it a “bot” and blocks it.

    Sometimes this log gives a bit more information that help explain things (e.g. it shows that site can’t set cookie or get captcha solved and that’s why it rejects it) but not always. There’s always a chance that such links will be there, I’m afraid, and “code wise” we can only react to response the target server gives. A 403 is quite a strict response from the target server so I’m afraid in this case, if you are sure that those links are fine, a recommended action would be to just “dismiss” them so they wouldn’t show as broken anymore.

    On a sidenote though – we are working on making the check better to minimize such cases but there’ll always be limitations and cases that can’t be “forced” to work, I’m afraid.

    Kind regards,
    Adam

    I contacted Pixabay about this. Their response:

    We recently enabled a feature from Cloudflare called Bot Management, which automatically blocks suspicious bot crawling activities. Unfortunately, this kind of URL checking tool is likely to be blocked by this same feature.

    So as I understand from this ticket, we can exclude Pixabay from being checked. However, that means that it won’t report on actual broken links either.

    Can you work with Cloudflare to get whitelisted somehow?
    Or is there a way to only exclude links when they return a 403?
    Or have a setting to never report on 403 errors?

    Thanks
    JP

    Plugin Support Imran – WPMU DEV Support

    (@wpmudev-support9)

    Hello @jpnl

    I trust you are doing well!

    Cloudflare has a security firewall that can block bots from accessing a site. I’m afraid this is something that can’t be changed at the moment, as the plugin when triggers a link check and pinging these 3rd party links is behaving like a bot.

    You can exclude links from Pixabaym from being crawled. Please go to Settings > Link Checker > Which Links To Check. Into the Exclusion, list field add Pixabay domain.
    Please see this screenshot:
    https://monosnap.com/file/zHRYfuzS3IVhDXFuQ3ZWlVLBJn21gT

    Let us know if you have any further questions!

    Cheers,
    Nastia

    I know it behaves like a bot. But a bit has characteristics due to which they can be identified as a bot. There are good bots and bad bots. Google is a good bot, so pixabays images still show up in the search results. I believe your plugin is a good bot too, so if you work with Cloudflare you might get whitelisted.

Viewing 15 replies - 1 through 15 (of 16 total)
  • The topic ‘403 Forbidden all over’ is closed to new replies.