CRITICAL ISSUE- 520 Error – The Invisible Killer (Cannot Reach Origin Server)

  • Resolved mystyleplatform

    (@mystyleplatform)


    1 year, 9 months ago

    Hi,

    I wrote this out more or less the same on the Slack channel and nobody responded.

    I love this plugin and Quic Cloud CDN service speed but there is one singular absolute deal-breaker issue, recurring intermittently, flying under the radar which is the 520 errors happening without being able to know where or when they happen.

    When the 520 Error happens, it’s an error page that says the site can’t be reached. The site is down basically, but only for certain regions. It’s usually because my hosting company has blocked the IP of a regional server, in my case it’s usually the server NA-US-SJC-81 but it could be any server/location or more than one. This means that entire regions of the country going thru that CDN server basically cannot reach our site for long periods of time (weeks/months), the site is effectively down, even through it looks up for us from our location, and we lose 100% of sales in the effected region(s) during this downtime, which can be a huge loss.

    What’s worse, there is no way to know when this happens to respond quickly – no warning that it’s happening – no notification in our account dashboard – no SMS or Email notification sent, nothing. It simply quietly dies, as does our revenue for that entire area, and we unknowingly bleed losses until by some chance we realize it. In one case, it was because we were running Google Page Speed Insights, which must have used this CDN server, and was coming back with the unreachable error page instead of our homepages. Since our support for our users is via our website, none of our users in that area can where it’s happening even tell us that it’s happening, they just get lost as a customer. It’s a brick wall of down time and lost customers with no warnings and no visibility.

    This is an invisible killer.

    We can’t see it happening, we get no warning it’s happening, and all the while it’s just absolutely killing our service/sales for whole cities of people.

    Seeing how the entire livelihood of our websites is put in the hands of your servers being up and able to reach the website to serve it, I would think that knowing when that primary function is failing is pretty important to all users. I also have to imagine that right now, many users have no idea that it’s happening, but it’s happening to them as well, as we did not know either for our entire holiday rush season, where we incurred [tens of] thousands of dollars in losses with no warnings. As far as our Quic Cloud dashboard goes, our credits are on auto-refill and everything is running fine, when in fact it’s horribly broken for millions of potential users.

    We (CDN Customers) desperately need visibility on this issue to continue using the service without incurring huge losses. We’ve already lost thousands and thousands of dollars in sales to regional downtime that we cannot see from our location – and nobody seems to care? Support’s ultimate answer is basically, then don’t use the service or switch hosting companies. Isn’t there a better solution to just provide visibility and/or notifications on this issue, rather than sweep it under the rug and hurt your customers business or lose customers?

    Even right now, I have no way of knowing is my website up or not at every location for more than 10 active websites. I’ve signed up for Pingdom and tried other Geo-Located uptime monitors as an attempt to get my own visibility on the issue, but that’s not seeming to catch the issue either while it’s happening for some reason.

    There are so many ways you could address this which are not complex, either bringing visibility for users, or all out preventing downtime. Such as:

    1. Notification in the dashboard when 520 errors are occurring letting us know a regional server is broken.
    2. Email / SMS Notification of the same.
    3. Allow users to customize their own Error 520 page so that we could add our own google analytics event, or pixel that would track this error
    4. Provide an error log in the dashboard somewhere (like most hosting companies have) so we can see all errors that are happening
    5. In the case of a 520 Error, have the server check with the next closest regional CDN and route through there instead (eliminating downtime!)
    6. A simple diagnostic tool that can run all of the active regional CDN servers and check if they can reach the origin server OK with a checkmark or X green or red for each server.
      • This is what Support does manually for me when I report this on Slack, and then I can quickly resolve this with the hosting company. But Quic Cloud Support says there is no way for us (CDN customers) to check this for ourselves, so the response time on this issue ends up being really bad (weeks? months?).

    Please provide a way to get visibility on whether or not the CDN is down regionally for any given website using your service, because otherwise your service is by far the fastest, and best I’ve seen for optimizing WordPress.

    I’m more than happy to stop paying 3rd party uptime monitors and pay for it within the Quic Cloud dashboard if that’s what it requires to simply get a report of which of our enabled CDN servers are OK, and what servers are down/broken connections to the origin site.

    I even have some enterprise level customers that I’d love to bring on the service, but I simply can’t knowing that we have uptime SLAs and this is going on. More lost business for Quic Cloud, too, so it’s not just affecting my business, but it’s preventing me from bringing clients to yours as well.

    Please help!

    • This topic was modified 1 year, 9 months ago by mystyleplatform. Reason: More suggestions for solutions
Viewing 2 replies - 1 through 2 (of 2 total)

  • Hello Mystyleplatform

    We apologized for the inconvenience.

    If you have 520 issue, could you create a ticket on QUIC.cloud support

    https://www.quic.cloud/support/

    The suggestion of “There are so many ways you could address this which are not complex, either bringing visibility for users, or all out preventing downtime. Such as:” is very important to us

    We will discuss it internally

    Most 520 issue is caused by the host not properly responding to CDN request. If the main cause is related to firewall, security plugin, or heavy loading, the 520 error may occur in a short time. Currently, we can only inform which CDN nodes are unable to reach the host. and share the CDN log with you.

    All the best

    Wade

    Thread Starter mystyleplatform

    (@mystyleplatform)

    Hi Wade,

    Yes thanks for responding, you’re the same person that responded to my most recent support thread on the slack channel.

    You’re right that it’s an issue caused by the hosting company’s firewall, but it’s a 2 part issue, one part is them detecting and blocking bots, and the other part is your servers hitting their servers to crawl and cache files, seeming like a bot and thus getting blocked in one location.

    Since your service is offered as something that works with third party hosting companies, and those companies represent literally millions of potential customers, this is part of the nature of your service to need to be able to mitigate issues with hosting companies and their firewalls when they occur.

    When this occurs, I can contact my hosting support with the IP address of the regional Quic Cloud CDN server that is not able to reach the origin host, and I can resolve it myself without needing your support staff to step in.

    The only problem is that only your support staff and your systems know when a regional server cannot reach an origin website. We go weeks or months with a huge chunk of the country not able to reach our website and we just simply don’t know.

    So, even if I do open a ticket on Slack or thru Support tickets, this is a delayed response days, weeks, or months too late. If an alert notification is sent to me when it occurs, I can respond with my hosting company and resolve the issue within minutes or hours.

    So, for now we just need to know when it’s happening and what server / IP it’s happening on.

    I think the easiest way to do this for now would be email notifications or SMS notifications (or both since they’re almost the same thing) as an option in every Quic cloud account settings. Then customers can handle it from there and it wouldn’t cost too much resources to send out the notification email.

    That said, if the server system was updated to catch 520 errors and in addition to sending a notification to the account holder it would also then ping the next closest CDN server to see if that one is hitting the website OK, then it could proxy through a nearby regional server and continue to reach the origin server that way, then downtime could be avoided all together, but the customer could still be notified that a regional server was having issues and cannot reach the website etc and can take it upon themselves to resolve it, while the broken node continues to proxy thru an alternate node to keep the site alive in the meantime.

    Notifications could be queued and limited to once an hour, or even once per day, to throttle it to something reasonable.

    I tell you if you can make this happen then your service will be flawless for me and I will bring all of my WP sites and enterprise client’s sites onto it.

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘CRITICAL ISSUE- 520 Error – The Invisible Killer (Cannot Reach Origin Server)’ is closed to new replies.