Intermittent 500 Internal Error Messages

Hi folks,

We’re having issues with sites displaying intermittent 500 errors to users, and the set of circumstances surrounding these errors makes diagnosing them extremely frustrating. Replicating the issue requires no more than visiting a site, but it happens so infrequently, you might have to refresh 1,000+ times (not an exaggeration) to see it.

1) Intermittent = one or two instances per day, with 99.9%+ traffic operating fine. Sometimes it’s happening once or twice a week – very hard to pin down.

2) It happens on multiple hosts. We run our own servers, and we’ve received complaints for three sites we host. We’ve also received complaints for a site we manage but is hosted by a 3rd party. It’s not limited to our server configuration.

3) It will fail in one browser, and immediately work fine in another a few seconds later. There is no browser pattern (it will fail for someone in Firefox and work fine in Safari, fail in Safari for someone else and work in Chrome, etc.).

4) Nothing shows up in hosting account error logs, and nothing shows up in WP error logs with WP DEBUG enabled.

5) The sites that have reported the issue receive anywhere from 100 to ~1,500 visitors per day – enough that over time the issue will appear – not enough to stress a server, particularly with caching in place. We limit servers to 50 accounts each, and they rarely surpass a load of 2.00 (typically run around 1.00 on 8 cores). Rarely is more than 1/2 of the server’s memory in use. IOW, the server is not choking on requests.

6) All themes are custom-programmed. Each one is completely unique, built from the ground up. There isn’t a theme base that’s consistent between them, with the exception of some common functions used here and there on most sites we build. We never use deprecated functions, and all sites experiencing issues are less than 2 years old.

7) Every site that’s reported an issue is running WP 4.8. We’ve held off on upgrading most accounts we manage to WP 4.8; we tend to leave a little lag time, typically a week or two, updating major and minor releases (4.x to 4.x – we allow micro-updates). We’ve only upgraded about a half-dozen sites, and only these have reported issues.

8) Common plug-ins we’re using across all sites: iThemes Security Pro, BackupBuddy, iThemes Sync, Gravity Forms, WP Rocket, Yoast SEO and Advanced Custom Fields Pro. These are all very well established, mature products with strong support teams and histories of handling bug/fix releases very quickly. While a plug-in could be the problem, I can’t find a single relevant complaint among their support forums anywhere that matches our issue.

9) We’ve tried recreating .htaccess files in case the issue is file corruption. This did not improve the situation.

10) It may be happening to all of our sites, but traffic is low enough that it’s not happened yet, or is not being reported.

Replicating the issue requires a steady volume of requests and anywhere from a day to a week to replicate. Creating a test account eliminates the traffic, and turning off plug-ins or switching themes for up to a week on a heavily trafficked site is not an option. This makes troubleshooting maddening.

I’m hoping someone else has seen a similar issue and stumbled on a solution they might share, or can give me additional pointers for trying to troubleshoot or solve this issue. Any assistance would be most appreciated.

Thanks!