Random 404 Errors Caused by Yoast SEO

@illuminice

We’re sorry you’re having trouble with our Yoast SEO plugin that’s causing random 404s when accessing pages, etc. I imagine this can be frustrating for you. We’d like to look a closer look.

We expect a little more information about your setup. This might help us replicate the issue on our end:

1. Can you confirm you are using the most recent Yoast SEO, v16.7?

2. Are all your non-Yoast plugins and themes also updated?

3. Can you confirm you are using WordPress 5.7.2? Please know that if you are using an older version of WordPress Core you may experience unexpected behavior with Yoast. This guide explains more:?https://yoast.com/why-we-dont-support-old-wordpress-versions/.

4. Does your setup meet our plugin requirement?

5.

If everything is updated and meets plugin requirements, but still the error persists then we would recommend that you perform a conflict check.

Often, we see problems occur in combination with another plugin or theme. The fastest way to rule out any conflict, is to deactivate all non-Yoast plugins and switch to a standard theme like Twenty Twenty.

Please test this on your development or staging site, if you have one. If not, we recommend using the Health Check & Troubleshooting plugin. This plugin has a troubleshooting mode, which does not affect normal visitors to your site.

If you’re unfamiliar with checking for conflicts, we’d like to point you to a step-by-step guide that will walk you through the process: How to check for plugin conflicts

If you do not feel comfortable to do this yourself or if this does not solve your issue, our Yoast SEO Premium plugin comes with one year of (technical) support.

Can you inform us of the results?

Hi Suwash,

Thanks for getting back to me about this problem. I’ll answer your questions one by one.

1. Yes, we are using the most recent version of Yoast SEO, and always have been. This problem has been unchanged now for months despite always having the latest version of Yoast.

2. All of our software is fully up to date, including WP core, all plugins, and themes. We write most of our own code.

3. We’re running the latest version of WP core.

4. Yes, we meet all of your basic plugin requirements.

5. We have spent months troubleshooting this problem and do not believe it is a conflict at this point. In fact, we permanently removed several other plugins as part of this process that we felt may have been contributing to the problem. Ultimately this was how we discovered Yoast was the source.

Because the 404s occur randomly, it isn’t the easiest to test for. Sometimes you see a lot of 404s in a row, and other times you don’t see them at all. But the problem occurs very regularly (about 25% of all page visits).

Through lots of testing we found that when Yoast is disabled the problem completely goes away. We confirmed this by having Yoast disabled for weeks in our development environment. During that time not a single 404 was encountered. Literally the day I reactivated Yoast the 404s all came back.

I suspect this problem may be related to how Yoast functions in our infrastructure on AWS. We are not a typical use-case. We have a large platform in development and it uses a sophisticated infrastructure stack at AWS. This includes use of Docker, Cloudfront, and several other AWS services. I can only speculate here, but perhaps there is an issue with Yoast in AWS environments.

We have recently updated our php version as well and have been checking for all php related errors and correcting them. The php update did not resolve this problem.

…..

Have you ever seen this kind of behavior with Yoast before? Are you aware of any specific plugins that may cause a conflict like this? If so, I can check against our plugins. But at this moment I really don’t know what the cause is.

I welcome your thoughts…

~ Michael

• This reply was modified 3 years, 4 months ago by Michael Samson.

Hi @illuminice

Thank you for confirming that you are using the latest version of Yoast SEO for WordPress, and that you meet all of the necessary plugin requirements.

You mentioned that the 404s happen randomly and you are unable to exactly pinpoint the cause. Unfortunately, without any error messages or something in the server log, it’s very hard for us to diagnose this issue. This is why we recommend a conflict check, just to be sure that on a default WordPress setup on 5.7.2, with a default WP theme such as TwentyTwenty, and only the Yoast SEO plugin enabled, that the issue doesn’t occur, and then activating the plugins and themes, one by one, until the issue is reproducible – https://yoast.com/help/how-to-check-for-plugin-conflicts/

Something that comes to mind is the ‘remove categories prefix‘ feature. Are you currently using it on your site? We do know that this feature can cause unexpected URL behavior and is not recommended in most cases. Aside from that, we’re not sure what else could be causing the random 404 errors on your website.

Hi Michael,

I have my own development team and we have identified a possible cause of this problem directly in Yoast SEO. I should be able to provide you with this within the next 24 hours.

There are no errors produced when these 404s occur. It is extremely difficult to diagnose, as you said. However, we do not believe it is a conflict. I was informed yesterday of a possible cause directly in Yoast. I’ll send that over as soon as I have it.

All I ask is that you please relay this information to your development team so they can resolve the problem in Yoast core.

I’ll be in touch again shortly…

P.S. We are not using the remove categories prefix feature. So it’s definitely not that!

~ Michael

• This reply was modified 3 years, 4 months ago by Michael Samson.

Hi @illuminice

You’ve mentioned that it isn’t a conflict. It could be because of your complex set-up (as you’ve mentioned), but without being able to reliably reproduce the 404s or if there are no error messages or notices, it’ll be very difficult to diagnose exactly what is going and what we can do to resolve it.

We recommend that you perform a conflict check, and if you are able to reproduce the issue with a default setup with a default WP theme and only Yoast SEO v16.7 as the plugin enabled, we recommend creating a bug report by following the steps here – https://yoast.com/help/how-to-write-a-good-bug-report/

We are going ahead and marking this issue as resolved due to inactivity. If you require any further assistance please create a new issue.

Hello,

This issue is not solved. One of our developers who had found some of the responsible code wasn’t able to locate it again. He spent some time looking but couldn’t find it.

I do not believe this issue has anything to do with our infrastructure. To be honest we have no idea why this is occurring. We only know that Yoast is the cause. It also isn’t practical for us to test Yoast by itself in a default WP setup. We can try conflict testing more, but that’s all we can do.

At this time we’re working on a number of big projects, and while this issue is important to us, we need to return to it later. There’s a strong chance we’ll have to abandon using Yoast completely if we cannot find the cause.

As I mentioned, one of our developers did find some responsible code (directly in Yoast), but then wasn’t able to find it the second time. We’ll try to relocate that and will report it here. But we need some time to get back to this.

~ Michael

@illuminice I agree that’s very hard to debug it. I didn’t understand from previous comments:

• Are there any pattern related to 404 errors (only pages, posts, taxonomies, feeds,…) ?
• Can you send list of installed plugins? What is object cache (Redis,Memcache,…)?

I had an issue with buddypress – https://buddypress.trac.www.ads-software.com/ticket/6448, but it was very hard to find out correct root of cause.

You can see github issues/PRs (but I’m not sure that they are related):
https://github.com/Yoast/wordpress-seo/pull/17286
https://github.com/Yoast/wordpress-seo/issues/13383

I’ll try to help if you send more details…

@stodorovic

I appreciate your willingness to help. Are you associated with the Yoast team?

If there’s a pattern to when the 404s occur, I haven’t been able to see it. It seems to be random in nature, but it happens often. I think it occurs more on taxonomy pages, but it isn’t limited to them if memory serves me.

I don’t typically release the names of our plugins. Many of them we have written ourselves. It’s possible there’s a conflict, but it will be difficult to track down.

We do use BuddyPress, and this is something we can look into on our end. Thank you for sending me the GitHub issues. We’ll look at all of these.

Let me show this to my own team first and then I can report back here when we have new information. There’s definitely something strange going on.

~ Michael

@illuminice Not directly, but I’m “contributor” on Github and I’ve created many pull requests there. I know well Yoast codebase,… I’m trying to support WP/Yoast community when I have spare time.

One of my clients had similar issue (but only pages were affected) with BuddyPress/GamiPress. If same URL works, but after N hours it doesn’t work then I don’t think that Yoast is cause. Some issues on github are related to “plugin update” where “wrong rewrite rules” are saved during update process.

I’ve created custom plugin which logs all 404s (together with WP query details) + all changes related to rewrite rules. After couple weeks we were able to find exact root of cause.

I still can’t conclude more details.

When you see “false 404” did you try to “flush rewrite rules” (Settings ? Permalinks ? just click on save) and refresh same URL? Also, you can try PHP snippet from https://www.ads-software.com/support/topic/recurring-404-error-for-all-pages-but-not-posts-or-bbpress-topics/ which will do same.

You can try to temporary remove line which adds the filter pre_handle_404(the method register_hooks in wordpress-seo/src/integrations/front-end/handle-404.php ). If it works then some plugins use/add feeds on wrong way. I’ve created this class and I may help in this case.

I believe a client of ours is having the same issue, of having random pages popping up that are 404. They do appear to be found on google search results with random title tags, and metadata descriptions, but the page itself does not exist. I can’t delete them from WordPress, since those sites do not appear on WordPress sites, even though I look at all pages.
I’m currently using manual labor to delete those random 404 pages from the google search engine.

As the originator of this thread, I wanted to say that indeed, this issue has never been resolved by the makers of Yoast SEO. I did not set this thread to resolved (despite it’s resolved status).

Far from it, this same issue persists, even in the latest version of Yoast. In fact, my development team has confirmed Yoast is the cause of these random 404s many times. We have tested this issue and are 100% certain the cause is Yoast. With that said, we were never able to find the exact cause within Yoast.

We run a very large WP site on AWS. It may be this problem is caused by some kind of conflict with another plugin we use, or perhaps something outside WP itself. We just don’t know.

What we do know is when we enable Yoast we get random 404s on our site, with no observable pattern. When we disable Yoast this problem vanishes.

We’ve unfortunately been forced to abandon our use of Yoast and are coding our own SEO plugin. I wish the Yoast SEO authors had taken this issue more seriously. We really didn’t want to give up using Yoast, but we can’t operate our site with random 404s.

If anyone else is seeing this issue, I highly recommend bringing this thread back to the attention of the Yoast development team.

Good luck everyone!

P.S. I was able to set this thread back to unresolved. Maybe it will get some attention.

~ Michael

• This reply was modified 2 years, 5 months ago by Michael Samson.
• This reply was modified 2 years, 5 months ago by Michael Samson.

We are so sorry to hear that you guys are still having this issue. We are open to investigating the issue further. However, we have no idea what to look at and how to reproduce it as the issue occurs randomly and doesn’t provide any information at all.

So, if anyone could share the relevant information that would help us to deep dive into this further, we can start working on it internally from our end.

@mazedulislamkhan

While this issue does seem to occur at random, we did notice the following:

1. We have a front-end publishing tool, and noticed that the 404s would often occur when a new post was published.

2. In Yoast SEO we found code that led us to believe this was associated with “flushing rewrite rules” more frequently than normally expected.

I don’t know if this will lead to the source of the problem, but it’s all we have.

All my best,

~ Michael

Thank you for following up @illuminice. What frontend publishing tool is that you are using? How does that work with your site? Do you have the exact file name and line number for me that you believe is the cause of the issue on Yoast SEO that flushes rewrite rules?