Random 404 Errors Caused by Yoast SEO

@mazedulislamkhan

The publishing software is part of our platform (iazo), which is launching in a few weeks. It’s custom code, and not directly the cause of this 404 issue.

I can ask my engineer about the line numbers he suspected. He’s the person who told me about this.

I’ll get back to you with that information, but I should say this was only a suspicion. Yoast is a large and complex plugin, and we’re certainly not familiar with its code base.

I wish I had better information for you. This 404 issue is really nasty.

~ Michael

I would like to add that I experienced a very similar problem on my client’s website. They started getting random 404 errors on two custom post types, events and news. The 404’s would occur randomly on old and new posts multiple times per week. The 404’s would resolve themselves within 1-2 minutes. It was very strange. This went on for months. I tried everything I could think of to solve the problem, resetting permalinks, updating all plugins, disabling caching, etc. I then came across this forum post and tried deactivating the Yoast plugin. The 404’s immediately stopped happening. After several weeks I reactivated Yoast and the 404’s returned. I do not have the expertise or bandwidth to explore the problem further but it was very clear that it was caused by some interaction with the Yoast plugin.

@mgnewton Sounds like the identical problem that we identified. I’m glad that our thread here helped to you to know Yoast was the cause. We spent a lot of time testing before we narrowed it down to Yoast.

@mazedulislamkhan From what I can see here, you have a very serious problem in Yoast that is causing randomized 404s for many users. I wish I could provide you with more specific information, but we simply don’t have it. All we can see is the result of this problem; endless random 404s.

We made a decision here as a company to abandon using Yoast completely. Since we have our own development team, we are writing our own code to cover the features we need from Yoast. This problem is simply too big a vulnerability, and we don’t know the intricacies of the plugin enough to find the trigger for the problem. We’re better off not using Yoast and writing our own code.

With that said, I highly recommend you look more into this. Perhaps some of the other users here like @mgnewton can let you into their sites to see the problem directly. This is not an option for us as we have developed a proprietary platform.

Good luck everyone!

~ Michael

Finally I’ve discovered the issue with this weird thing. Yoast SEO has an option to auto remove the initial slug from categories (normally, it’s like “/category/“) and this is a nice and useful feature, BUT, when using WooCommerce there is a big problem, caused Yoast, and it’s that woocomerce base slug is the shop itself. So if the yoast plugin tries to remove the slug before the product category name, it’s removing the shop base name.

This causes racing conditions where sometimes works and others it doesn’t (it depends also from page cache plugins installed or not).

So it should be solved by yoast. The option to remove the base category slug should be intelligent enough to detect if its a woocommerce page or not.

For example:

Categories are like this:

https://www.domain.com/category/mycategory -> YOAST -> https://www.domain.com/mycategory

And that it’s fine. But let’s see what happens with WooCommerce:

https://www.domain.com/shop/shirts -> YOAST -> https://www.domain.com/shirts

And that’s WRONG cause products should be in that case: https://www.domain.com/shop/shirts/cool-blueshirt

And Yoast sometimes removes the slug and not others, causing problems with cache and 404 errors randomly. Yoast shouldn’t remove the shop basename, like it does when there is a normal post category. If it’s WooCommerce or similar plugin, the slug shouldn’t be removed.

And that’s also the reason why it sometimes solves by going into permalinks settings in WordPress and just saving.

• This reply was modified 2 years, 4 months ago by fidoboy.
• This reply was modified 2 years, 4 months ago by fidoboy.

@fidoboy

I just wanted to point out that the issue you have discussed above is an entirely different issue than the one my team identified. It really belongs in a separate forum post.

The random 404s this thread is about have nothing to do with the feature in Yoast that removes the categories slug. It’s also completely unrelated to WooCommerce.

I’m mentioning this to help others avoid confusion when reading this post.

~ Michael

May be that this is also related:

https://core.trac.www.ads-software.com/ticket/24612

https://core.trac.www.ads-software.com/ticket/13459

As far as I know, there is no way to check the permalinks structure to look for duplicates that potentially could be causing 404 errors due to racing conditions.

Regards,

• This reply was modified 2 years, 4 months ago by fidoboy.

@zluck @mazedulislamkhan

Love how they keep marking this thread as “Completed.” What a joke.

We were never able to solve this problem. When Yoast was active we had random 404s across our platform, and when Yoast was disabled those 404s disappeared. Yoast was the cause, but we never discovered the reason why.

We’ve since removed Yoast and written our own customized SEO plugin. Tbh we needed to do this anyway as we have particular needs and requirements.

I wish you the best of luck with this!

~ Michael

@illuminice I understand that you hate seeing the thread mark as closed even when the issue isn’t. As you have already seen in the past, we tried to reproduce the issue and are still open to doing it. However, there are no exact steps that you were able to share, which makes it extremely hard for us to reproduce the issue on our end. If we can’t reproduce it, finding out what’s causing the issue is challenging.

That said, we close a thread after a few days of inactivity to keep the overview. However, that doesn’t mean we forget the issue or are not interested in offering any assistance. If you can still share more information on how to reproduce it, we can deep dive into this again.

This issue has seemed to crop up on one of our websites after updating to Version 20.

Inexplicably, randomly and without consistency we would receive 404s or redirects when trying to access standard content such as a:

• Post
• Category
• Product
• Tag
• Page
• Custom Taxonomy

Revisiting the same URL would sometimes cause the issue to not re-appear and then would suddenly crop up a few minutes later and then a few minutes after that everything would be fine.

Flushing the permalinks multiple times did not yield success.

There have been no other changes to the stack other than updating Yoast. We have also been unsuccessful in running the SEO data optimization if that is a relevant piece.

If it continues again I am happy to hook into whatever action/filter is necessary to trace this.

This issue is clearly not resolved yet.

@thebaumgartler My company isn’t using Yoast anymore. We developed our own SEO plugin. But I’m not surprised to see this issue persists. It is very insidious.

When we were dealing with this we were able to establish that Yoast was the cause by disabling the plugin and retesting. The issue only occurred when Yoast was active. It seemed to occur fairly randomly, and only on certain types of pages. It also occurred with great frequency. In our case the result was always a 404, and not a redirect.

I wish I could share more insights with you. I only know Yoast was the cause, or at the least, the trigger for it.

I hope you can get to the root cause, but it’s probably very difficult to determine.

• This reply was modified 1 year, 8 months ago by Michael Samson.

@illuminice I appreciate your response! We are using the premium versions and there have been 3 instances in the last year where Yoast has caused crippling issues to our website.

Sure enough, deactivating Yoast not only seems to have suppressed the 404 errors but we have noticeable performance gains.

I am just hoping that opening this up will get the Yoast team looking at it. I am now exploring alternatives such as RankMath or developing an in-house solution so we can have complete control and confidence in our stack.

@thebaumgartler We were actually using the free version of Yoast at the time. Having random 404s was a terrible issue, and it ultimately pushed us to develop our own SEO solution. I’m fortunate enough to have my own developers.

Yoast is a great plugin, but it is also very large and bloated. This is the case with most all-in-one plugin solutions. It’s not really their fault; they’re simply trying to provide the best tool for every use-case. But all that extra functionality and code can lead to unintended interactions, like these 404s.

When we actually examined what we needed out of Yoast, the requirements were very specific. So we developed our own plugin to meet our exact needs. I should mention this is for a large platform, and we do develop most of our own code. For the common user this wouldn’t be an option obviously.

Anyway, I feel your pain on this one. My guess is this is some kind of conflict. The problem is trying to pin that down. It is probably impossible without Yoast helping directly, and even then will be difficult.

I wish you the best of luck with it!

P.S. Developing your own SEO solution would be my recommendation, if that is within your power.