Yoast blocking search engines

Hi Steve,

That’s really strange and we understand your concern. Yoast SEO doesn’t make any changes to your robots.txt file, so we’re not sure how this is happening. Can you give us more details about any other SEO plugins you’re using or a link to the robots.txt file so that we can take a look?

The blocked by robots warning message in the Google Search Console, doesn’t always refer to the robots.txt file. It could refer to the robots meta tag or robots HTTP headers. There are also some URLs that should be blocked from appearing in search results like your sitemaps or your admin pages.

Can you give us an example URL mentioned by Google so that we can investigate the problem?

Hi Priscilla,
Thanks for getting back to me about this.
Only I have access to my sites and I did NOT go into the robot.txt file and change it.
The file had a slash added (which I did NOT add!), telling search engines not to index the site – the ENTIRE site, not just admin pages. ALL my live pages are being ignored, which make up the majority of those 92 pages that are affected.

Even though that file has now been corrected and my host says the robot.txt should be fine, Google is still not indexing it, hence the error message.

For SEO, I am using Yoast. That’s it. The only other plugin I use that interacts outside my site is Slimstat for analytics. Everything was working perfectly up to two weeks ago, then traffic started to fall off from 600 per day to under 100 now.

To try to isolate the problem to Yoast (as it must have been yoast that changed the robot.txt), I have temporarily deactivated it and installed Rank Math. Now, if my traffic goes back up, that proves Yoast is the problem. (I only ever use one SEO plugin at a time, so duplication is not the issue.)

Here is a page:
https://seriesbooksinorder.com/tom-clancy-books-in-order-jack-ryan-series-in-order/

Thanks again for looking at this,
Steve

@stevenw123 Thanks for your reply and for providing a link. However, if the Yoast SEO plugin is deactivated on the site, none of the it’s settings would affect a page so we unfortunately can’t see whether a ‘noindex’ tag was being output for a page by Yoast.

If you want to test whether this is the case, it would require reactivating the plugin and then viewing the source code of a page and searching for ‘noindex’ in the code.

As for the robots.txt file and the site’s traffic, if your site was being blocked globally and no longer is, users being able to find the site in Google would be the cause of the traffic going back up, not the use of the plugin. There is no setting in the plugin to disallow indexing on the entire site. There is a setting for that in WordPress itself, independent of the Yoast plugin, but it doesn’t block access to the site using robots.txt either. Furthermore, Yoast SEO will actually warn you if that feature is enabled to let you know to turn it off. The only time Yoast SEO will ever write to the robots.txt file is if you specifically edit the file manually using the File Editor feature, and then save that edit. This is the reason we were asking if you have any other plugins that are SEO related.

Also, it isn’t just plugins that can edit a robots.txt file, as it is a physical file on the server. It’s also possible to be edited using any type of editor that has access to the server, such as a FTP application, or file editor or SEO features available from the hosting control panel for example.

Carl,

“users being able to find the site in Google would be the cause of the traffic going back up, not the use of the plugin.”

Yes, but they’d only find the site in Google because Google believed the site was functioning properly and so displayed it in search results. Yoast is NOT letting that happen if it is responsible for the robot.txt file being changed. And who else could change it???

The only way I know to access the robot.txt file is through yoast. And the only reason I know that is because I investigated it when I discovered this problem.

Why would I or someone at my hosting company go into that file and add a slash? They wouldn’t. No one has manually added anything to that file.

And as I said, I don’t use any other SEO plugin, so the proof will be in the pudding – I’ve changed to Rank Math. If Google can now crawl my site properly, then yoast is to blame.

Steve

@stevenw123 We’re sorry that you experienced this issue, and agree that the directive in the robots.txt file definitely caused the issue. However, the Yoast SEO plugin doesn’t write to the robots.txt file unless you use the file editor in Yoast, and save the file. However, that is the same ability to edit the file as anything with write access to the site is able to do. We didn’t mean to infer this was an intentional edit of the file. Can you please let us know what let you to believe that the Yoast SEO plugin was what edited the file? As I mentioned in my previously reply, any application (including but not limited to plugins, FTP applications, etc) can write to the file that has write access to the site.

We are going ahead and marking this issue as resolved due to inactivity. If you require any further assistance please create a new issue.