Blocked by robots.txt even though they shouldn’t be

Hello,
We checked these URLs using the Google Mobile Friendly test page and we saw no blocking by robots.txt either. You can also test here: https://search.google.com/test/mobile-friendly

In this case, we suggest you remove all rules from robots.txt (since they are not blocking anything anyway) and use Fetch As Googlebot on those pages by following the instructions here: https://kb.yoast.com/kb/fetch-as-googlebot/

Strange enough. I checked again and now there are no errors. It must be a Google glitch.

Thank you for the reply.

The error came back and the site has disappeared from Google.

Here is the error page:

https://snag.gy/F9inEP.jpg

Here is the erring sitemap:

https://www.ovenandhobgoblin.co.uk/sitemap_index.xml

Here is the robots.txt

https://www.ovenandhobgoblin.co.uk/robots.txt

I am lost to think why is that happening?

• This reply was modified 7 years, 3 months ago by sarumbear.

The error shows a detected date of the 8th. Google processed the sitemap, most recently, on the 10th. If the error was still current, the last detected date would have updated to the 10th. Since it didn’t, we can safely say the error is cached and not a current issue.

I don’t think it is cache issue: Here is the page that shows today’s date, 12th. 10 pages were in the sitemap and all 10 have errors.

https://snag.gy/UFYsmN.jpg

and here is the cause according to Google:

https://snag.gy/XhoCrl.jpg

Here is the sitemap showing those 10 pages:

https://www.ovenandhobgoblin.co.uk/page-sitemap.xml

For what it’s worth, there is no cache plugin on the site.

It’s a cached error in Google Search Console not the server where your site resides.

To verify, take one of the URLs that shows as an error in the sitemap and run in through the live robots.txt tester in Google Search Console (here). Does it show a block still?

The sitemap errors has now disappeared, again, and shows no issues/errors. This has happened before but the errors re-started to show, hence this post.

I will monitor the site and report back here if anything else happens again.

Thank you for your guidance so far.