XML sitemap X-Robots-Tag: noindex, follow

Sometimes resetting the permalink structure will quickly solve a 404 error from a recent change to your installation. Go to Admin > Settings > Permalinks and click save without altering anything. (How to Guide available here.)

If the issue remains, please try the suggestions here: My sitemap index is giving a 404 error, what should I do?

Thank you for your response.

It’s not giving a 404 though.

When we check the header we see “X-Robots-Tag: noindex, follow”

I’ve followed the instructions anyway, and the issues remains.

https://beportugal.com/sitemap_index.xml

To add to this, Bing has fetched the sitemap successfully and the old console says

Sitemap: /sitemap_index.xml
Type: Sitemap index
Number of children in this Sitemap index 4
Error details: No errors found.

Don’t you just loooooove Google? ??

• This reply was modified 6 years, 3 months ago by zoldor.
• This reply was modified 6 years, 3 months ago by zoldor.

It’s standard header for sitemaps (and you can’t remove it) because they aren’t shown in google index (their self URLs). Similar is for robots.txt. So, it isn’t cause of your issue.

Also, by quick checking it seems that sitemaps are correct (Content-Type, XML validation, URLs,…). Maybe you could try to temporary exclude sitemaps from caching. Did you try resubmit in GSC? Did you check in old version of GSC – Is it same error?

Please check server logs (Apache, Nginx, cPanel, …). Sometime some security rules could block search engine to access. I had couple issues related to mod_security.

When I loaded the sitemap yesterday, it was showing a 404 not found error. Now, the sitemap loads correctly.

The sitemap should have a X-Robots-Tag: noindex, follow as you do not want Google to show the sitemap page in search results. The follow is what allows Google to process the links in the sitemap for inclusion in search results.

What is URL of the Google Search Console profile where you submitted the sitemap? It should match EXACTLY to https://beportugal.com/. If you submit it to a profile using http and/or www, you may see unexpected errors.

Hello, thank you both for your responses.

@stodorovic -> I’ve done that, but it didn’t work (excluding from caching). Old version of GSC says that there are no errors -> it even returns ‘Number of children in this Sitemap index 4’ which is correct.

@amboutwe thank you for suggesting to dig in this direction. I think I’ve found one potential explanation. We use Cloudflare flexible SSL option. I.e. our digital ocean droplet doesn’t have SSL set up and all the requests go through Cloudflare.

https://github.com/Yoast/wordpress-seo/issues/3877

I’ve used the snippet.

Will see if it works.

• This reply was modified 6 years, 3 months ago by zoldor.
• This reply was modified 6 years, 3 months ago by zoldor.
• This reply was modified 6 years, 3 months ago by zoldor.
• This reply was modified 6 years, 3 months ago by zoldor.
• This reply was modified 6 years, 3 months ago by zoldor.

Update:

The snippet has worked. Thank you both again for your help ??