Yoast SEO "Your homepage cannot be indexed by search engines"

I had the same problem, my site that was getting this message in the dashboard:

“Your homepage cannot be indexed by search engines. This is very bad for SEO and should be fixed.”

This message is actually from the WordPress SEO plugin by Yoast. I searched the forums and found several people who were getting the same issue, and who were able to rectify it by visiting the Yoast widget in the dashboard, and clicking the OnPage.org “Fetch again” button:

https://www.ads-software.com/support/topic/updated-and-now-it-says-homepage-cant-be-indexed
https://www.ads-software.com/support/topic/your-homepage-cannot-be-indexed-by-search-engines

However, there are several people who were unable to clear the message, and I found myself in the same situation. I was about to write off the error as erroneous, when I decided to check the headers using several different tools on a couple of different sites. When I did, I discovered that the site was actually returning a 503 Internal Server Error to bots, and when I went through the access logs I realized that it was doing the same to Googlebot as well.

I disabled all of the plugins and re-tested, and the error went away. I then re-enabled them all aside from Wordfence, and it stayed away, but as soon as I enabled Wordfence all of the bots, including Googlebot, were blocked again:

74.125.18.224 – – [27/Nov/2015:00:21:06 -0600] “GET /robots.txt HTTP/1.1” 503 2438 “-” “-“
74.125.18.224 – – [27/Nov/2015:00:21:06 -0600] “GET / HTTP/1.1” 503 2438 “-” “Googlebot/2.1 (+https://www.google.com/bot.html)”

I verified with ARIN that those are indeed Google IP addresses. I have no idea why Wordfence has started blocking Google IP addresses, but this will kill your seo dead if it is happening to you. Also, there are 0 IPs showing in the “Blocked IP” list, so these appear to be blocked by default and not blocked due to bad behavior.

After disabling Wordfence I was able to also successfully use the Fetch Again button in the Yoast SEO widget, and OnPage.org was able to verify that the site was visible.
Problem fixed.I hope it works for you guys too.

https://www.ads-software.com/plugins/wordfence/

This worked for me.

https://fairlymarvellous.co.uk/articles/using-yoast-yones-dont-panic/

I got the same problems after updating Yoast. ??
Tried “Fetch current status” but still same problem.
So I went to Analyze entire site as suggested by Yoast, but it’s redirecting me to Onpage.org with no solution.

So I kicked out Yoast and everything is working fine again.

@yoast: have you found a solution?

Same problem, just upgraded wp today and followed everyone’s direction and became frustraded

“Tried “Fetch current status” but still same problem.
So I went to Analyze entire site as suggested by Yoast, but it’s redirecting me to Onpage.org with no solution.”

This does not work anymore – awesome Yoast! https://fairlymarvellous.co.uk/articles/using-yoast-yones-dont-panic/

@yoast: have you found a solution?

Disable
Wordfence Security and try again Worck for me
https://www.topfullmovies.com/

I had the same Yoast error message and disable Wordfence and the problem went away. Anyone find out from Wordfence why this is happening?
Thanks!

I never heard of the wordfence issue so not sure. To me Yoast has just gone commercial now and is bloated with crap that can cause more issues. I’m not using it anymore on any of my new sites or client sites. It’s still a great plugin but when you go commercial and start partnering with third parties, you know it will undoubtedly get bloated with extra crap that’s not needed. We never had these problems even a year ago.

Hi Mellco – I turned to Github and posted a solution because of the frustration from Yost and agree Cliche Yost has gone way to commercial and bloated but still much needed unless you use all-in-one-seo: https://www.ads-software.com/plugins/all-in-one-seo-pack/

Mellco – here is a fix that worked for me: https://github.com/Yoast/wordpress-seo/issues/3686

FIX FOUND:

Step 1: Goto WP Dashboard
Step 2: Goto Yeost SEO Dashboard
Step 3: click on Fetch again at the bottom.
Step 4: Clear the error and refresh you admin panel.

Edit: Clicking fetch and refreshing the dashboard fixed the problem for me.

Hi tjportfolio

I contacted Wordfence and their support explained the problem. The interesting thing is that when I turned on Wordfence I had just received an update today from Yoast. Now with both on, the light is green index fine. /shrug.

It has to do with Wordfence seeing the company that Yoast hired to crawl sites as a fake google crawler. They do not have static IPs and Wordfence can not white list them as a result of that. If you have the block fake google crawlers option checked you could see the Yoast red error.

Great info Mecllco! Thanks for sharing.

Hi Mellco, thank you for the follow up and definitely great info for sure as I run Wordfence as well. I see that under options > Firewall Rules and do not have it checked. I started going down that road with OnPage.org but was able to resolve.

Thank you all again!

Hi all,

WordFence for WordPress is a big topic for OnPage.org. We are looking for a solution.

Mellco is right. We are using dynamic IPs. They are changing very fast and it’s possible that we crawl your website with the same Useragent but with different IP-addresses.

It’s good to know that our user-agent is “Onpagebot”.

Unfortunately there is no way to whitelist an user-agent in WordFence. Just an IP. And the IP-addresses of our crawlers are many thousand and are changing the whole time.
One way is to whitelist the whole IP-range of Amazon Webservice (AWS) Europe in Ireland. But this would open the doors for many more crawlers.

You can follow Mellco’s hint and turn off the block fake google crawlers options in WordFence. Just for a few hours. Until we finished the crawling of your domain and turn it on again afterwards.

Cheers
Chris of the OnPage.org Team

Thanks for the info Chris.

This is helpful for individual sites but when you manage over 50 sites as I do, it’s just not viable at all. Before I use to use WF and Yoast but not any longer. I don’t have time to troubleshoot every site. Until all 3 of you can create a stable longer term solution, it’s a no go for me and that’s pretty sad considering that all 3 of your products’ continue to evolve and offer more features. I know it’s not easy to always make everything compatible but short term fixes and bandaids won’t help those who own a lot of sites or have the responsibility of managing client sites. We are forced to use other alternatives. I’ll be interested to see how this plays out. Seems OP and WF could create a more long term solution that would help bridge the current functionality issues but given part of the challenge is dynamic IP’s, that’ll be tough.

Thanks again Chris.

@chris_onpage

Yoast needs to change that “Your homepage cannot be indexed by search engines” warning message to OnPage.org can not crawl your site. See FAQs about Wordfence. or something to that effect.

Then make a FAQ about how to fix it.

Ramon at Yoast customer support declined to discuss this matter with me BTW. It is interesting that Yoast has avoided commenting here.

The message came back again. My final solution for now was to turn off Enable OnPage.org indexability check in Yoast settings.

Hi Mellco,

thanks for the update.

We are currently working on a solution. But it takes time. We have to change many things in our server setup.

I will post here when we are ready ??