Why has Google stopped indexing my site?

Hello @dbreithart

Thanks for reaching out about your robots.txt file. Was it the robots.txt Tester that reported the error? If not, have you tried it?

The robots.txt is the first file the bot looks at when it arrives at your site; if it can’t access it, it goes no further. ?

Both the mobile-friendly and rich result tests are based on the same infrastructure as the bot. If these can’t reach the site, then you are blocking Google – Failed: Robots.txt unreachable

Since this is the case, you’ll need to look at your server configuration, DNS, Firewall, and CDN (if you use one).?Please speak with your web hosting provider.

Do let us know.

Yes, the robots.txt Tester tool on Google Search Console is what gave me that error.

I don’t have a firewall or CDN active. My hosting is currently through Bluehost and they say nothing is blocking Google on their end. Do you have specific ideas for what I could look at or get them to look at in regards to server configuration or DNS? Because I’ve went back and forth with them a few times now and they haven’t been super helpful.

Thanks for the feedback, Devon.

I noticed you have a physical/static robots.txt file, while WordPress also generates a dynamic or virtual one.

We currently have it in our product roadmap to fix these two disparities. In the meantime, a workaround is to copy the contents of the virtual file (?robots=1) and merge the directives with the contents in the physical file (robots.txt) on your server. Next, test in the robots.txt Tester and let us know how it goes.

It seems like something made my site visible to Google again and it started indexing some of the pages on my site, but not all. Any ideas for getting it to index the other pages?

Can you share some of the URLs that aren’t indexed yet?

There are a lot, but here are a few:

https://devonbreithart.com/love-about-school-based-ot/
https://devonbreithart.com/ieps-504s-school-based-ot/
https://devonbreithart.com/ot-serve-speech-only-ieps/
https://devonbreithart.com/when-school-based-ots-discharge/
https://devonbreithart.com/common-elementary-school-ot-assessments/

Thanks for sharing. Only https://devonbreithart.com/when-school-based-ots-discharge/ and https://devonbreithart.com/love-about-school-based-ot/ aren’t indexed yet. They’re also both in the Yoast SEO-generated post sitemap. And there’s nothing in the robots.txt file that may be blocking crawling or indexing.

Since you’ve?added your site to Google Search Console, then submit Your Sitemap, and you could even?request indexing for those two URLs. If there are any errors reported, do share them with us.

Hi,

Thanks for letting me know that some of the URLs got indexed. I’m still having issues with many pages returning the “Crawled – not currently indexed” error. I did request specific indexing for those two URLs in addition to asking them to redo all of the other URLs falling under that error. I still have hundreds of pages stating this.

The crawled – currently not indexed notification suggests that there isn’t a problem on your site but that Google was just taking some time to fully crawl and index your site, which is perfectly normal.

It’s not actually normal considering these pages were all previously crawled and indexed and they randomly stopped for some reason. And I still have hundreds of pages that I have asked Google to re-crawl and re-index and it still hasn’t for some reason.