Googlebot is blocked by robots.txt

Check:

Admin Dashboard>Settings>General>Privacy

What is setting?

The Privacy setting is currently “Allow search engines to index this site”. I changed this yesterday.

I changed this yesterday.

Use Google Webmaster Tools to re-index and re-submit your robots.txt file. If using a caching plugin make sure to flush it.

Hi Seacoast,

Thanks for your help. Can you please explain to me what a caching plugin is and also, how would I go about flushing this if I am using one?

Thank you,
Tash

Thanks, Seacoast.

I googled ‘caching plugin’ and found out you can add a caching plugin to your website that generates static html files from the php code, improving browsing performance for people viewing the site.

I’m not using a caching plugin so I don’t have any html files to flush/regenerate.

Cheers,
Tash

Has anyone else experienced the googlebot robots.txt blocking problem when they’ve changed the site Privacy setting from “Ask search engines not to index this site” to “Allow search engines to index this site”?

Does this problem go away by itself or will Google not be able to index my site? What can I do to resolve this?

Thanks for your help,
Tash

A) Please provide a site link for us to review all this. Without, your queries are moot.

B) Server caching, when done properly with a supported and well maintained plugin such as WP Super Cache ‘records’ the generated source code (HTML) in a simple manner and sends that to your site visitors. It reduces the page load time and combined with proper local browser caching rules allows the local browser to carry the weight on retun visits.

As for Google – wait. Add good and unique content. It’s not overnight.

Thanks, Seacoast. Site link is pretty much as above without the robots.txt – https://www.recphysiomassage.com.au

I’ll try WP Super Cache too. Our page load time is very slow so this is very useful to know.

Will work on improving the content also, and wait ??

Cheers,
Tash

I have a similar problem described here
https://www.ads-software.com/support/topic/robotstxt-and-google?replies=11 and despite it being set as ‘resolved’ it is still a problem and I don’t have a solution.

@ruatuki, the file does not appear to be web readable – contact your webhost and seek their assistance in resolving this issue.

I waited a couple of days, then went back into Google Webmaster Tools. When I went to Fetch as Google & hit the ‘Fetch’ button, the status was ‘succeeded’ and I was able to submit the index.

I did nothing other than wait, although I did send a tweet with my website address referenced in it in the hope this would make Google crawl my site again sooner, so as to find the new, non-blocking, robots.txt file.

Just wanted to provide an update that waiting works.

Thanks to everyone for their help.