robots.txt and google

Maybe your FTP client is not overwriting the file? Another possible reason is that your site’s web server is using a cache…. file is not updated on time (ask your host).

Thanks for the help David. There was no robots.txt file in the root folder before I uploaded one. My research suggested it could have been a dynamically generated one and that uploading a new, corrected file might solve the problem but it appears not.

Then what if you actually edit that file, using the File Manager from your host’s panel (like the cPanel, DirectAdmin, etc.)

The robots.txt file that is now in the root folder was the one I created and uploaded I’m not sure what editing it will achieve. It is the only robots.txt file that has ever been there. It contains “User-agent: *
Disallow: /wp-content/plugins/”

Checked the link

https://www.anact.co.nz/robots.txt

but it’s still

“User-agent: *
Disallow: /”

so I think it’s the cache problem from your web server. See what your host support has to say.

If I go to https://www.anact.co.nz/robots.txt then, in Firefox go tools>page info it says “Modified Friday, 3 November 2006 2:20:08 p.m.” whilst the one I uploaded has todays date. Suggesting it is dynamically generated?

Maybe you are editing the wrong robots.txt file?

Try a test, create a file called

test.txt

and put something in it, see if you can access it directly by

https://www.anact.co.nz/test.txt

Yep that worked.

Then indeed, I think it’s replacing that file internally like you said. Only your host support can solve the problem.

The robots.txt file in the root folder does not match the result I get when I go to https://www.anact.co.nz/robots.txt. The problem is not resolved.

OK, maybe my post ‘yep that worked’ was misleading, I meant ‘yep that worked I can upload a test file no problem’ NOT yep that fixed my robots.txt problem. The original problem remains.

Hi,

Please refer to my post here:

https://www.ads-software.com/support/topic/googlebot-is-blocked-by-robotstxt?replies=12#post-3129828

I waited a couple of days, and it seems Google has crawled my site again because it’s no longer saying the robots.txt file is blocking site indexing, even though I changed nothing.

Waiting seems to work.