Robot.txt file

Hey there,

Could you elaborate on where/how you are seeing this message:
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php”

Also, do you have any plugins activated in your WordPress? If so, could any be related to SEO?

P.S. I see the robots.txt located here:
https://www.ocscreeding.co.uk/robots.txt

This essentially means that the robots.txt file is sitting in the “root” directory where you WordPress is installed. I am not sure how it is being generate though, so you will still want to check for plugins that might be generating that.

Hi,

Thanks for coming back to me. The plugins I have active are Add Meta Tags, Akismet, All In One SEO Pack, Contact Form 7, Dropdown Menu Widget, iThemes Sync, NextGEN Gallery by Photocrati, BackupBuddy and WP-Mail-SMTP. I am guessing from what you wrote that the root directory will be in the All In One SEO Pack? How can I look into this plugin, is it in settings maybe? Thanks again for your help

Hey there,

The root directory actually refers to your hosting file structure. This isn’t something that is a part of any plugin.

Basically, log into your webhosting account through FTP and browse to the “root” WordPress directory. This directory contains files like “wp-config.php” and folders like “wp-content” and “wp-admin”. You should see “robots.txt” listed in the same directory as those files.

It’s possible that All in One SEO Pack tries to generate the robots.txt file for you. Are you able to disable that plugin, and then deleting the robots.txt file in your root directory (as described above)?

Hi again, I had a look at the All In One SEO pack and I saw in the Feature Manager that it has a Robots.txt section to generate and validate a file. In your opinion do you think it would be beneficial to activate this or would it confuse my situation? Thanks

Hi, sorry crossed messages there! I had a look on Filezilla where I have copies of my host files but I didn’t see any named robot.txt file. I will contact them tomorrow and double check. If it isn’t there do you have any idea where else it could be? Thanks for your help.

With the way your robots.txt file is set up, I don’t see any issues with it blocking anything so that is kind of strange. If Google says it is having an issue, though, it’s good to listen to that if you want to rank on Google. Try disabling the WP SEO plugin and rechecking with your Google to see if the robot.txt issue gets resolved.

Hey there! Just to add to what I think you might be seeing if I could..

Googlebot cannot access our CSS and JS files because of restriction in the robot.txt file. I have been trying unsuccessfully to find my robot.txt file

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

That’s what you would generally see if you are using the WordPress generated virtual robots.txt file. That’s usually the result of allowing search engine visibility in your General Settings – so if you don’t find a robots.txt file in your public_html directory, that might be why.

As @johnstonphilip noted, the default robots.txt virtual file doesn’t seem to be blocking the CSS and JS files the Google message is referring to. In fact, I believe the addition of Allow: /wp-admin/admin-ajax.php to the file is intended to address just that issue. Some discussion that seems to be on point: https://core.trac.www.ads-software.com/ticket/33156

Also some good reading material on Robots.txt Optimization for WordPress.

I’m not sure how your SEO plugin wants to handle any additions to robots.txt (if it wants a virtual or real file) but if you actually create a robots.txt file and place it in your root directory, it will automatically override the default (virtual) file.

Just some information I though you might find, well.. informative. ??

I hope something I mentioned helps in some way.

Hi Johnstonphilip and Clayton James, sorry about the delay but I was away. Thanks for your feedback which is really appreciated. I have looked through my hosts files and can not see a robot.txt file so I believe there must be a virtual one. I deactivated the All in one SEO plugin, went to Webmaster Tools fetched and rendered, had the site recrawled but it does not seem to have changed the message. I think my main problem is that the site is not mobile responsive so I will undertake updating this and hopefully this will solve the problem. Thanks again for your help

I was getting this error and really it took toll of my nerves

Finally I found a solution and I suggest either of these works

1. If you use plugin called “Google XML Sitemaps” then uncheck “Add sitemap URL to the virtual robots.txt file.” under Basic Options. I have a line for the XML Sitemap in the Robots.txt file so I am not using this option.

2. Try installing some virtual robots.txt editor plugins. I have used WP Robots Txt and then edited the file. It worked!

Please note: I still have the robots.txt on my root. I just copied the text on WP Robots Txt plugin and it worked! Gosshh

[I will post the same answer to some of the similar questions.. so those who are in same situation may get instant relief]