Add Actual robots.txt or use Virtual robots.txt?

If you never plan to use an actual robots.txt file, feel free to use that option. If you do plan to use a real robots.txt file, disable the option and create/upload your own robots.txt file with the following line (use the location of your sitemap, of course):

Sitemap: https://www.yourblog.com/sitemap.xml.gz

I think it means, that because you have chosen to use the virtual robots.txt, you just need to make sure that an actual “robots.txt” file does not exist in your wordpress directory.

“A real robots.txt file must NOT exist in the blog directory!”

If it does, the real one will be used, and not the virtual file.

Thanks.

Although I haven’t “chosen” to use a virtual robots.txt file. It seems to be a default of WP.

I want to be 100% certain that it’s okay to create a new robots.txt file even though WP creates a virtual one ??

Looks like some good reading on this page, too. There is an example of an optimized robots file for WordPress about halfway down the page.

https://codex.www.ads-software.com/Search_Engine_Optimization_for_WordPress

WordPress does not make a robots.txt file unless you tell it to. I.E. set your blog as “Private, allow regular users, block bots” – else WP does not do a robots.txt file unless you use a plug-in of some sort.

My test site had a virtual keep out robots.txt file (I checked the keep bots out box) but my main site did not have a robots.txt file until I finally got tired of seeing the “file not found”errors in my log and made one myself.

Thanks & yes, read that already along with many other threads but still no concrete answers. It would seem from the codex that creating a robots.txt is fine but then what happens to the virtual? Are there now duplicate robots.txt files? I don’t want to take chances with the bots, ya know?

Does anyone have any clear answers to such issues?

Anyone have any concrete answers?

I think you would be best off using a real robots.txt… this way at least you know which one is being used and if the plugin breaks or something goes wrong, there is less problems!

Thanks but…

What happens to the virtual? Are there now duplicate robots.txt files?

WordPress’s virtual Robots.txt is only enabled if you select “I would like to block search engines, but allow normal visitors” in Settings/Privacy or if a plugin enables it via a setting like the one in the sitemap plugin.

Regardless of whether or not it’s active, most robots will take an actual robots.txt over WordPress’ virtual robots.txt.

macmanx,

Thanks but the site is set to allow all & yet there is still a virtual robots.txt file. We have never, to the best of my recollection, ever had the Privacy set to not allow bots. Hmmm? I suppose it is possible that when I initially set up the WP site, that the Privacy setting was set to not allow, but if so, it has not been that way for a year.

What do you mean “there is still a virtual robots.txt file”? If it’s WP’s virtual robots.txt file, you won’t see it. How do you know that it’s still there?

I noticed the robots.txt file in Google Webmaster Tools. I, of course, can go to the url of the robots file as well. Therefore, mysite[dot] com/robots.txt. Here is the contents of the file:

User-agent: *
Disallow:

Sitemap: https://www.mysite [dot] com/sitemap.xml.gz

I use the Xml Sitemap plugin & it has the following option, which I have ticked.

Add sitemap URL to the virtual robots.txt file.
The virtual robots.txt generated by WordPress is used. A real robots.txt file must NOT exist in the blog directory!

This explains the Sitemap reference in the robots.txt file

Strange, that’s not a virtual robots.txt file. That’s a real robots.txt file. If you’re concerned, untick the “Add sitemap URL to the virtual robots.txt file” setting with the sitemap plugin, create your own robots.txt file at mysite[dot] com/robots.txt and fill it with the following text:

User-agent: *

Sitemap: https://www.mysite [dot] com/sitemap.xml.gz

Well, I have no robots.txt file on my server & have read many posts in many threads of others who have noticed the same. Why do you say it is not a virtual? Can’t virtual mean that it is created within WP yet actually exists?

Just a thought. I really want to be certain before I create a new robots.txt. I will untick that plugin option though & see if that removes the file. Stay tuned & thanks!