Virtual robots.txt missing

WordPress is supposed to generate a robots.txt…
….On latest 4.4. pristine install I cannot find it.

That’s because it’s a virtual file. Create a robots.txt file and upload it to your WordPress root directory. That file will then automatically override the virtual file.

Hello,

Thanks for your reply.

I understand that it’s not an actual file.

When I say I cannot find it I mean “https://<mysite>/robots.txt&#8221; returns “Not Found”

What I don’t understand is how is it generated e.g. is it some sort of web server rewrite rule ?

I don’t mind writing my own robots.txt , no big deal, except e.g. the “Googla XML Sitemaps” plugin have an option to add sitemap URL to virtual robots, etc.

Regards.

Do you have your WordPress files installed in a sub-directory?

Linux + Apache + VirtualHost on localhost

It is installed under the same dir as defined by the DocumentRoot directive.

WordPress 4.4 plain vanilla install, out of the box.

Thanks,
Cheers.

It is installed under the same dir as defined by the DocumentRoot directive.

Installed under the same directory as the document root? Or in the document root?

When you open the site in your browser, is the address https://localhost or https://localhost/some_folder_name ?

Here’s what I believe to be true:

When WordPress is installed in the root directory of a domain, and the “Discourage search engines from indexing this site” option is not checked in your Reading Settings, https://localhost/robots.txt should look like this –

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

When the “Discourage search engines from indexing this site” is checked, you should see this –

User-agent: *
Disallow: /

If WordPress has been installed in a sub-directory inside of root – example: https://localhost/some_folder_name/ you’re probably going to get a 404 on https://localhost/some_folder_name/robots.txt

I don’t think WordPress will create a virtual robots.txt file if it’s in a sub-directory install.

Some references to take a look at: class-wp-rewrite.php line 1250

wp-includes/functions.php line 1301

That should shine some light on just how little I actually know about the WordPress virtual robots.txt file (LOL!) but I think that’s all good information. I can’t think of any other reason off the top of my head why you wouldn’t be seeing a robots.text file unless the WordPress files are in a sub-directory.

Hopefully someone who actually knows what else could be going on might chime in and point us in the right direction if I’m totally off the mark.

Make sure you have pretty permalinks turned on, and have an .htaccess file created for the site.

Make sure you have pretty permalinks turned on, and have an .htaccess file created for the site.

Oh, yeah… then there’s that, too (kicks toe in the dirt and hides face)
Thanks Otto! ??

James: It is installed _in_ the DocumentRoot but as a VirtualHost

Samuel: I see that Pretty Permalinks is always on, my only choice is to select what format I want. I tried a couple, made no difference.

The .htaccess is the default one coming with WordPress.

This is a pristine WordPress 4.4 install.

Many thanks to all that replied.

I see that Pretty Permalinks is always on, my only choice is to select what format I want. I tried a couple, made no difference.

The “default” setting is for pretty permalinks to be Off, actually. If your URLs look like ?p=1234, then they are not enabled.

The .htaccess is the default one coming with WordPress.

WordPress does not come with an .htaccess file, it creates one when you enable non-default permalinks, or it tells you what to put in the file itself if it cannot create the file. Furthermore, the .htaccess will only work if you are using Apache and have Apache configured to allow that sort of thing correctly.

These are things you need to check. If permalinks are not working properly, then the “default” robots.txt piece, which relies on them working, won’t work.

You need to perform checks and tests on the site in order to solve the problem. We don’t care how “pristine” the site is, it is a unique site, with a unique configuration. We need to know that configuration.

You say that you tried some other permalinks settings. What settings? How did you test after those settings? What was the result of your tests? What is the actual content of your .htaccess file? What permalink settings do you currently have? Do the pretty URLs work for your site in other places, like Single Post pages or Archives?

These are the kinds of things we need to know to diagnose the problem.

Samuel and all — thanks again for your help.

Permalinks: there is no such thing as ‘off’ but there is ‘plain’ so I guess this is what you refer to.

If it doesn’t work under a couple of permalinks settings, it is pointless to try them all, I’m not willing to dance naked under full moon and sacrifice a rubber chicken just to to make it work ?? ?? ??

I was expecting it to work under most circumstances, especially without any potentially interfering plugins and changed settings. Clearly that’s not the case and I am not into poking blindly in the dark.

It’s not in ‘WordPress for Dummies, 7th ed’
It’s not in ‘WordPress 4.4 Complete’ (Packt)
It’s not in ‘Professional WordPress, 3rd ed’ (Wrox)
The first thing that comes up in a Codex search is a 6 years old post
I’ve searched on the web for almost a full day, nothing. Everybody is using a real robots.txt.

So clearly something is going on: it’s not being used and/or it’s not working anymore, nobody’s paying attention to it … or something equivalent.

I’m giving up.

But again thank you all for your help, time, and effort, much appreciated ?? (even if it did not lead to a solution)

All the best.
Regards.

Oh, boy.. don’t give up yet! You’ve come so close!

I’m not so sure you’re actually suffering from a WordPress condition at this point. I think “pretty permalinks” probably aren’t really working.

Permalinks: there is no such thing as ‘off’ but there is ‘plain’ so I guess this is what you refer to.

If it doesn’t work under a couple of permalinks settings, it is pointless to try them all,

While you will always see the permalink choices in your dashboard settings, they only work if you’ve enabled mod_rewrite on your server, and then added the proper directive to your virtual host configuration.

So whether or not “pretty permalinks” work is completely dependent on your apache and virtual host configuration. What Linux distribution did you choose for installing WordPress? Was it Ubuntu by any chance?

It might actually be a pretty simple fix. Feel like giving it one more shot? Maybe we can confirm whether or not it’s actually an apache configuration oversight that’s causing the problem.

Bummer! OK fixed. James, thanks for the mod_rewrite reminder

mod_rewrite: was enabled in the main config but not in the VirtualHost
*facepalm* ??

<VirtualHost ...>
    ...
    <Directory>
        ...
        AllowOverride All       <- needed for .htaccess
                                   not sure 'All' is best (security ?)
        RewriteEngine On        <- this was it
        ...
    </Directory>
    ...
</VirtualHost>

Thanks ??

Glad you got it worked out!

I was getting this error and really it took toll of my nerves

Finally I found a solution and I suggest either of these works

1. If you use plugin called “Google XML Sitemaps” then uncheck “Add sitemap URL to the virtual robots.txt file.” under Basic Options. I have a line for the XML Sitemap in the Robots.txt file so I am not using this option.

2. Try installing some virtual robots.txt editor plugins. I have used WP Robots Txt and then edited the file. It worked!

Please note: I still have the robots.txt on my root. I just copied the text on WP Robots Txt plugin and it worked! Gosshh

[I will post the same answer to some of the similar questions.. so those who are in same situation may get instant relief]