Robots.txt questions

Hi Dave,

That’s a very good question, which I’ve asked myself too.

I came to the conclusion that the robots.txt file is used for general purpose (directory) blocking, and not page specific.

From what I’ve found, blocking a page within the robots.txt file will also prevent Search Engines from crawling the page in the first place, which will prevent Search Engines from finding external and internal linking, resulting in missing pages in the Search Index. Which is a bad thing.

More info is found on Moz, with a few examples.

I hope this clears things up! Let me know if you have any more questions and I’ll be happy to answer them :).

Sybre,

Thanks for the link to the article. It helped clear this up some.

I think where I was getting a little confused was that there are 2 robots. One is the robots.txt and the other is Meta Robots file. It does look like you can use both but need to be careful of conflicts.

Even though the NOINDEX page showed up in webmaster tools as Allowed for some reason, when I do a google search for info:https://mysite.com/page it does not show up. Also, only the robots.txt file showed up in webmaster tools. So I deleted that one and elected to go with the Meta Robots method for now.

So as far as I can tell, all is working well. ??

Thanks again,
Dave

Hi Dave,

No need to be careful :). Just know that the robots.txt file takes precedence over the meta tag.

The meta tag is just much more convenient (and therefore recommended by many), as it’s rendered when the page is, and not in bulk for another file (which is difficult to program, to keep updated and also heavy on the server).

Plus it also has the added benefit for allowing following of link, as stated earlier. The meta tag is more advanced, in this context.

I understand there can be some confusion. I’ll see if I can clear that up within the plugin.

If you have any more questions left unanswered, feel free to ask.

Thanks and have a great day! ??