• Resolved Alan

    (@alanwhiteman)


    Hi,

    The robots.txt file automatically added a page and its child pages for something I had not configured. When I excluded another page from the sitemap, I thought that it would automatically be placed on the robots file. However, that’s not happening.

    Also, I cannot find anywhere in the XML Sitemap settings where to configure the robots.txt options.

    Your help is greatly appreciated.

Viewing 2 replies - 1 through 2 (of 2 total)
  • Hi, you can add custom robots.txt rules on Settings > Reading. This is independent of any XML Sitemap setting.

    I should maybe explain some fundamental things to consider:

    1. Excluding a page or post from the sitemap will not prevent search engines finding and indexing that page or post. It will also not add any rules to the robots.txt file.
    2. Adding an exclusion rule to the robots.txt manually will prevent search engines from re-visiting a certain page or path on your site but it will not remove that page or directory from their index.

    If you really want to get and keep (!) a page or post out of the index and search results, then you best do two things:
    1. Go to your Search Console / Webmaster Tools account for each major search engine (if you do not have such an account, create one) and ask for a removal of the specific post/page
    2. Add a robots:noindex meta tag to the page itself. This can be done with most (if not all of the) SEO plugins available on the plugin repository.

    I’m considering adding this feature for this plugin as well but it’s outside its core function so I cannot guarantee that ever happening.

    Thread Starter Alan

    (@alanwhiteman)

    Yes, thank you for for the info. Your comment actually made me do some research about the effectiveness of a robots file and I came across this interesting article (sorry- it comes from one of your competitors, which I can assure you that I’m not using at all):

    https://yoast.com/wordpress-robots-txt-example/

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Robots.txt’ is closed to new replies.