Google XML Sitemaps with my own robots.txt

Viewing 3 replies - 1 through 3 (of 3 total)

  • Yes – I don’t like the robots.txt either. I would like it to exclude tags, categories and pages, but I don’t see a way to modify it through the plugin. I placed a robots.txt file in the blog directory and Google Webmaster Tools is picking that up rather than the virtual file. So far, so good. Now to remove the urls through Webmaster Tools.

    It’s my understanding that WP generates a virtual robots text file if you have prevented indexing during site construction.

    When you finally tie the site into Webmaster Tools, this virtual robots file prevents indexing until you rework the robots.txt file.

    Did so on our site, and indexing on 40 pages began immediately but did not finish for about two weeks.

    [Text moderated]

    Thread Starter xa2

    (@xa2)

    Same happened to me, thanks!

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘Google XML Sitemaps with my own robots.txt’ is closed to new replies.