Sitemap contains urls which are blocked by robots.txt.

  • Resolved lady_r0gue

    (@lady_r0gue)


    8 years, 3 months ago

    Hi, I may be being a bit dumb, but I’m not sure what I’ve done wrong, unless I just need to be patient? I can see the sitemap (lovely, thank you) but Google Webmaster Tools states that “Sitemap contains urls which are blocked by robots.txt.” Unfortunately, these are all the pages on my site, it seems!
    My robots.txt file says:
    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php

    Sitemap: https://mysite.com/xmlsitemap.xml

    Allow: /rsssitemap.xml
    Allow: /htmlsitemap.htm

    I have unchecked ‘discourage search engines’; I did this yesterday, and created the sitemap today. Do I need to wait 24 hours? Or should I create a ‘real’ robots.txt? And what should I put in it? It’s all a bit like black arts to me!

    https://www.ads-software.com/plugins/www-xml-sitemap-generator-org/

Viewing 3 replies - 1 through 3 (of 3 total)
  • Thread Starter lady_r0gue

    (@lady_r0gue)

    Or maybe this is fine, and I don’t need to worry, and I just need to wait for Webmaster Tools to catch up? #googlingfuriously

    Thread Starter lady_r0gue

    (@lady_r0gue)

    Here she is again, sorry. Assuming that everything’s fine, and I just need to hold my horses until Google has crawled the site… Can I add my sitemap anyway, or should wait until there are no warnings on the test? Thanks in advance ??

    Thread Starter lady_r0gue

    (@lady_r0gue)

    OK, so I found that I can use the Fetch as Google function in Webmaster Tools, which I’ve done. I suppose that’s that, then. Silly me. Sorry to anyone who may have been composing a reply. Done. Thanks again…

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘Sitemap contains urls which are blocked by robots.txt.’ is closed to new replies.