• Hi there – any help would really be appreciated.

    3 days ago I submitted my yoast seo plugin sitemap to google and it listed the following:
    452 URLs submitted
    1 URLs indexed
    1 errors
    945 warnings

    when i click on the webmasters info for the 945 warnings – it says
    Url blocked by robots.txt.
    Sitemap contains urls which are blocked by robots.txt.

    the robots txt file seems to be the following:
    User-agent: *
    Disallow: /wp-admin/

    I really dont know how to fix this – any help would be greatly appreciated?!

    Thanks in advance

Viewing 6 replies - 1 through 6 (of 6 total)
  • Moderator Samuel Wood (Otto)

    (@otto42)

    www.ads-software.com Admin

    There’s nothing to fix. The robots.txt is telling Google not to scan the wp-admin, and that’s a good thing. That’s probably what the Yoast plugin is doing intentionally. You don’t need Google trying to load your admin pages. It can’t anyway, so blocking it is the correct action.

    Thread Starter tonyz20

    (@tonyz20)

    Hi Samuel

    But then when you enter my domain into Google – it states:

    A description for this result is not available because of this site’s robots.txt – learn more.

    None of my web pages or images appear in Google?
    Webmasters shows: of 452 Pages Submitted – 1 Indexed?

    Moderator Samuel Wood (Otto)

    (@otto42)

    www.ads-software.com Admin

    Google works on its own terms. It will index your site when it gets around to it. Which probably won’t be immediately after you submit a sitemap.

    A description for this result is not available because of this site’s robots.txt – learn more.

    You might have a conflict going on between the WordPress virtual robots.txt and a real one, if you have one, and you might also double-check to be certain “Discourage search engines from indexing this site” is un-checked at Dashboard > Settings > Reading.

    Thread Starter tonyz20

    (@tonyz20)

    Samuel – surely Google wouldnt give so many warnings if it was going to just index everything?

    Leejosepo – if that were the case – do ou know what i should do to fix it?
    is there any way to find out if that is the case?

    Try mysite.com/robots.txt and you will see or not whatever Google sees or not.

    At Google WebMaster Tools, you can have Google crawl for your robots.txt file and give you a report within just a few moments. The deal with the virtual file is that at least the sitemap plugin I use will try to add its sitemap to the virtual robots.txt unless told to not do so, and I make one of my own. Mine is nothing fancy, but the virtual WordPress robots.txt will not try to over-ride it unless the plugin I use tries to force that…and then I get nothing.

    https://www.google.com/search?q=Google+robots.txt

Viewing 6 replies - 1 through 6 (of 6 total)
  • The topic ‘robots txt’ is closed to new replies.