Robots. txt blocking urls

  • jronder

    (@villas-diani)


    11 years ago

    I have found somewhere online these robots text file and it said it will protect my site from hacking.

    User-agent: *
    Crawl-delay: 5
    Disallow: /feed/
    Disallow: /trackback/
    Disallow: /wp-admin/
    Disallow: /wp-content/
    Disallow: /wp-includes/
    Disallow: /xmlrpc.php
    Disallow: /wp-*

    Sitemap: https://villasdiani.com/sitemap_index.xml

    But now I got 109 warning in webmaster tool
    ” Url blocked by robots.txt.Sitemap contains urls which are blocked by robots.txt. Example: Value: https://villasdiani.com/wp-content/uploads/2011/05/Sunset-Villa-Night-300×200.jpg “

    I do not want to block these images ?? They are a part of a property listing. Can somebody please help with this?

Viewing 2 replies - 1 through 2 (of 2 total)
  • Rasierer-Test

    (@rasierer-test)

    Disallow: /wp-content/plugins/
    Disallow: /wp-content/cache/
    Disallow: /wp-content/themes/
    Allow: /wp-content/uploads/

    from an example on https://codex.www.ads-software.com/Search_Engine_Optimization_for_WordPress

    I am not sure if it is necessary to disallow wp-content subfolders, but if you replace your “Disallow: /wp-content/” with these 4 lines, your problem with the blocked URLs should be solved.
    The 4 lines above block only certain subfolders of “wp-content”, while your “Disallow: /wp-content/” blocks wp-content and all subfolders (uploads included).

    Thread Starter jronder

    (@villas-diani)

    Hallo Rasierer-Test and thank you so much for your kind reply. Now I understand were I made mistake:-(
    Thank you so much for your time and help!

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Robots. txt blocking urls’ is closed to new replies.