Dealing with Robots.txt

  • How does this plugin deal with Robots.txt file?
    Would it be effective to add the following lines to the file, or does the Sitemap Generator already deal with this in some way?

    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /trackback
    Disallow: /feed
    Disallow: /comments
    Disallow: /category/*/*
    Disallow: */trackback
    Disallow: */feed
    Disallow: */comments
    Disallow: /*?*
    Disallow: /*?
    Allow: /wp-content/uploads
    Sitemap: https://www.askapache.com/sitemap.xml

  • The topic ‘Dealing with Robots.txt’ is closed to new replies.

Tags