[Plugin: Google XML Sitemaps] Does robots.txt need any more in it than the pointer to the sitemap?

  • delete

    (@moonbatwingnut)


    16 years, 6 months ago

    I installed Google XML Sitemaps and it added a robots.txt that looks like this:

    # BEGIN XML-SITEMAP-PLUGIN
    Sitemap: https://wendelbrume.com/sitemap.xml
    # END XML-SITEMAP-PLUGIN

    So it sends the robots to the sitemap, which I’ve set to contain only the pages I want crawled. I see a lot of discussion around the web about how important it is to add directions to the robots.txt file to give direction to the spiders. But since robots.txt directs them to the sitemap, and it has only the pages I want crawled, am I done?

    Thanks,

    TW

  • The topic ‘[Plugin: Google XML Sitemaps] Does robots.txt need any more in it than the pointer to the sitemap?’ is closed to new replies.