robots.txt

  • Question: Why disallow /feed/ in the robots.txt file? How does the spider find all the posts if it can’t see the feed? Is it finding it from the sitemap.xml file and is that sufficient?

    Also, what if you have over 700 posts in your sitemap, will the googlebot find all the posts? I’m of the thinking that it won’t go past the first 100 links or is that me in my old school thinking?

    Thanks in advance…

Viewing 2 replies - 1 through 2 (of 2 total)

  • Spiders in general can find content on your site by using the sitemap files, by following links on your site and by following links to your content on other people’s sites.

    Google could certainly index 700 posts on your site but it probably doesn’t. The number of pages/posts it indexes will depend, in part, on the rank, popularity, importance, age, update frequency, internal link structure and so on.

    In my experience using wordpress with the google sitemaps.xml plugin has yielded great indexing routines from google.

    thedevnull

    (@thedevnull)

    Why is there no robots.txt in wordpress?

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘robots.txt’ is closed to new replies.

Tags