WordPress and robots.txt

  • Hi, i want to exclude some urls from being indexed.
    But wordpress does not use html pages, so how can i specify to spiders into the robots.txt that i don’t want index some of my posts?

    THX

Viewing 4 replies - 1 through 4 (of 4 total)

  • an URL is an URL, regardless of if there’s a text file laying around.

    Just specify the post URLs in robots.txt like you normally would.

    Thread Starter hawk21

    (@hawk21)

    So i can just put this in my robots.txt

    User-Agent: *
    Disallow: /archives/2007/postname

    without any specification of the file type (html, php…) and it works?

    Moderator Samuel Wood (Otto)

    (@otto42)

    www.ads-software.com Admin

    Yes, you can.

    A URL is a URL. It doesn’t have a “filetype”. Those extensions you’re talking about don’t mean the same things on the web.

    Thread Starter hawk21

    (@hawk21)

    OK, thanks.

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘WordPress and robots.txt’ is closed to new replies.