Where put robots tag and robots.txt?

  • I read some discussions about search robot exklusion, but still I need to ask for help. I want to make search engines not index my website. This I know: There are two ways – using the robots tag in the head section of the web pages, and putting a robots.txt document with Disallow commands at the root of the web site. So far I’m informed.

    Now, I am not sure of the details:

    1) Where do I find the actual document of a blog post in WordPress? I can just see the post in the database of my site, but not real web page documents in HTML. So, where do I find the Head sections to put meta data as the robots tag in?

    2) Where should I put the robots.txt document, and how should I refer to catalogs in it? I put a robots.txt in the httpdocs catalog, and only wrote disallow for the next level of catalogs. Do I need to put the txt file on another place, and should I mention catalogs on sub levels too?

Viewing 4 replies - 1 through 4 (of 4 total)

  • 1) the actual document of a blog post is created as the visitor comes to the site. The click on the url causes wordpress to take the post content from the database, put it into the theme template and apply the stylesheet.

    ….. so the robots instructions are not in the post.

    If you look in the theme files you will see the php code to create the meta tags in the page. They are probably in a file called header.php

    2) I assume you mean directories when you say catalogue. The robots.txt file should be in the root directory of your blog – that is to say the directory called when the url https://www.yourdomain.com is clicked

    Thread Starter pelia

    (@pelia)

    1) OK, I tried adding the noindex stuff in the name tag. From start it looked like this:

    <meta name=”generator” content=”WordPress <?php bloginfo(‘version’); ?>”

    It ended up looking like this:

    <meta name=”robots” content=”noindex,nofollow”>
    <meta name=”generator” content=”WordPress <?php bloginfo(‘version’); ?>” />

    Do you think this is correct? Should I just add a new line like this?

    2) Yes, I meen directoris. OK, that would be in the httpdocs, because then I can see the txt file if i enter my domainname.domain/robots.txt in my browser. But wich directories should I type? Now I have this:
    User-agent: *
    Disallow: /wp-content/
    Disallow: /index.php
    Disallow: /wp-content/themes/
    Disallow: /wp-content/themes/index.php
    Disallow: /wp-content/themes/durable/
    Disallow: /wp-content/themes/durable/index.php
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    So, I just wrote the first level of directories and also, I tried writing som single files in a manner I made up, with the slashes. Are those unnecessary or perhaps incorrect? Should I also add subdirectories?

    Many thanks for your help!

    Moderator Samuel Wood (Otto)

    (@otto42)

    www.ads-software.com Admin

    Yes, adding new lines like that is fine, but your new meta tags should end with /> to be proper XHTML. Look at the existing lines and do like they do.

    And if you don’t want any of your site indexed, all you need is this:

    User-agent: *
    Disallow: /

    When you disallow one thing, everything under it is disallowed as well.

    Thread Starter pelia

    (@pelia)

    Many thanks for your helpful advice!

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘Where put robots tag and robots.txt?’ is closed to new replies.