robots.txt

  • i’m trying to do some seo on my wordpress blogs

    i’m thinking of adding a robots.txt file

    – i use xsitepro to create my websites – this automatically creates a robots.txt file – can i upload the same one to the blog?

    – is there a plugin that would create the robots.txt file for me?

    – i’ve been doing some searching and found a download wordpress robots.txt file – see below

    ————–

    User-agent: *
    # disallow all files in these directories
    Disallow: /cgi-bin/
    Disallow: /z/
    Disallow: /stats/
    Disallow: /dh_
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /contact/
    Disallow: /tag/
    Disallow: /wp-content/b
    Disallow: /wp-content/p
    Disallow: /wp-content/themes/askapache/4
    Disallow: /wp-content/themes/askapache/c
    Disallow: /wp-content/themes/askapache/d
    Disallow: /wp-content/themes/askapache/f
    Disallow: /wp-content/themes/askapache/h
    Disallow: /wp-content/themes/askapache/in
    Disallow: /wp-content/themes/askapache/p
    Disallow: /wp-content/themes/askapache/s
    Disallow: /trackback/
    Disallow: /2006/
    Disallow: /2007/
    Disallow: /*?*
    Disallow: */trackback/

    User-agent: Googlebot
    # disallow all files ending with these extensions
    Disallow: /*.php$
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /*.gz$
    Disallow: /*.cgi$
    Disallow: /*.wmv$
    Disallow: /*.png$
    Disallow: /*.gif$
    Disallow: /*.jpg$
    Disallow: /*.cgi$
    Disallow: /*.xhtml$
    Disallow: /*.php*
    Disallow: */trackback*
    Disallow: /*?*
    Disallow: /z/
    Disallow: /wp-*
    Allow: /wp-content/uploads/

    # allow google image bot to search all images
    User-agent: Googlebot-Image
    Allow: /*

    # allow adsense bot on entire site
    User-agent: Mediapartners-Google*
    Disallow: /*?*
    Allow: /about/
    Allow: /contact/
    Allow: /wp-content/
    Allow: /tag/
    Allow: /manual/*
    Allow: /docs/*
    Allow: /*.php$
    Allow: /*.js$
    Allow: /*.inc$
    Allow: /*.css$
    Allow: /*.gz$
    Allow: /*.cgi$
    Allow: /*.wmv$
    Allow: /*.cgi$
    Allow: /*.xhtml$
    Allow: /*.php*
    Allow: /*.gif$
    Allow: /*.jpg$
    Allow: /*.png$

    # disallow archiving site
    User-agent: ia_archiver
    Disallow: /

    # disable duggmirror
    User-agent: duggmirror
    Disallow: /

    Sitemap: https://www.askapache.com/sitemap.xml

    ——————–

    is this an ‘across the board’ robots.txt file, or do i need to customise/personalise it in any way?

    do robots.txt files help in seo?????????????

    would appreciate your advice!

    kathryn

Viewing 1 replies (of 1 total)

  • That is more of a learning example to show what is possible and how it works. Check out the Updated robots.txt ffor WordPress.

    User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/askapache/
Disallow: /trackback/
Disallow: /2006/
Disallow: /2007/
Disallow: */trackback/
Disallow: */feed/

User-agent: Googlebot
Allow: /wp-content/uploads/

User-agent: Googlebot-Image
Disallow:
Allow: /*

User-agent: Mediapartners-Google*
Disallow:
Allow: /*

User-agent: ia_archiver
Disallow: /

User-agent: duggmirror
Disallow: /

# Does anyone care I love Google Apache htaccess

Sitemap: https://www.askapache.com/sitemap.xml
Viewing 1 replies (of 1 total)
  • The topic ‘robots.txt’ is closed to new replies.