my robots.txt blocks my categories.

  • Resolved do77

    (@do77)


    16 years, 1 month ago

    Since a couple of weeks I have been now trying to fix my robots.txt in a way that the relevant information will show up in the google index or being crawled by googlebot. But either the robots.txt blocks this page or another one and then I change it and suddenly my blog doesnt show up in the index at all. I really don’t know what to do anymore. Can please someone help me with that!?

    My robots.txt looks like that:

    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /trackback
    Disallow: /feed
    Disallow: /comments
    Disallow: /category/*/*
    Disallow: */trackback
    Disallow: */feed
    Disallow: */comments
    Disallow: /*?*
    Disallow: /*?
    Allow: /wp-content/uploads

    # Google Image
    User-agent: Googlebot-Image
    Disallow:
    Allow: /*

    # Google AdSense
    User-agent: Mediapartners-Google*
    Disallow:
    Allow: /*

    # Internet Archiver Wayback Machine
    User-agent: ia_archiver
    Disallow: /

    # digg mirror
    User-agent: duggmirror
    Disallow: /

    Sitemap: https://www.santabarbarabums.com/sitemap.xml

    I read somewhere that this is the default setting and works perfectly with google. But now robots.txt blocks my categories which contain the important keywords to show up in the index.

    Is that the way it is supposed to be? what should I change? hope someone has advice.

    dom

  • The topic ‘my robots.txt blocks my categories.’ is closed to new replies.