Robots Error

  • When I look at my website with robots.txt in a browser I see the following

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    However, in the root directory of my wordpress installation the robots text contains

    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-content/
    Disallow: /wp-includes/
    Disallow: /go/
    Disallow: /comments/feed/
    Disallow: /feed/
    Disallow: /trackback/
    Disallow: /index.php
    Disallow: /xmlrpc.php
    Disallow: ?comments=*
    Disallow: /search?
    Disallow: /?p=*

    User-agent: Mediapartners-Google*
    Allow: /

    User-agent: Googlebot-Image
    Allow: /wp-content/uploads/

    User-agent: Adsbot-Google
    Allow: /

    User-agent: Googlebot-Mobile
    Allow: /

    Does anybody have ideas about why my robots file is not properly working? The only SEO plugin I’m using is WordPress SEO. Thanks

Viewing 4 replies - 1 through 4 (of 4 total)
  • Brandon Kraft

    (@kraftbj)

    Code Wrangler

    To clarify, both the robots.txt files are at the root of your domain? If your WP install isn’t in root (e.g. https://example.com with WP installed in https://example.com/wp/), having the robots.txt in /wp/ won’t help.

    The WordPress SEO allows you to edit a robots.txt directly within the plugin, in the Files menu. It may make things simpler to copy/paste your desired text into the plugin, then delete the physical file.

    What’s your website?

    Thread Starter johnny79

    (@johnny79)

    I figured out what the problem was. Thank you for the help.

    Brandon Kraft

    (@kraftbj)

    Code Wrangler

    Do share the solution. It could help someone else searching for a fix later on.

    Glad it is fixed!

    Thread Starter johnny79

    (@johnny79)

    Apparently it had something to do with my hosting company not updating my robots file as fast I would have liked.

    All I did was delete the old hosts file that was in the root directory of the public-_html folder and uploaded a new file within the root directory of the specific domain I was working on.

    Thanks for reaching out to help. It’s good to see another Texan!

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘Robots Error’ is closed to new replies.