Robots.txt Live different from File on server

  • There was never an issue in the past with my Robots.txt but I was doing some reading on how to make it more secure in regards to not having some items crawled and so I decided to edit it and by mistake I deleted the file and so I had to add one back on the server. I’m using a file manager plugin to carry out thiese tasks as I’m not very effective with linux commands. My concern is that I may not have put back the Robots.txt file back in the right spot because when I click on “see live robots.txt” in Googlewebmaster tools it is different from what I have on the robots.txt file on the server.

    Isn’t the Robots.txt live supposed to be exactly what is on the robots.txt file on the server? And if so why might my live robots.txt not be showing the same contents as the server robots.txt file?

    Oh and I should add that when I view the robots.txt file from ALL IN ONE SEO plugin, it’s outputted robots.txt content matches that of the file on the server. So Based on that I would think that confirms I did put back the robots.txt file back in the right place yes?

    • This topic was modified 8 years, 1 month ago by skygazer.
    • This topic was modified 8 years, 1 month ago by skygazer.
Viewing 4 replies - 1 through 4 (of 4 total)

  • have you done these steps?
    https://support.google.com/webmasters/answer/6078399?hl=en

    Thread Starter skygazer

    (@skygazer)

    Hi thanks for the reply.

    I have figured out what the problem is, CLOUDFLARE.

    Well Cloudflare itself is not the problem, it was the caching on Cloudflare that was the issue I think. I switched the ‘development mode’ on and then checked the live robots.txt and then I finally saw the recently edited version I put on the server.

    So looks like all is good.

    By the way can someone say if this looks okay for robots.txt and still great for Google indexing my articles?

    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-content/
    Disallow: /wp-includes/
    Disallow: /recommended/
    Disallow: /comments/feed/
    Disallow: /trackback/
    Disallow: /index.php
    Disallow: /xmlrpc.php
    Disallow: /wp-content/plugins/

    Yes, looks about right.

    Thread Starter skygazer

    (@skygazer)

    Thanks gnoric for taking the time to write, appreciate it, sorry haven’t been able to come back on in a little while.

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘Robots.txt Live different from File on server’ is closed to new replies.