Robotstxt questions

  • I have had so much trouble with robots.txt I paid someone to do one for me. When I run it through Robots.txt checker, it has errors. Can someone ‘please’ tell me should I use it?

    Sitemap: https://concerningspiritualgifts.org/sitemap.xml.gz

    User-agent:  *
# disallow all files in these directories
Disallow: /cgi-bin/
Disallow: /go/
Disallow: /archives/
disallow: /*?*
Disallow: *page=
Disallow: *?page=
Disallow: /search?
Disallow: *?replytocom

User-agent: Mediapartners-Google*
Allow: /

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Mobile
Allow: /

Viewing 5 replies - 1 through 5 (of 5 total)

  • Read Google’s instructions on how to use robots.txt here:
    https://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
    Here is an example showing Google’s robots.txt:
    https://www.google.com/robots.txt

    Please note that the the basic purpose of robots.txt file is to block robots from certain links. If you want to allow all robots, keep only the first line. If you want to disallow, use the links from what you quoted and/or more links/folders. Avoid extra text and symbols.

    Thread Starter 2besure

    (@2besure)

    What is the difference between allow and disallow. Don’t disallow mean let something happen and allow mean don’t let something happen. It is all so confusing to me.

    Please try using Google as there are better resources out there for explaining robots.txt files. This really isn’t a WordPress issue.

    This from Google explains it (read the first link above):
    To prevent pages on your site from being crawled, while still displaying

    AdSense ads on those pages, disallow all bots other than Mediapartners-Google. This keeps the pages from appearing in search results, but allows the Mediapartners-Google robot to analyze the pages to determine the ads to show. The Mediapartners-Google robot doesn’t share pages with the other Google user-agents. For example:

    User-agent: *
Disallow: /

User-agent: Mediapartners-Google
Allow: /

    Edit: As esmi explained, you may use Google resources for a better explanation as this is not a WordPress issue.

    Thread Starter 2besure

    (@2besure)

    Thank you all for your help!

Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘Robotstxt questions’ is closed to new replies.