Robots.txt is blocking googlebots

  • hi!
    I have problems because google webmaster was notified me that googlebots can’t access to my site because of the robots.txt file.

    I have check my robots.txt and it’s actually “virtual” because i can find it.

    I don’t know how do the robots is blocking the googlebots.

    Can someone help me!

    The site I have problems is https://www.tamcostarica.com/

Viewing 5 replies - 1 through 5 (of 5 total)

  • I doesn’t look like there is anything in your robots.txt file that would block google:

    User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

User-agent: Googlebot
Allow:
Sitemap: https://tamcostarica.com/sitemap.xml

    You can test it in Google Webmaster Tools >> Crawl >> robots.txt.Tester

    Is that a historical warning from a day in the past, or this a current issue now?

    EDITED:
    Try running this past your robits.txt Tester with the trailing slash after Allow: /

    User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

User-agent: Googlebot
Allow: /
Sitemap: https://tamcostarica.com/sitemap.xml
    Thread Starter jmasis09

    (@jmasis09)

    Is a current issue…

    google webmaster notified this:
    Over the last 24 hours, Googlebot encountered 29 errors while attempting to access your robots.txt. To ensure that we didn’t crawl any pages listed in that file, we postponed our crawl. Your site’s overall robots.txt error rate is 100.0%.

    If I Crawl as google the site is says “It’s out of access”

    When I run your robots.txt through the Google Webmasters tester it comes up fine without any problems.

    Have you tried:

    User-agent: Googlebot
Allow: /

    or

    User-agent: Googlebot
Allow: *

    Are you running any security plugins? Try disabling all plugins and then crawl as Google. If you can, then reactivate one by one until you can identify the conflicting plugin.

    Thread Starter jmasis09

    (@jmasis09)

    I try with 

    User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

User-agent: Googlebot
Allow: /
Sitemap: https://tamcostarica.com/sitemap.xml

    And now is working. Also I use a anti-malware scan, and fix a file that it have a Backdoor script so made, that was the problem generator.

    Glad to hear that Google can see you again!

Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘Robots.txt is blocking googlebots’ is closed to new replies.

Tags