IP Geo Block Block Googlebot?

  • Sn00z389

    (@webmakers2011)


    6 years, 7 months ago

    Hi, today I saw that the plugin prevent Googlebot from crawling some url parameters

    Request
    GET[443]:/wp-content/plugins/facebook-button-plugin/css/style.css?ver=4.5.2
    User agent
    Mozilla/5.0 (compatible; Googlebot/2.1; +https://www.google.com/bot.html)
    HTTP headers
    HTTP_REFERER=https://sitename.com/tag/podaraci-za-8-mart/page/4/?orderby=price-desc,HTTP_FROM=googlebot(at)googlebot.com
    $_POST data

    Is this dangerous for the SEO, since I saw in the Google Search Console that the URL-s crawl fine without the ?orderby ?

Viewing 6 replies - 1 through 6 (of 6 total)
  • Plugin Author tokkonopapa

    (@tokkonopapa)

    Hi @webmakers2011,

    User agent string can be spoofed. So please check its IP address on “Search” tab.

    Regarding to the googlebot, this plugin follows to Verifying Googlebot.

    You can adjust the rules at UA string and Qualification in “Front-end target settings” section.

    But I think usually googlebot will not access to CSS files. There is a possibility of searching the vulnerability of the plugin.

    c.f.

    Thread Starter Sn00z389

    (@webmakers2011)

    Hi, yes it is strange, since it is not in the frontend, but in the backend, Googlebot is trying to crawl plugins that are long not on our website, it is also trying to get to the wp-login.php and the plugin blocks it there too.

    But I can see that it passes googlebot to the frontpages, so everything should be OK.

    Thanks for your reply and the great plugin! It does great job for protecting against Bruteforce and xmlrpc attacks! ??

    Thread Starter Sn00z389

    (@webmakers2011)

    The IP address is a real Googlebot agent – 66.249.76.111

    Plugin Author tokkonopapa

    (@tokkonopapa)

    The IP address is a real Googlebot agent – 66.249.76.111

    OK.

    Hi, yes it is strange, since it is not in the frontend, but in the backend

    Agreed.

    I think your server software is Nginx. If someone from blacklisted country accesses to a non-existent file, then this plugin might block with “403 Forbidden” error (by default) when you enable “Prevent Zero-day Exploit“.

    (In case of Apache, one gets just “404 Not found” error)

    On the other hand, if the bot accesses the existent css file, this plugin never block it and it might get just “404 Not found” error.

    It means that Googlebot had accessed a non-existent file in your site. So I think the point is the difference between “404 Forbidden” and “404 Not found” from the view point of SEO.

    I’m not sure about it at this point. So if you worry about “403”, I’ll provide you some code snippet to serve “404” when the accessor is Googlebot.

    Thanks.

    Thread Starter Sn00z389

    (@webmakers2011)

    Yes, our software is LiteSpeed actually I don’t see any notifications in GWT so it should all be OK. If I see some I will continue the thread so that we can think of a workaround!

    Thank you for your time and support!
    Best Regards!

    Plugin Author tokkonopapa

    (@tokkonopapa)

    Hi @webmakers2011,

    OK, I keep this thread open.

    And thank you for your wonderful review!

Viewing 6 replies - 1 through 6 (of 6 total)
  • The topic ‘IP Geo Block Block Googlebot?’ is closed to new replies.

Tags