Firewall: Block Fake Googlebots is blocking Google

  • Hi everyone,
    I have spent a couple of days troubleshooting Google Search Console: Google couldn’t see any content in my sitemaps even though I could load them no problem. Eventually, with the help of the GSC forums, it transpired that requests with Googlebot user agent were returning empty answers. The response code was still 200 OK, but no content to the answer.
    I narrowed down “All In One WP Security & Firewall” as the culprit because Google could load my sitemap ok if I turned the plugin off. I then went through the process of turning various settings on and off until I hit on the right one. It seems logical in retrospect, and I wish I’d thought of it at the start!

    Anyway, tl;dr is ‘Block Fake Googlebots’ can also block bona-fide Google.

    Cheers,

    M.

    The page I need help with: [log in to see the link]

Viewing 13 replies - 1 through 13 (of 13 total)
  • Plugin Contributor mbrsolution

    (@mbrsolution)

    Thank you for reporting this. The plugin developers will further investigate your findings.

    Kind regards

    Plugin Contributor wpsolutions

    (@wpsolutions)

    Hi @appleisleprospector,
    Do you have the latest version of the aiowps plugin installed?
    There was a change recently to the function which retrieves a visitor’s IP address.
    If you haven’t tried it already can you please try testing the latest version and let me know if you still see the same behaviour?

    Hi, I’m using the latest version of the AIOWPS

    Is the feature: Block Fake Googlebots blocking google AMP?
    Beacause Google cant cache my AMP pages if I checked the feature
    I tried to test my AMP pages on https://search.google.com/test/amp
    and it took a very long time for testing my AMP pages, and sometime will fail

    Currently I have disabled the feature, my AMP pages are working fine.

    Could you check the feature? because it very useful to avoid the fake bots.

    Many Thanks ??

    • This reply was modified 7 years, 5 months ago by newwper3.

    Hello,

    I also have this issue, and i am using the latest version of Allin One Security. I am not sure how significant this is but it started to happen when i started to use Cloudflare, primarily to provide SSL certificates in my conversion from HTTP to HTTPS.
    Unless this feature is disabled Fetch with Google does not work either ie it returns with zero content on any page including sitemap, and it is my impression that zero content on Fetc with Google is going to impact Rankings ? Yes?

    Thanks

    John W

    Plugin Contributor mbrsolution

    (@mbrsolution)

    @ejwjohn, is your issue related to the following support thread?

    Hello,

    Yes it would seem the same and i have asked Cloudflare to comment ???

    Thanks

    John W

    Same here,

    the setting for blocking fake Google bots is actually blocking the real Google bot. All sites of my sitemap have been removed from the index because of the issue. It worked perfectly until October 2017 but then stopped.

    Do you have any further ideas how to fix this?

    Cheers,
    Mario

    Hello,

    Have you started to use Cloudflare?

    John W

    No, the setup didn’t change when the problem occurred.

    Hi,

    My issue was that i started to use Cloudflare and i noticed that after that Google Bots failed, as the requestor IP was that provided by Cloudflare, Not a Google IP address, and the Fake Bot feature was being set.

    If it is wrong IP related have you tried doing a Fetch as Google via your search console, try this ( with the block Fake Bot feature enabled) on any page within your site and if you only get the HTTP header and nothing else, then somehow it looks as if your server is getting a none Google IP,

    John W

    You are right.

    If spam protection is ON and I fetch the page as Google, only the HTTP header is delivered.

    Now the question is, why is the server getting a wrong IP?

    Cheers,
    Mario

    I would say the next question should be to your hosting company??

    Sorry cannot be of more use.

    John W

    Your hints helped a lot already, John ??

    The hosting provider is Amazon AWS (an EC2 server + Route 53 domain config). I don’t even know if it is possible to mess up with the Googlebot there if you don’t specifically want to do that ??

    Cheers,
    Mario

Viewing 13 replies - 1 through 13 (of 13 total)
  • The topic ‘Firewall: Block Fake Googlebots is blocking Google’ is closed to new replies.