Sucuri = Fake Google Crawler?

Is your site using CloudFlare or any other reverse proxy? Some hosts may use Varnish or nginx. This can cause Wordfence to see the IP of the proxy or your own server, so multiple visitors (or services) may be seen as having a single IP.

The best way to check is on the Live Traffic page, to see if your own IP address shows up correctly — you may need to view the site in a second browser without logging in, to check.

If IPs are not correct, you will need to change the option “How does Wordfence get IPs” on the Options page. More details on the choices are available here:
How does Wordfence get IPs

Let me know if this helps, or if you are still having trouble.

-Matt R

@wfmattr

I can see my own IP without problems. It really looks like the Sucuri checker is seen as a Fake Google Crawler.

Sucuri reads a number of pages from the site. It might be that only one of those calls is seen as an FGC. I have noted that they use several different user agents.

Ok, if they do actually use a Google user agent, that could do it. In that case, you could disable “Immediately block fake Google crawlers”, which would let them through in that case, but also lets other fake Google crawlers through. (This option is off by default, and it isn’t needed on most sites — but it is good for blocking some bad bots and some tools that try to steal content in bulk.)

Alternately, you can add the sitecheck IP to “Whitelisted IP addresses that bypass all rules” on the Wordfence options page. That would let the scanner through to check everything without being blocked, but it might also cause some false positives, if it checks for issues that Wordfence normally protects.

You can find the IP on the Blocked IPs page under the Wordfence menu, shortly after they are blocked. I would double check that the hostname below is theirs as well, just in case there are any unusual issues on the server.

-Matt R

@wfmattr

Thanks for your reply. Really appreciated.

I would like to have the option “Immediately block fake Google crawlers” on. It blocks quite a lot of unwanted traffic.

Your alternative to whitelist certain IP’s sounds OK, but I have seen that they use at least 2 IPs, but maybe more. That sounds like a lot of work to keep track of. And besides, I don’t like to whitelist certain IP’s, for the reasons you indicate.

And your alternative might work for me, but not for anybody else with the same settings.

So, I’m stuck here.

It would be nice if WordFence could see that the Sucuri traffic is not from a Fake Google Crawler (1), but if that is impossible I’ll have to ask the folks at Sucuri to do their check another way (2). In the last case they need to know how you identify a Fake Google Crawler.

My preference is solution number 1. Is that a realistic approach? Or should I contact Sucuri. What would you advise?

It may be possible — if you can make sure they are currently unblocked and run their scan, then send me the end of your site’s access log that shows where they were blocked, our dev team can look at it, to see what the best way is to handle it.

You can send me the access log by email: mattr (at) wordfence.com

-Matt R

@matt

Thanks very much!

I’m rather busy right now, but I will look into it the coming days and send you the details.

Great, thanks! I’ll watch for other reports of this issue as well.

-Matt R

Mailed on nov 26, 2015. Waiting for reply/solution. If it comes I hope to remember to post it here…

Thanks for sending the details by email (still catching up from the holiday here.) We do actually have a built-in whitelist for Sucuri’s sitecheck, but it looks like they have a new IP for one of the servers that is not included.

This will be updated in a future version, but I think the best option for now is to add the new IP shown in your block message to the Whitelisted IP addresses field, under the “Other options” heading on the Wordfence options page.

(Edit: I didn’t mention that I also sent this on to our dev team, and the reference number for this request is FB1174.)

-Matt R

@matt,

Thanks for the update. I’ll wait and see what happens in the next update. There is no rush with this.

Thanks again!