Google crawler blocked when Wordfence activated

Did you accidentally blocked GoogleBot? take a look to your htaccess

Hi, I have not had a reply from Wordfence so changed to iThemes security plugin and deleted Wordfence. Access for Googlebot is fine now.

I’m noticing the same problem, when I tell Google to crawl my sites and render a preview, the results are just a bunch of messed up code – this cannot be good for SEO. I didn’t catch this initially with some of my other sites, because I had set up Google Webmaster Tools long before I installed Wordfence.

I will be removing this from all of my sites as well, if there aren’t any official responses on here sometime soon.

I have no problem with Google crawling any of my sites.

In my Options-Firewall rules, I have the following crawler settings:

Immediately block fake Google crawlers: (unchecked)
How should we treat Google’s crawlers: Verified Google crawlers have unlimited access to the site
If anyone’s requests exceed: 60 per minute then throttle it.
If a crawler’s page views exceed: 60 per minute then throttle it.

I would imagine that on larger sites then the request/page views upper limit might need to be increased to allow for genuine crawlers.

I have the same options and still wordsfence will not let google bots crawl

I am having the same problem too and have had to turn off Wordfence. This has been a great plugin for me until recently.

Same issue for me… https://tjohns.co/website-development/31-wordpress/problems-bugs/117-wordfence-security-blocks-google

Me too. It sucks because I love this plugin and now have to get rid of it

Me too. It really sucks. I even paid for the pro version.

Are any of you guys blocking the US for pro version? If so, Google will not be able to crawl your site, nor will Bing or Yahoo.

@tomhouy That sounds like an issue with multiple caching going on. Do you have ours and someone elses running? If two are using gzip browser caching at the same time, the browser sees the gzip compression and decompresses it once, resulting what you see.

@ryan Can you turn off everything but wordfence and see if you are still blocking google? I’d like to see your htaccess file as well.

@anne Same thing as Ryan but can you post in the premium forums so I can assist you there? If you prefer the free ones, then open a post just for your issue and post the link here for me so I see it.

@everyone I totally understand posting similar issues together. I appreciate being able to see that a larger problem exists outside of one person’s installation. However, the wordpress forum rules ask you post separate issues. So if you guys can post separate issues and then comment on the similar ones it would avoid the mods coming in. Thanks.

tim

Hi all,

I did as Tim said above and unblocked USA. I was then able to fetch as google and render the site.

Thanks Tim!

Aisling

Not a problem ??