Googlebot Getting Blocked from CSS & JS

  • Resolved Burnspot

    (@burnspot)


    9 years, 4 months ago

    So I’ve started getting a lot of warning emails this week from Google about various client sites giving Googlebot trouble when it tries to access CSS and JS files. From a bit of testing and topic research, it’s clearly Wordfence causing the issue despite WF options being set to provide cart blanche access to Googlebot.

    One resolved topic in here about the problem was to unblock the US…in the Pro version of WF. Is that the only option available to fix this problem…buying the Pro version?

    I like WF and the options it provides, but I can’t have it giving Google trouble…

    https://www.ads-software.com/plugins/wordfence/

Viewing 15 replies - 1 through 15 (of 44 total)

  • Yep. Same issue here.

    We manage quite a few WordPress sites and as of this morning we too started getting emails from Webmaster Tools on from multiple sites saying that “Googlebot cannot access CSS and JS files”. The commonality among the sites is that Wordfence & the Falcon Engine Cache is enabled on all of them.

    Our robots.txt files are very “standard” on these particular sites. We also use AIOSEO and their sitemap.xml generator (although I seriously doubt that this is related). Like you, we too have given Google carte blanche access to Googlebot.

    I’d love to hear community feedback on possible settings / causes for this new Googlebot alert.

    I’ve had a couple of clients email me about it, and I can see from doing the “fetch as google” test on google webmaster tools that a Wordfence file is trying to be accessed from the admin.

    My robots.txt disallows /wp-admin/ which I believe is a pretty standard thing!

    Confirming the same issue as the above folks. Seems to be every site we manage that uses Wordfence, and it just started this morning since the latest update.

    @finestimaginary
    Our robots.txt also disallows /wp-admin/

    Same for all of my websites this morning. Wordfence, please advise…

    Thread Starter Burnspot

    (@burnspot)

    FinestImaginary, that’s the same issue I have on the “fetch” test on Webmaster tools. Disabling WordFence “solves” the problem immediately and returns a clean render in Webmaster Tools. I should note that none of my clients are using anything beyond normal WF caching (i.e. no Falcon).

    I have one other plugin causing a similar blocking issue, WordPress Popular Posts. That one isn’t mission-critical to the sites, but Wordfence is (as my preferred security plugin). Simple answer seems to be a release to fix where this file is being served from!

    /wp-admin/admin-ajax.php?action=wordfence_logHuman&hid=0CDB5E96DF38AFEA7304A645B2C6F9FE&r=0.03589733876287937

    Looks like something running through admin-ajax.php.

    admin-ajax.php is also the file that’s being used through the popular post’s plugin and throwing the same issue.

    Same here, we have been considering an upgrade for some of the sites we manage, but will NOT upgrade if this is not fixed.

    If you block wp-admin you prevent us from working. It will also break any other plugin that make calls to admin-ajax.php The correct way to do this is to follow this guide.
    https://www.wpbeginner.com/wp-tutorials/how-to-password-protect-your-wordpress-admin-wp-admin-directory/
    Pay attention to the last bit that discusses this.

    Considering that we have not changed our plugin code, it likely is an error on google’s side. The report I got from Thomas in the premium forums said that Google emailed and said this:

    restrictions in your robots.txt file

    Wordfence does not alter or restrict access to robots.txt. I believe you may need to reach out to google and ask if something changed there. I will (as I have since I saw this issue in our premium forums, twitter, and here) interface with our dev team to make sure that it is not anything we’re doing

    tim

    Plugin Author Wordfence Security

    (@mmaunder)

    Hi all,

    As Tim said, we haven’t done a release for a couple of weeks, so it sounds like something changed with Google.

    We’re actually minutes way from doing a release so please note that this occurred before we did today’s release.

    Can you please give us as much data in the form of screenshots showing why you think Wordfence is the cause? What are you seeing when you fetch as googlebot?

    Also, you mentioned that disabling Wordfence fixes this and someone mentioned the logHuman URL we use. You can try to disable Live Traffic, then fetch as Googlebot and see what happens. If you have disabled Googlebot’s access to /wp-admin/ (which you should not do) and you have live traffic enabled, then you might get an error that Googlebot can’t access the logHuman URL in /wp-admin/. So if disabling live traffic fixes that, then I’d suggest not blocking /wp-admin/

    Hope that helps.

    Thanks.

    Mark.

    Following

    Following

    I’m following too.

    Have tried to disable Live Traffic (who knew how many dodgy customers were trying to log into the site!). That didn’t work.

    I also have a robots file that’s only disallowing /wp-login.php nothing else.

    Here’s a screenshot:
    screenshot

    Here’s the text:
    Googlebot couldn’t get all resources for this page. Here’s a list:
    URL Type Reason
    https://www.(clientwebsite).com/wp-admin/admin-ajax.php?action=wordfence_logHuman&hid=24324A4ED2C9B53C343071CFBC18CA9E&r=0.6350122326985002
    Script Blocked robots.txt Tester

    For this site (and most of my others with WordFence), wp-admin is blocked in robots.txt. We didn’t do that intentionally, I’m wondering if it’s possible that WordFence added that to robots.txt as a security thing?

Viewing 15 replies - 1 through 15 (of 44 total)
  • The topic ‘Googlebot Getting Blocked from CSS & JS’ is closed to new replies.