Googlebot Getting Blocked from CSS & JS

  • Resolved Burnspot

    (@burnspot)


    9 years, 4 months ago

    So I’ve started getting a lot of warning emails this week from Google about various client sites giving Googlebot trouble when it tries to access CSS and JS files. From a bit of testing and topic research, it’s clearly Wordfence causing the issue despite WF options being set to provide cart blanche access to Googlebot.

    One resolved topic in here about the problem was to unblock the US…in the Pro version of WF. Is that the only option available to fix this problem…buying the Pro version?

    I like WF and the options it provides, but I can’t have it giving Google trouble…

    https://www.ads-software.com/plugins/wordfence/

Viewing 15 replies - 16 through 30 (of 44 total)

  • Update. Big discussion on the google webmaster forums that we are reading now. This is definitely a google change that is causing it, but we are investigating a workaround.

    tim

    Blocking /wp-admin/ is very common with WP installs, it’s a default for one of the most popular SEO plugins for WP.

    Expect these messages to be the tip of your iceberg in the next few days.

    Thread Starter Burnspot

    (@burnspot)

    Yea, I was suspecting a Google-side issue the more I read up on stuff. I always thought wp-admin was blocked by default by WordPress or Yoast…and Google’s not complained about that before this week.

    I’d upload a screen cap if I could, but here’s the text from Search Console. When I run “Fetch as Google” this is what comes up:

    Googlebot couldn’t get all resources for this page. Here’s a list:
    URL Type Reason
    https:// (url removed) /wp-admin/admin-ajax.php?action=wordfence_logHuman&hid=FB2809D2EB6B4D07598EA1F8A50488C2&r=0.3520287969149649
    Script Blocked robots.txt Tester

    @carolkn We do not add anything to robots.txt. Its a default file that wordpress adds if it doesn’t find one on your site. Google changed something today and we’re looking at a workaround now.

    tim

    @wfsupport : thanks for a great plugin guys, and also for diving into this issue. I know you’ll solve it as it is critical to the efficacy of your plugin, even if it’s not your fault.

    I get a LOT of benefit for free from you guys – so I just wanted to throw out a thank you and a “I believe in you”.

    Carry on!

    bippity bump

    following. I hope there will be some solutions soon, because I altered all my robots.txt files, but I’m still blocking wp-admin for security reasons. The only problem I now have is with the Wordfence Live Traffic – same script as above, so there must be another solutions or workaround for this.

    Please update to version 6.0.14
    Thanks!

    tim

    Yep, that was it! Thanks for the quick solution! You guys do a great job there.

    If someone is interested, how the “right” WordPress robots.txt now should look like, here it is (I use WordPress Robot Txt plugin to display and edit in WP Settings menu and Yoast SEO plugin – sitemap)

    User-agent: *
Disallow: /wp-admin/
allow: /wp-content/plugins/
allow: /wp-content/themes/
Sitemap: https://domain.tld/sitemap_index.xml

    I’m still seeing WordFence show up in Webmaster Tools. I want to make sure this is just lingering or if this is still an issue:

    Instead of seeing
    https://www.somedomain.com/wp-admin/admin-ajax.php?action=wordfence_logHuman&hid=123456... etc

    As “Type”: Script and “Reason”: Blocked

    I am now seeing this
    https://www.somedomain.com/?wordfence_logHuman&hid=123456... etc

    As “Type”: Script and “Reason”: Not Found

    Thanks for making an update available so quickly.

    When I “Fetch and Render” in Google Webmaster Tools, I still receive a result of “Partial” render instead of “Complete”.

    The message Google gives is:

    Googlebot couldn’t get all resources for this page. Here’s a list:

    URL: https://www.(domain)/?wordfence_logHuman=1&hid=BC792EBED726F48B127EB62EE3B894E5&r=0.03589733876287937
    Type: Script
    Reason: Not found

    So it is no longer trying to access this in the wp-admin directory, which is good. And the message is no longer “blocked” which is also good.

    But is it a problem that Google thinks it still can’t access a file because it is “Not found”?

    I solved same for my blog by editing Robots.txt file.

    Set my robots.txt as
    Method 1:

    User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
allow: /wp-content/plugins/
allow: /wp-content/themes/

    Method 2:

    #Googlebot
User-agent: Googlebot
Allow: *.css
Allow: *.js

# Other bot spider
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

    Using Method 2 issue will be fixed only for GoogleBot. If in future Bing / Yahoo or any other bot also throws same warning then you apply method 1

    Detailed Article: How to fix “Googlebot cannot access CSS and JS files”

    With method 2, won’t every hacker using a fake googlebot user agent have access to your admin files?

    Remember, we’re talking about the robots.txt file, which is purely a suggestion to search engines. It’s up to the search engine to follow the instructions or not.

    We’re not talking about editing the .htaccess file, which would remove access completely to some files or directories. If the access was completely blocked, then you wouldn’t be able to login to your site, nor would WP even function.

Viewing 15 replies - 16 through 30 (of 44 total)
  • The topic ‘Googlebot Getting Blocked from CSS & JS’ is closed to new replies.