Googlebot Getting Blocked from CSS & JS

For anyone else that thought the .info link was anything more than a spamming attempt, please just ignore that submission.

The robots.txt is by default “allow”.

Method 1 says “All agents can access everything except /wp-admin/ and /wp-includes/ and further states that all agents can access /wp-content/plugins/ and /wp-content/themes/” which were never disallowed… !?

If you wanted something actually relevant you might try:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

This is explained here: https://support.google.com/webmasters/answer/6062596?hl=en (Notice it’s not a .info domain?)

I’m using the latest version of Wordfence on my site, and I’m getting the error while I have the plugin activated, but Google is fetching fine when I have it deactivated.

https://www.shrimpsaladcircus.com

The crawler complaint is a minor concern that Google’s just getting around to emailing us about. Once you have the latest version of the Wordfence plugin you should stop getting the error when you re-test in webmaster tools.

A much larger concern would be your pagespeed score: https://developers.google.com/speed/pagespeed/insights/?url=http%3A%2F%2Fwww.shrimpsaladcircus.com%2F&tab=desktop

If you install the ‘EWWW’ image compression plugin, tell it in the options to remove the meta-data from images, and tell it to scan and optimize your whole site (in the Media menu), your score will jump up massively and the site will be much more peppy.

Want more? In Yoast there’s a ‘Tools’ section, and in the advanced tools there’s a ‘Files Editor’. In that section you can add things to your .htaccess file. This is where you can add some rules that will turn on browser caching and compression. I’d normally just fire off a link to a David Walsh post on stealing the best parts of the .htaccess file from HTML5 Boilerplate, but I can’t find the right one so here’s a pastebin of the caching rules (WP doesn’t like long code dumps):
https://pastebin.com/6yQX33E4

Pasting the rules into your .htaccess and saving will instantly improve your PageSpeed scores and stop people from re-downloading everything on each visit which will make the site run better/use less bandwidth.

There’s also rules for server side compression that are usually safe to add to your .htaccess but have a higher chance of not working with your particular server so I don’t share them as readily.

I checked the two sites that I got the notification about yesteday and got a blocked message. I tested them both today and it retrieved them. I haven’t made any changes as I was waiting to let things sift out in case there was an error somewhere. Is anyone else seeing this resolve?

Following
Thank you very much Wordfence – BTW really like your new logo

Life Afloat – Explore Earth with Jack and Jude

OK, I researched it. This is what I found and this is what I did to solve the blocking problem. It’s easier than I thought.

What I found:
1) More and more people who know a lot more than I do about this subject, say that for most websites, a robot txt file isn’t even needed anymore. Why? Because the only bots that care about what it says are the good guys like Google that it’s blocking. The bad bots don’t say “I want to hack a site but look, they don’t want me to go to this file, shucks”. The robot txt file is just a suggestion, some bots don’t even bother to look at it.

2) Yoast (I don’t use that plugin anymore but I have a lot of respect for them) wrote that he doesn’t block WP Admin or WP includes at all anymore.
See: https://yoast.com/wordpress-robots-txt-example/
I went to yoast.com and looked at their /robots.txt and sure enough, they only block something to do with affiliates. In his post, he said that when you delete the robot txt file, WP will automatically at something. WP isn’t stupid.

So I tested it. I deleted my robot txt file that my seo plugin added and checked my /robots.txt and sure enough, a new file was there:

User-agent: *
Disallow: /wp-admin/

Sitemap: https://www.(removed for privacy).com/sitemap.xml.gz

I went to google webmasters and did a fetch as google AND render (you have to choose that option). Result? No more blocked css or java.

I must say I had the confidence to do this because I have Wordfence (free) an excellent plugin that was not the cause of this issue for me, and Bulletproof Security (free) another excellent security plugin and have made use of BPS’s custom codes. I also have a CDN with firewall (paid) that keeps some bad actors from even getting to my site.

I can’t guarantee this will work for you but it’s easy enough to try and test if you’ve signed up with Google Webmasters. It’s free and you just have to verify you own your website.

It’s not a Wordfence issue, as I’ve tested on some of my clients websites that don’t have Wordfence and the issue is still there.

I have a sneaky feeling its something to do with Yoast but I need to look into it some more.

Wordfence is an excellent plugin by the way

Hello,
same situations here: got notification for three my sites, all my sites have WordFence, when I deactivate the plugin, JS a CSS are not blocked any more and everything just works…
Any idea what is the latest on the issues? I would not really like stop using WordFence, it is a great software…
Thanks for any insight…
Jan

Important update:
– only now i realized I did not update WordFence on two of the three sites, when I did it, problem disappeared (sorry for confusion), interesting piece of puzzle: the two sites does not run Yoast Seo (All in one SEO instead)

– so the only site that is not working properly has combination of WordFence and Yoast SEO (both fantastic plugins), could be just a coincidence, but worth mentioning…

still be glad for any insight…
jan

I have noticed that a site where I did not have robots.txt was not affected…

However that site was running yoast but not running wordfence.

@ben Please make sure you have updated to 6.0.14. We never blocked the wp-admin folder and caused the google warning. They have had this change in place since Oct 2014 but only started warning about it now. The only reason we are associated with it was because a call we made was flagged as well. We changed this behavior to address the issue and fix the issue for Wordfence. We never blocked css and js files.

tim

Ben,
I finally solved it with the rather “unscientific” approach suggested by SRzeus above. I simply regenerated robot txt and removed Disallow: /wp-admin/ as suggested by Yoast SEO.

And yes, it worked…

Jan

Thanks Jan!