I'm getting massive amounts of attacks from this beastie

NinjaFirewall cannot block this because it is already blocked by your HTTP server: you can see the 404 (Not Found) error code.
The problem is that you seem to have the default WordPress .htaccess which contains that code:

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress

It redirects 404 errors to the index.php page of your blog (26Kb according to your log), which will load WP, its plugins and will send many SQL queries as well. When you are under attack, such rewrite rules will make your CPU jump to the roof.
Consider removing or commenting out those rules from your .htaccess. Your HTTP server will simply return a 404 page not found.

Ah, I get it. What you have outlined is exactly the problem. I thought that all these bots were attacking index.php and I couldn’t work out why they would all just repeatedly hit index.php. Now I realise that they are all 404 errors being redirected to index.php. Then, when I block the IP it becomes a 403 error and none of that WordPress + plugins stuff loads and, as you state, the CPU usage goes right back down.

How much of:

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress

do I need to remove? All of it?

BTW You guys are amazing. I’m going to write a 5 star review of a product that is indispensable to this oft-misunderstood aspect of running a WP website.

You can safely comment out all lines between ‘BEGIN WordPress’ and ‘END WordPress’ during the attack.

‘during the attack’ – So this isn’t something that I can permanently do? I guess the effect of this is that I’m effectively removing WordPress’s 404 catch-all and I can’t do this permanently?

You can replace it with this:

ErrorDocument 404 /404.html

Then, create your own nice and light (a few hundreds of bytes) 404.html page and upload it inside the same folder as the .htaccess. 404 errors will show that page.

Athough I love the genius of your solution, I couldn’t get it to work in my DEV environment. I created a nice little basic HTML page and everything. Upon reading more I’ve become aware that WP permalinks don’t really work so well with ErrorDocument 404 entries. APparenly 403s are fine but 404s don’t work. But I found another way. I’ve enabled the ‘Cache 404’ in the Page Cache of my W3 Total Cache plugin. I think this will work perfectly. I’m now sitting in front of the server logs with popcorn… But thanks you for your time!

I’ve been having the same issue. Do you run ads on your site? I believe I traced mine to a problem with at least one of my ad networks.

If I had to guess, it’s a coding error on their side that generates that “undefined” part of the URL that results in a 404. That’s probably supposed to be something like “https://www.adnetwork.com” and be appended to the part following “undefined.” Since it isn’t, all that’s being requested is:

undefined/fp?zoneid[plus the rest of it]

Your server thinks that’s a local file and generates a 404.

I was able to alleviate it by configuring my varnish server to generate its own 404 header if it sees “zoneid” in a URL.

You can probably do the same in your .htaccess file. Here’s an example:

Of course, you want to redirect to a static page rather than index.php

Hope that helps.

Hello vhagerty,

I think you are exactly right! Let me guess, you are with the sovrn / lijit advertising network? I found that the zone ID married up perfectly with the id of one of my ad zones. My fairly brutal fix was to add this:

RewriteRule ^(.*)undefined/fp(.*)$ - [R=403,NC,L]


to .htaccess.

Basically, we’ll never have a URL like that ever in a million years so I’m keen just to 403 it away.

Things I’ve taken away from this:
1) Ad networks will always be looking at new ways to f*** *** up
2) I will never laugh at my cat chasing her own tail again
3) WordPress is bad at statically handling 404s and I’ll need to work out a more server-efficient way to do this than through 404.php

Cheers for your help! What a brilliant call on your part!

Yes, we use sovrn. I think this is something fairly new and might actually be something specific to certain browsers, as it only shows up in my logs with “AppleWebKit/537.36” in the user-agent string.

I sent sovrn an email about it about an hour ago. Based on my experience with them, I’m actually pretty confident they’ll be looking into it.

In any case, glad you were able to resolve it.

CHeers, yes the full UserAgent we are getting is:
Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.103 Safari/537.36

A related question: I thought that the WordPress default in .htaccess is needed for the permalinks to work. I made the assumption because that seems to show up when you enable permalinks. So removing it doesn’t break the permalinks?

I’m so incredibly happy I found this thread. We have been fighting an issue that sounds identical to what is mentioned here. Site visitors have been experiencing slow page loads, 500 and 508 errors, cannot establish DB connection errors, etc. On the backend, our server has been running a constant 50% CPU and has been maxing out on physical and virtual memory, IO, IOPS, processes, everything! Because of that, we’ve been experiencing a bunch of core dumps on the server. It’s been painful and frustrating trying to find the cause.

I hired a developer and he found tons of 404’s for */undefined/fp. We also run sovrn so I’m pretty sure this is the issue because our code has not changed and I started noticing the problem about a week ago . We’re implementing the change now so we’ll see if that fixes it.

Question for the group: Has anyone experienced an issue where your page loads just fine, but then the sovrn ad overtakes the screen…meaning the screen is all white except for the sovrn ad running on the top left corner? We’ve experienced this issue randomly and am wondering if this might be related to /underfined/fp issue. If what vhagerty said is correct and there’s a coding issue on the sovrn side (and I believe that’s the case), that might manifest itself in this manner as well. Just wondering if anyone else has experienced this.

How did you get on hesnctrl? I haven’t seen that issue where the sovrn ad overtakes the screen. What Browser/OS have you seen this on? This sounds like it could be the issue that just starts the random “attacks” on the Server…

Its happening on all browsers and OS’s. We’re experiencing it here on OS X and both Safari & Chrome and multiple site visitors have reported it to us on Windows and IE. I’ve been in communication with sovrn about it and they initially couldn’t track down the cause even though I sent them the complete page source. A few days later, they said that they removed some possible offending ad demand. I’m not sure if the file that has the code issue comes with the ad itself or is part of the ad unit. I’m going to let my sovrn rep know about the /underfined/fp issue so they can see if it’s related. I’ll report back what I hear.

If you use VPS, you can block these attacks with fail2ban. Very easy.