Apparent attacks and why WordFence is mentioned in the logs

Hi @bjf2000,

This may not help, but you’re not on your own. I’m currently investigating this too as my server is getting bogged down in similar intervals. Your logs look very similar to mine.

As a temporary measure, I’ve had to remove WF, plus remove all the data via the WF assistant. Currently I’m using an alternative and utilised cloudflare’s ‘I’m Under Attack’ just in case, but it’s certainly a temporary measure. I’ll be keeping an eye on your post, and will let you know if I find anything further!

Thanks! Do you know when it started? And are the IPs involved also extremely UNsuspicious-looking in your case? If this is someone using a proxy, it’s the most realistic one I’ve ever seen, since all the IPs I’ve checked are close by and from local ISPs (seemingly). It’s one of the things that makes me think it’s not a deliberate attack but rather a malfunction.

An hour ago, I disabled Live Traffic, and so far things have settled way down, like is normally the case. It’s too early to know if that helped though, since during the last 24 hours (roughly the duration of whatever this is), there have been periods extending an hour or two where all appeared normal or close to normal.

Plus, we’ve had Live Traffic enabled for months, so why would it be doing this now? The only reason I thought to disable it is because in an old thread here, a rep tied the “GET /?wordfence_lh=1&hid” lines to Live Traffic, though that thread wasn’t about this problem, more a general query about the line.

I notice that v7.1.4 made a change related to Live Traffic, but if that were related, the problem should have started a week ago–unless WF just changed something additional server-side?

You’re not the only one. In the past 24 hours, I just got a sudden spike in CPU usage at my hosting company and they all turned out to be Wordfence related. Damn strange. Apparently Live Traffic view is to blame.

Example from logs:

223.225.56.122 – – [10/May/2018:17:18:45 +0800] “GET /?wordfence_lh=1&hid=372E32C95BE39A0632DA27A4E79117A9&r=0.6086598498347686 HTTP/1.0” 200 – “https://www.example.com/opinion/1437/time-rethink-ias/” “Mozilla/5.0 (Linux; Android 7.1.2; Redmi 4 Build/N2G47H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.126 Mobile Safari/537.36”
223.225.56.122 – – [10/May/2018:17:18:45 +0800] “GET /?wordfence_lh=1&hid=372E32C95BE39A0632DA27A4E79117A9&r=0.2711989207846879 HTTP/1.0” 200 – “https://www.example.com/opinion/1437/time-rethink-ias/” “Mozilla/5.0 (Linux; Android 7.1.2; Redmi 4 Build/N2G47H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.126 Mobile Safari/537.36”
223.225.56.122 – – [10/May/2018:17:18:45 +0800] “GET /?wordfence_lh=1&hid=372E32C95BE39A0632DA27A4E79117A9&r=0.30474362283357825 HTTP/1.0” 200 – “https://www.example.com/opinion/1437/time-rethink-ias/” “Mozilla/5.0 (Linux; Android 7.1.2; Redmi 4 Build/N2G47H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.126 Mobile Safari/537.36”
223.225.56.122 – – [10/May/2018:17:18:45 +0800] “GET /?wordfence_lh=1&hid=372E32C95BE39A0632DA27A4E79117A9&r=0.48325145812357806 HTTP/1.0” 200 – “https://www.example.com/opinion/1437/time-rethink-ias/” “Mozilla/5.0 (Linux; Android 7.1.2; Redmi 4 Build/N2G47H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.126 Mobile Safari/537.36”

• This reply was modified 6 years, 6 months ago by madmanweb.

I’m having what looks to be the same issue, sudden spike in CPU, sites going down, then everything goes to normal just to spike all over again.

So I disabled WordFence Live Traffic and it made a huge difference. How much? Well, see the graph for yourself.

Drop in CPU usage after disabling Live Traffic on Wordfence

Here’s another graph, with the timeframe being one week:

View post on imgur.com

You can see how all three are normal until, wham, Wednesday and most of today. At the extreme right are the last several hours of Thursday with Live Traffic off.

Interesting discussion. I’d like to entirely disable “Live Traffic” as I find it mostly useless, but can not figure out where that setting is. Where are you guys finding that? I can not find “Enable Live Traffic View” option in the incredibly concise and clear Wordfence user interface. I found the following documentation, but it doesn’t make sense as I can’t find this under “Options.”

In the documentation, Wordfence says “Options Enable Live Traffic View
This option enables or disables the Live Traffic View. On most servers that have sufficient resources, Live Traffic in Wordfence works flawlessly. In fact we enable Live Traffic on our own production servers which receive a significant amount of web traffic. However if you are using a low cost hosting plan that severely limits the resources you have available, you may consider disabling live traffic to reduce the load on your web server.”

Hi,

We’ve seen a few similar cases today, and so far, it looks like they’re all using CloudFlare’s “Rocket Loader”. Can you confirm if you are using this as well?

A page on CloudFlare’s site about Rocket Loader says:

Rocket Loader does have the potential to break some JavaScript and jQuery functions as a beta feature.

https://support.cloudflare.com/hc/en-us/articles/200169456-Why-is-JavaScript-or-jQuery-not-working-on-my-site-

If anyone still has the Live Traffic option enabled, you can try turning off Rocket Loader, at least temporarily. (You might also need to clear CloudFlare’s cache or wait for pages to expire, but I’m not certain.

The “wordfence_lh” hits normally only happen once per visitor, which helps separate humans from bots on the Live Traffic view. In this case, it looks like other javascript is likely interfering, preventing our code from working normally.

Mountainguy2: It’s the first option under the collapsible “Live Traffic Options” section on the Live Traffic page. It can also be found on the “All Options” page, either using your browser’s Find function, or the search box at the top.

-Matt R

Thanks Matt, are you referring to “Enable live traffic logging” ? That’s not “Enable Live Traffic View.” I’m mildly confused, please help. I want to entirely disable anything to do with “Live Traffic” that would use any server resources. I already have “Enable Live traffic logging” turned off.

Is your documentation wrong? As I quote above, your documentation states “Options Enable Live Traffic View This option enables or disables the Live Traffic View…”

Is this a mistake in the documentation? I looked all over the place and don’t see any “Enable Live Traffic View” option/switch.

Thanks, MTN

@mountainguy2 I don’t blame you for not finding it. I spent a lot of time going through the labyrinth of menus and options. Go to “All Options” and scroll for 5 minutes till you get to this:

View post on imgur.com

Yes, we do use CloudFlare and have had (all along) Rocket Loader on Automatic.

I can’t get away with experimenting with it today though, by turning it off and Live Traffic back on, so someone else will have to try that. On balance though, if it’s a choice between the two now, I’d rather keep Rocker Loader.

I have the exact same problem – I was notified by SiteGround that I had exceeded my CPU allowance and face outages. They went through my logs and pointed out the WordFence information:

Most executed scripts:
20721 https://raynergobran.com/index.php /home/raynergo/public_html/index.php
169 https://raynergobran.com/wp-cron.php /home/raynergo/public_html/wp-cron.php
30 https://raynergobran.com/wp-admin/admin-ajax.php /home/raynergo/public_html/wp-admin/admin-ajax.php

IP addresses with most hits:
[email protected] [~/access-logs]# cat ./* | awk ‘{print $1}’ | sort | uniq -c | sort -nr | head -n 10
3616 107.77.233.228
2978 38.126.139.1
1287 1.202.82.131
764 141.228.106.151
340 192.135.123.22
232 76.120.66.175

Access logs related to most active IP Addresses:
raynergobran.com-ssl_log:107.77.233.228 – – [11/May/2018:10:11:32 -0500] “GET /?wordfence_lh=1&hid=69D5BB809B58CCACBBB93ADB7ABBE28C&r=0.8341438933270944 HTTP/1.0” 200 – “https://www.raynergobran.com/2018/05/hedge-fund-data-hygiene/” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1 Safari/605.1.15”
raynergobran.com-ssl_log:38.126.139.1 – – [11/May/2018:10:42:09 -0500] “GET /?wordfence_lh=1&hid=020FD7A61C46BC6B554A18EC5286190C&r=0.3069151671291799 HTTP/1.0” 200 – “https://www.raynergobran.com/2017/05/biggest-hedge-funds-by-assets-under-management-may-2017/” “Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36”
raynergobran.com-ssl_log:1.202.82.131 – – [11/May/2018:10:19:08 -0500] “GET /?wordfence_lh=1&hid=69D5BB809B58CCACBBB93ADB7ABBE28C&r=0.02065667550572714 HTTP/1.0” 200 – “https://www.raynergobran.com/2018/05/hedge-fund-data-hygiene/” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36”

I have disabled RocketLoader on Cloudflare, but made no other changes. So I guess I will be the test case to see if it fixes the problem. I will only know in an hour or so when SiteGround’s activity logs are updated.

Any additional suggestions about how to do a controlled test would be welcome!

UPDATE: I have already seen a dramatic decline in last 2hrs CPU time usage (20,000+ -> 3,000+) and last 2 hrs account executions (25,000+ -> 4,500+). I made the change about 35 minutes ago.

Ian

PS: I flushed the cloudflare cache after making the change to rocket loader

• This reply was modified 6 years, 6 months ago by blueeventhorizon. Reason: Additional info in "PS"
• This reply was modified 6 years, 6 months ago by blueeventhorizon.
• This reply was modified 6 years, 6 months ago by blueeventhorizon. Reason: Update on progress

Just an FYI:
https://community.cloudflare.com/t/rocket-loader-update-is-active-in-wp-admin/17900

There have been some changes to RocketLoader which have been rolled out to a subset of zones. Additional changes/ tweaks may be forthcoming.

I’ve asked the team to look into possible WordFence issues.

I believe a blog post will be forthcoming. I don’t know the full details on the changes, but Rocket Loader has been getting some serious love from the development team.

@bjf2000: Thanks for the link. I understand not being able to test the changes on a live site. I have a site with cloudflare for testing compatibility issues, and that site doesn’t get the same rocket-loader.min.js file that I’ve seen on a couple others with reported issues — that must be due to the gradual rollout mentioned in that link. (I get a rocket.min.js with a comment dated back in February.)

@blueeventhorizon: Thanks also for confirming that disabling Rocket Loader helped on your site.

Hopefully their code will be adjusted to prevent this type of issue, based on the feedback in the link above — though we’re also looking at adding another check in case of similar issues (either in future Rocket Loader versions, or in any other scripts that might modify/remove/call other scripts’ event handlers), in our next release or two.

-Matt R

@wfmattr. Thanks, sounds good, especially the part about adding in some kind of general protection for this sort of thing in the future. There’s always going to be random things like this out there.

Here’s what the source for our site shows for Rocket Loader. I don’t see any date in it, though no doubt it’s the new one (and unless it was a typo, maybe you’re saying that the old one has a different filename, which would make them easy to distinguish).
https://ajax.cloudflare.com/cdn-cgi/scripts/dba9ecf7/cloudflare-static/rocket-loader.min.js