Big phpXXXXXX files in /home/USER/.cagefs/tmp

Files were a bit bigger than I initially thought:

ll -htr /home/$USER/.cagefs/tmp/php??????
-rw------- 1 $USER $USER 344G Sep 14 10:44 /home/$USER/.cagefs/tmp/php7gIUGQ
-rw------- 1 $USER $USER 353G Sep 14 16:09 /home/$USER/.cagefs/tmp/phpp4YmGT
-rw------- 1 $USER $USER 352G Sep 14 21:24 /home/$USER/.cagefs/tmp/phpvjfIOe
-rw------- 1 $USER $USER 345G Sep 15 02:28 /home/$USER/.cagefs/tmp/phpNOd0nH
-rw------- 1 $USER $USER 365G Sep 15 07:32 /home/$USER/.cagefs/tmp/phpNaCuXj
-rw------- 1 $USER $USER 344G Sep 15 12:31 /home/$USER/.cagefs/tmp/php7TM8yO

I just downgraded alt-php70 to 7.0.21-1.el6.x86_64 and alt-php71 to 7.1.7-1.el6.x86_64 and so far the issue seems gone.

The issue still persists in both 7.0.21, 7.0.22 and 7.0.23 at least on EasyApache 4 (PHP versions are delivered by CloudLinux in this case).

Thanks for all the updates guys and thanks those of you who sent diagnostics and files. We have received one file and I’m going to get another one soon. The file we got appears to be a dump of some sort but it’s not a regular core dump. There is no reference to Wordfence but we see variable names and strings from other plugins.

At this point I have two more things

1. If you want to, you can try enabling “Delay IP and Country blocking until after WordPress and plugins have loaded (only process firewall rules early)” on the Firewall page. We are not sure at all that this will help, but if someone wants to test, please go ahead.

2. I’d like to know if anyone is seeing anything in their PHP error logs that could be related? If you don’t want to post here, please email it to [email protected].

Thanks!

@wfasa

> We have received one file and I’m going to get another one soon. The file we got appears to be a dump of some sort but it’s not a regular core dump. There is no reference to Wordfence but we see variable names and strings from other plugins.

The one I sent (being about 10 gigabyte), all the files in the zip file where the temp files generated from long running admin-ajax.php files with strace right before the file starts to grow being:

[pid  1247] lstat("/home/customer/public_html/customerk.com/wp-content/plugins/wordfence/wordfence.php", {st_mode=S_IFREG|0644, st_size=2177, ...}) = 0
[pid  1247] lstat("/home/customer/public_html/customerk.com/wp-content/plugins/wordfence", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid  1247] access("/home/customer/public_html/customerk.com/wp-content/plugins/wordfence/wordfence.php", R_OK) = 0
[pid  1247] access("/home/customer/public_html/customerk.com/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/init.php", R_OK) = 0
[pid  1247] access("/home/customer/public_html/customerk.com/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/waf.php", R_OK) = 0
[pid  1247] access("/home/customer/public_html/customerk.com/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/utils.php", R_OK) = 0
[pid  1247] access("/home/customer/public_html/customerk.com/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/storage.php", R_OK) = 0
[pid  1247] access("/home/customer/public_html/customerk.com/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/storage/file.php", R_OK) = 0
[pid  1247] access("/home/customer/public_html/customerk.com/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/config.php", R_OK) = 0
[pid  1247] access("/home/customer/public_html/customerk.com/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/rules.php", R_OK) = 0
[pid  1247] access("/home/customer/public_html/customerk.com/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/parser/lexer.php", R_OK) = 0
[pid  1247] access("/home/customer/public_html/customerk.com/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/parser/parser.php", R_OK <unfinished ...>
[pid  1247] <... access resumed> )      = 0
[pid  1247] access("/home/customer/public_html/customerk.com/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/parser/sqli.php", R_OK) = 0
[pid  1247] access("/home/customer/public_html/customerk.com/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/request.php", R_OK <unfinished ...>
[pid  1247] <... access resumed> )      = 0
[pid  1247] access("/home/customer/public_html/customerk.com/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/http.php", R_OK <unfinished ...>
[pid  1247] <... access resumed> )      = 0
[pid  1247] access("/home/customer/public_html/customerk.com/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/view.php", R_OK <unfinished ...>
[pid  1247] <... access resumed> )      = 0
[pid  1247] access("/home/customer/public_html/customerk.com/wp-content/plugins/wordfence/waf/bootstrap.php", R_OK <unfinished ...>
[pid  1247] <... access resumed> )      = 0
[pid  1247] access("/home/customer/public_html/customerk.com/wp-content/plugins/wordfence/waf/wfWAFUserIPRange.php", R_OK <unfinished ...>
[pid  1247] <... access resumed> )      = 0
[pid  1247] access("/home/customer/public_html/customerk.com/wp-content/plugins/wordfence/waf/wfWAFIPBlocksController.php", R_OK <unfinished ...>
[pid  1247] <... access resumed> )      = 0
[pid  1247] stat("/home/customer/public_html/customerk.com/wp-content/wflogs/",  <unfinished ...>
[pid  1247] <... stat resumed> {st_mode=S_IFDIR|0755, st_size=135168, ...}) = 0
[pid  1247] lseek(4, 0, SEEK_SET <unfinished ...>
[pid  1247] <... lseek resumed> )       = 0
[pid  1247] lseek(4, 0, SEEK_END <unfinished ...>
[pid  1247] <... lseek resumed> )       = 0
[pid  1247] write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192 <unfinished ...>
[pid  1247] <... write resumed> )       = 8192
[pid  1247] write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192 <unfinished ...>
[pid  1247] <... write resumed> )       = 8192
[pid  1247] write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192 <unfinished ...>
[pid  1247] <... write resumed> )       = 8192
[pid  1247] write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192 <unfinished ...>
[pid  1247] <... write resumed> )       = 8192
[pid  1247] write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192 <unfinished ...>
[pid  1247] <... write resumed> )       = 8192
[pid  1247] write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192 <unfinished ...>
[pid  1247] <... write resumed> )       = 8192

> 2. I’d like to know if anyone is seeing anything in their PHP error logs that could be related?

What I’ve seen is that after running for hours (when it reaches about 9 gigs in size), the php error logs will contain a memory allocation error, since it reached memory_limit of in this case 1 gigabyte.

[12-Sep-2017 02:01:40 UTC] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 534773792 bytes) in /home/user/public_html/domain.com/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/utils.php on line 839

Despite the biggest ajax request being made between 11 september 11.00 pm UTC and 12 september 9.00am UTC is 7.4 kilobytes.

2. I’d like to know if anyone is seeing anything in their PHP error logs that could be related?

In php error_log I do not have anything interesting.

I don’t know if this is related, but I have this in Apache error.log.

[Tue Sep 05 15:32:53.633376 2017] [lsapi:error] [pid 30675] [client EDITED:51799] [host https://www.EDITED.com] Client error on sending request(POST /wp-admin/admin-ajax.php HTTP/1.1); uri(/wp-admin/admin-ajax.php) content-length(739): user_get_body(tmpstackbuf, 16384): read from client failed, referer: https://www.EDITED.com/EDITED/

PHP downgrade didn’t seem to help after all. I now downgraded mod_lsapi and this seems to work. Will know for sure on moday I guess.

Do all you guys use mod_lsapi as well?

@shoentjen – servers with and without mod_lsapi has the same issue – I’d advise you to stop judging too quickly what works and what doesn’t – it just causes more confusion for everyone and doesn’t benefit neither Wordfence or other people that experience the issue.

@lucasrolff Sorry, It wasn’t my intention to confuse you.

That it happens on servers with and without mod_lsapi is new information. Nobody here mentioned how their PHP was loaded, and since we only use mod_lsapi ourselves *and* the issue appeared for us after we did an update containing both a newer mod_lsapi and new PHP7.x versions I thought it was a good thing to ask. To add to that, when you mentioned which ea-php70 packages you have, you listed php-litespeed. AFAIK this package is only useful if you use litespeed as a webserver or apache with mod_lsapi.

Anyyway, for me the issue seems fixed after I downgraded PHP 7.x and mod_lsapi, and restarted our webservers. I mean, .cagefs/tmp/php?????? file are still created, but their size is < 1MB where previously I saw files with a size of tens or hundreds of GB’s.

@shoentjen

> Sorry, It wasn’t my intention to confuse you.

Not to confuse me, but rather – giving information, that then later is “taken back” because you realize your “apparent” fix, isn’t actually a fix – makes things confusing for multiple parties.

> To add to that, when you mentioned which ea-php70 packages you have, you listed php-litespeed

Yes, of a single out of 30 boxes. Also just because you have the package installed, doesn’t mean all domains use lsapi, it’s quite easy to disable for a domain by setting another handler, or if you’re using php-fpm for certain sites.

> seems fixed after I downgraded PHP 7.x and mod_lsapi

It would be good to mention specific versions of your packages, there’s I believe 24x PHP 7.0.x releases, 10x 7.1.x releases.

For example we experienced the issue back in July as well and at that time we were using PHP 7.0.21 and lsapi 1.0-30

> Sorry, It wasn’t my intention to confuse you.

Not to confuse me, but rather – giving information, that then later is “taken back” because you realize your “apparent” fix, isn’t actually a fix – makes things confusing for multiple parties.

I might have been a bit too quick to post here again, yes. At least I am trying to be helpful.

> To add to that, when you mentioned which ea-php70 packages you have, you listed php-litespeed

Yes, of a single out of 30 boxes. Also just because you have the package installed, doesn’t mean all domains use lsapi, it’s quite easy to disable for a domain by setting another handler, or if you’re using php-fpm for certain sites.

Well, you didn’t say so and I had no way to guess. I still think my question if everybody experiencing the issue was using mod_lsapi was a very legitimate one that was helpful in determining the cause of the issue. But as I understand now, you are certain the affected users you see don’t use mod_lsapi. Do you care to mention what they do use?

> seems fixed after I downgraded PHP 7.x and mod_lsapi

It would be good to mention specific versions of your packages, there’s I believe 24x PHP 7.0.x releases, 10x 7.1.x releases.

We were experiencing the issue with:
alt-php70-7.0.23-2.el6.x86_64
alt-php71-7.1.9-2.el6.x86_64
httpd24-mod_lsapi-1.1-18.el6.cloudlinux.x86_64

After downgrading to the following, the issue was gone again:
alt-php70-7.0.21-1.el6.x86_64
alt-php71-7.1.7-1.el6.x86_64
httpd24-mod_lsapi-1.1-9.el6.cloudlinux.x86_64

For example we experienced the issue back in July as well and at that time we were using PHP 7.0.21 and lsapi 1.0-30

Now this would have been helpful to mention before. If you are sure it is the same issue: what at the time fixed it for you?

> But as I understand now, you are certain the affected users you see don’t use mod_lsapi

Yes, I’m always certain, I don’t comment on issues online unless being able to replicate issues again and again.

> Do you care to mention what they do use?

php-fpm, cgi, suphp, fastcgi

> Now this would have been helpful to mention before. If you are sure it is the same issue: what at the time fixed it for you?

It’s the same issue, same strace we had in july which is still happening – there’s been no fix – we just didn’t see it as a big issue since it only affected a few sites rarely (low-traffic sites), but recently some bigger (600-800k+ pageviews a day) sites started to use WordFence or upgraded their PHP version to 7.0 or 7.1, where the issue appears more frequently (multiple times a day).

>Do all you guys use mod_lsapi as well?

Yes, I only use mod_lsapi. I do not tested without it.

@rodriguezzz You might want to try downgrading mod_lsapi, see if that changes anything for you.

@wfasa any updates?

Hi,

Sorry for the delay. I’m working with @wfasa on this, but after reviewing some of the logs & data and doing some testing, last week became busy and I haven’t made any progress for a few days. I haven’t been able to reproduce the issue with or without cagefs yet, but from the sample files, it does seem like there is a PHP bug involved (likely a side-effect of something uncommon about the way we’re writing data), which we can probably work around if the issue can be narrowed further.

Has anyone found a way to reproduce the issue on demand?

-Matt R