HHVM crashed

Hi Donghyeok Kang,

Can you run this from the command line?

hhvm -l /path/to/wp-content/wflogs/rules.php

This should let us know if the Wordfence Firewall rules have generated correctly.

Questions:
Have you been using Wordfence on HHVM for a while now?
Is the first version of Wordfence that has caused any problems?
On the Wordfence Firewall page, what is the current Firewall status?

Experienced the same had similar stack trace with waf.php in it (lost it due to a reboot).
WordFence 6.1.12
HHVM 3.14.2

Firewall was Disabled when I checked.

hhvm -l /path/to/wp-content/wflogs/rules.php

didnt report any issues.

Now removed the auto-prepend in my php.ini to make sure it doesn’t happen again.

I ended up de-installing HHVM and switching over to PHP7-FPM.
Much more stable and the same or even better speed.

I think the issue is more related to HHVM than to the WordFence plugin.
In my case it would crash as soon as the Googlebot was crawling my site.

Hi wflandon,

/path/to/wp-content/wflogs/rules.php file exists.

hhvm -l /path/to/wp-content/wflogs/rules.php

This prints nothing.

I have been using Wordfence on HHVM for more than 6 months.
And this is the first case that HHVM crashes.

Now I am running PHP7-FPM but I will try HHVM again later.

Thanks.

Just one more “me too,” I moved a site to a new server last week and set it up with hhvm, and having the same type of crash.

hhvm: 3.14.3~jessie
wordfence: 6.1.12

Also nothing printed by hhvm -l wp-content/wflogs/rules.php

This has happened more than once, but I have the logs/stacktrace (and even hhvm bytecode repository) from last night, when another crawler (91.121.79.180) was crawling this site (a small woocommerce storefront).

# cat /tmp/stacktrace.24250.log

PHP Stacktrace:

#0  include() called at [/var/www/clients/client3/web8/web/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/waf.php:225]
#1  wfWAF->loadRules() called at [/var/www/clients/client3/web8/web/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/waf.php:134]
#2  wfWAF->run() called at [/var/www/clients/client3/web8/web/wp-content/plugins/wordfence/waf/bootstrap.php:259]
#3  include() called at [/var/www/clients/client3/web8/web/wordfence-waf.php:6]

And if helpful (probably not?), this was on console where I had restarted hhvm the previous day:

Assertion failure: /tmp/tmp.Yo6dgSlEaj/hphp/runtime/vm/jit/mc-generator.cpp:1713: bool HPHP::jit::{anonymous}::mcGenUnit(HPHP::jit::TransEnv&, HPHP::jit::CodeCache::View, HPHP::jit::CGMeta&): assertion '0' failed.
data block = cold
message: Attempted to emit 1 byte(s) into a 1338 byte DataBlock with 0 bytes available. This almost certainly means the TC is full. If this is the case, increasing Eval.JitASize, Eval.JitAColdSize, Eval.JitAFrozenSize and Eval.JitGlobalDataSize in the configuration file when running this script or application should fix this problem.

> Have you been using Wordfence on HHVM for a while now?
> Is the first version of Wordfence that has caused any problems?

Just put this site on new server/hhvm last week, so no prior baseline.

> On the Wordfence Firewall page, what is the current Firewall status?

Protection Level: Extended Protection
Firewall Status: Enabled and Protecting

I’ll switch to php-fpm for how; let me know if I can get more info that would help track this down.

One thought: this completely crashed the hhvm service for this website. That can’t be (only) a wordfence problem; no matter what php code you throw at the interpreter, it’s gotta be a bug in hhvm if it crashes the hhvm process.

I agree with you at some point.
Yes, it can be a HHVM bug.
But this crash is a recent problem after the wordfence update.
I think some changes affects this kind of crash and wish that you could fix it.
Thanks.

This is a very small/early improvement (possibly absolutely meaningless), but I had hhvm crashed when I came in the last 2 mornings, and this morning it was still running after changing the 4 settings referenced in my console message above.

I went over to https://docs.hhvm.com/hhvm/configuration/INI-settings#jit-settings__jit-translation-cache-size and looked up the default values and then doubled them, via adding to my site php.ini:

hhvm.jit_a_size = 125829120
hhvm.jit_a_cold_size = 50331648
hhvm.jit_a_frozen_size = 83886080
hhvm.jit_global_data_size = 31457280

I’m guessing it was jit_a_cold_size that I had a problem with, given the data block = cold in the error, but bumped them all for good measure.

This website is running the X theme with revolution slider and woocommerce, and not terribly large/bloated with plugins, so may be a common issue wordfence will encounter. If this proves to be “the fix” (or rather, tweaking those appropriately), maybe wordfence could determine if hhvm is in use and recommend adding those to the php.ini for a website just like they specify the auto_prepend setting.

the victory was short-lived, hhvm crashed again with the same error as earlier (from syslog this time):

Assertion failure: /tmp/tmp.2YEszDTIHd/hphp/runtime/vm/jit/mc-generator.cpp:1713: bool HPHP::jit::{anonymous}::mcGenUnit(HPHP::jit::TransEnv&, HPHP::jit::CodeCache::View, HPHP::jit::CGMeta&): assertion '0' failed.
data block = cold
message: Attempted to emit 1 byte(s) into a 1410 byte DataBlock with 0 bytes available. This almost certainly means the TC is full. If this is the case, increasing Eval.JitASize, Eval.JitAColdSize, Eval.JitAFrozenSize and Eval.JitGlobalDataSize in the configuration file when running this script or application should fix this problem.

Hey Guys,

We were able to recreate this error, however, I just don’t see how this is Wordfence creating it.

To try and recreate the issue, we setup an environment with WordPress / Wordfence / HHVM, and let it run a few days. It did eventually break in the way the original poster described. The HHVM process actually dumped core and quit running. On our installation, it did not restart itself.

I feel like it has to be an HHVM bug. I don’t see how a bug in PHP code alone would be able to cause that kind of HHVM issue.

Feel free to post any information you think might be helpful. I will continue to follow up periodically and let you know if we have anything as well.

Definitely a hhvm bug. To report that they understandably want you to compile and run the latest hhvm version, and if possible have a reproducable setup. I wonder if we could narrow down what wordfence is doing when that happens to help reproduce it? Eg. it’s always handling a request that gets blocked, or always running a site scan or anything.

Also if anything like that could be identified, maybe wordfence could do whatever it’s doing a little differently, and avoid triggering the hhvm bug.

I don’t know much about how/what bytecode hhvm caches, and in particular if eval() code is cached, but there’s not any recursive eval going on or something that would generate crazy levels of php calls is there?

For now I’m bumping up the relevant jit_*_size variables whenever this happens to see if it ever gets stable. Once I’m satisfied of that, I’ll simply turn hhvm off for this site (and maybe every other wordpress/wordfence site?), as the memory usage is crazy (hhvm daemon is >256M RSS) for a simple site like this.

Also seen this recently. I think it first appeared about 6 months ago, before that I don’t recall it breaking so often.

Any tips for bis we could switch off?

If it helps, in our logs this was where it broke:

#0  include() called at [/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/waf.php:228]
#1  wfWAF->loadRules() called at [/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/waf.php:134]
#2  wfWAF->run() called at [/wp-content/plugins/wordfence/waf/bootstrap.php:282]
#3  include(/wp-content/plugins/wordfence/waf/bootstrap.php) called at [/wp-content/plugins/wordfence/wordfence.php:54]