• Resolved Tranny

    (@tranny)


    I run a busy high traffic and high community participation site, so MySQL has always run very high on my server. But after upgrading to v4.3 (no other change), my CPU load tripled, and it’s crippling the server, causing frequent crashes and generally degraded responsiveness.

    The site is hosted on a high end dedicated server with 4 SSD drive in hardware RAID. I spoke with my host and they assured me no changes have been done to the server, and all server side applications are up to date.

    Problems started after the update to v4.3, No other change whatsoever has been done to neither the WP installation, nor the server itself.

    My log shows high http connection count, which use abnormally high CPU never seen before:

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    17368 mysql 20 0 9348m 441m 6852 S 42.0 1.8 8:35.43 mysqld
    17941 xxxxx 20 0 1474m 54m 4748 S 27.5 0.2 0:10.38 httpd
    18526 xxxxx 20 0 1463m 43m 4680 S 27.1 0.2 0:01.42 httpd
    16675 xxxxx 20 0 1467m 46m 4284 S 24.8 0.2 0:17.30 httpd
    16830 xxxxx 20 0 1463m 42m 4300 S 23.5 0.2 0:16.92 httpd
    17777 xxxxx 20 0 1454m 34m 4780 S 23.5 0.1 0:11.56 httpd
    18358 xxxxx 20 0 1460m 40m 4760 S 23.5 0.2 0:04.49 httpd
    17881 xxxxx 20 0 1460m 40m 4716 S 23.2 0.2 0:09.88 httpd
    18220 xxxxx 20 0 1441m 21m 4768 S 23.2 0.1 0:05.12 httpd
    18472 xxxxx 20 0 1460m 40m 4748 S 23.2 0.2 0:04.12 httpd
    18331 xxxxx 20 0 1505m 84m 4700 S 22.5 0.4 0:04.02 httpd
    16892 xxxxx 20 0 1463m 43m 4316 S 22.2 0.2 0:09.67 httpd
    16872 xxxxx 20 0 1441m 20m 4240 S 17.9 0.1 0:21.70 httpd
    18248 xxxxx 20 0 1446m 26m 4756 S 16.9 0.1 0:03.51 httpd
    16479 xxxxx 20 0 1461m 41m 4688 S 16.2 0.2 0:19.24 httpd
    16870 xxxxx 20 0 1458m 37m 4216 S 16.2 0.2 0:16.59 httpd
    18452 xxxxx 20 0 1474m 54m 4652 R 14.2 0.2 0:01.97 httpd
    16729 xxxxx 20 0 1461m 40m 4208 S 12.2 0.2 0:18.65 httpd
    16978 xxxxx 20 0 1455m 35m 4248 S 9.3 0.1 0:11.55 httpd

    Can anyone help me out with this? What happened? This is the first time I see my server getting crippled after a WP update and I’ve run this particular site on WP since 2008.

Viewing 15 replies - 16 through 30 (of 96 total)
  • Moderator Samuel Wood (Otto)

    (@otto42)

    www.ads-software.com Admin

    UPDATE: WordPress 4.3.1 has been released and it contains an automated fix for this issue. Please update to 4.3.1 instead of following these instructions.

    After reading through that ticket, I can see this being the cause of extreme load in some cases. Here’s how to fix the problem. You will need shell or file access to the server.

    Step 1. Fix the actual issue by applying part of the patch in that ticket. It’s a one liner. Open the wp-includes/taxonomy.php file, and go to line 4448. This is that line:

    wp_schedule_single_event( 'wp_batch_split_terms', time() + MINUTE_IN_SECONDS );

    The problem is that the arguments are reversed. Switch them, like so:

    wp_schedule_single_event( time() + MINUTE_IN_SECONDS, 'wp_batch_split_terms' );

    That will fix the underlying issue, but it won’t stop the load. To do that, we’re going to make a temporary mu-plugin.

    Step 2. Navigate to your /wp-content directory. Create a subdirectory called “mu-plugins”, at /wp-content/mu-plugins. The name of the directory is important. If you already have that directory, just go in there.

    MU Plugins are “must use”. All PHP files in this directory get auto-included by WordPress. It’s a handy way to run some code in your WordPress instance without having access to the Plugin page to activate plugins.

    Create a new file called “fix.php”. Or anything, the name doesn’t matter. This will be the contents of that file:

    EDIT: This is the new code which will not cause additional load on your site. The old code has been removed as it was problematic for some cases.

    <?php
    function clear_bad_cron_entries() {
    	// Fix incorrect cron entries for term splitting
    	$cron_array = _get_cron_array();
    	if ( isset( $cron_array['wp_batch_split_terms'] ) ) {
    		unset( $cron_array['wp_batch_split_terms'] );
    	        _set_cron_array( $cron_array );
    	}
    }
    clear_bad_cron_entries();

    This is the same code in 4.3.1 to clean up the problem. You need it to run now instead. So paste that code into the fix.php file, and save.

    Now, navigate to your site. The problem should clear up shortly. Once it does, you can delete the fix.php file. It only needs to run once, not every time your site loads.

    If you are using multi-site, then you need to load every site at least once to allow this code to run on all of them. Cron jobs are stored per-site, not per-instance.

    Thread Starter Tranny

    (@tranny)

    Gonna try to follow your instructions now. Still, I hope an update that will address this issue will be released soon.

    I can validate that the fix above solves the problem – Please be aware that the “FIX.PHP” in mu-plugins gave me a astronomic load for aprox. 4 minutes. I have 4 cores, and I was at 17 – It fell to below 1 after 4 minutes. and now the CPU load is back to prior 4.3 levels.

    Implemented the fix, although things seem to be running better I get a white screen after hitting ‘Publish’ for posts. The posts do indeed publish so no content is lost but having the white screen to navigate from is concerning. Server my install is on is running Nginx if that makes any difference as most other sites run Apache.

    I’m running Nginx + CloudFlare + Varnish

    I get no white screen after implementing the script

    Thread Starter Tranny

    (@tranny)

    @hummelmose

    I had the same experience with astronomical load for a while after running the fix ??

    Thanks for reporting this and posting a sollution, I found it when Updraftplus Backup plugin reported (in various websites of mine) that I had scheduled tasks with a date in the past. When looking into it, I eventually stumbled on this bug-report.

    I’ve corrected line 4448 in wp-includes/taxonomy.php and deleted the events used a cron manager plugin to delete the ‘wp_batch_split_terms’ entries, but that only corrects part of my problem, since there are also (and still) about 590 entries that don’t show in any of the cron plugins because they don’t have a hook (or any data) in it. I don’t know how to change the code mentioned above to delete those entries.

    When I echo the output of _get_cron_array() it starts out with a lot of entries like this, before showing entries with hooks and args. It might be because some other (previous?) bug, but I don’t like having this in my db.

    Can anybody please advice on how to delete these entries?

    Array
    (
        [1430901747] => Array
            (
            )
    
        [1430901752] => Array
            (
            )
    
        [1430901757] => Array
            (
            )

    Thanks, Hans

    While my blog’s performance really wasn’t impacted, I installed WP Control ( https://www.ads-software.com/plugins/wp-crontrol/ ) just to see what was there and I had about 10 of these. Anyway, I made the change to taxonomy.php and created the fix.php mentioned previously and it cleared them out.

    Same Problem. Here. Seems the solution is working, but how long does it take to clean it all. My server is spawning endless httpd processes until all resources are full and then colapses

    So I’m not an expert on code Samuel, but in this statement…

    Now, navigate to your site. The problem should clear up shortly. Once it does, you can delete the fix.php file. It only needs to run once, not every time your site loads.

    Are you saying your Fix.php will be executed every time ANYONE comes to the site? I’m going to go ahead and guess that is one reason your script is causing a horrible load on our servers. As those of us who have this issue, are already in this mess because wp-cron is executed EVERYTIME someone visits our site. So if we get 1000 hits per hour, we’re getting thousands of new created jobs. But now we’re also getting tons of instances of Fix.php running? Correct me if I’m wrong, but your fix seems to just be making the issue worse for those of us with heavily used wordpress installs.

    So happened to me with 400 users i got a massive number of jobs created and the server fell appart.

    Is there an easy way to delete all the jobs this fix created ?

    Moderator Samuel Wood (Otto)

    (@otto42)

    www.ads-software.com Admin

    Are you saying your Fix.php will be executed every time ANYONE comes to the site?

    Yes, which is why you only let it run long enough to clear out the excessive entries, and then delete the file. You don’t *leave it there*, you run it only once and then remove it.

    The purpose of the fix.php is to clear out all those excess cron entries. That’s what that code does.

    Fixing the taxonomy.php file will prevent new jobs from being created. Running this code will clear out the invalid existing jobs. Then, problem solved. It only takes one run. Maybe a minute or two to clean things up, tops.

    Yes, which is why you only let it run long enough to clear out the excessive entries, and then delete the file. You don’t *leave it there*, you run it only once and then remove it.

    Well. Why put it in the mu-plugins then. Wouldn’t it have been smarter to just call the fix.php from another folder once ?

    Moderator Samuel Wood (Otto)

    (@otto42)

    www.ads-software.com Admin

    Well. Why put it in the mu-plugins then. Wouldn’t it have been smarter to just call the fix.php from another folder once ?

    Well, I could make a much more complicated script and give you instructions on how to run it, or you could just do this simple fix one time and get on with things. It’s not a big deal to let it stay there, as such. It’s just not necessary. If you don’t have a bunch of excess cron jobs, then the script doesn’t do anything.

    Simplicity works better than complicated solutions.

    The fix does not create load. It solves the load by removing all the excess cron entries. Now, that does take some time and processing power, especially if you have a lot of entries. Still, once they’re gone, then the load drops.

Viewing 15 replies - 16 through 30 (of 96 total)
  • The topic ‘High CPU Load After Update to v4.3’ is closed to new replies.