“500” errors, server crash and redirect errors

1) “500” error (possibly linked to the “PHP Fatal error: Allowed memory size of 268435456 bytes exhausted” that I see in my apache log)
2) Server crashes every 3-4 days.
3) Getting several thousand “Request exceeded the limit of 10 internal redirects due to probable configuration error” errors a day in my apache log.

Well THERE’S your problem. Except for item #3, you’ve got memory issues. Item #3 means your site is successful, or you’re being DoS’ed. Either one will do that.

To try and fix this issue I have tried allocating as much as 1GB! of RAM in the PHP.ini file and in the wp-config.php file but to no avail.

I also tried moving to another server but the problem remains.

It is almost certainly a software issue and an amok plugin or theme will do that. One plugin did that exact same thing for my install and I get 1/100 of your traffic. Adding more RAM won’t fix it because with enough requests you’ll run out of any size pool.

Can you get your host to graph out memory utilization? On my server I was able to install Munin and once I found the errant plugin I was able to confirm that I kept below the physical RAM on my VPS.

You’ll need to get medieval with your plugins until you find the problem. Since you’ve hundreds of sites that may not be practical so your need to profile memory usage of your themes and plugins.

Do a top via ssh. Look at your access logs – are you being hit by bots repeatedly?

@ Jan Dembowski
I have contacted my host and expect an answer regarding Monin soon.
After your replay I have setup a ghost server to test whether a plugin is causing this problem or not. I will finish that test later today.

@ Andrea_r
I have talked to my host and they say that neither there firewall or network monitor system has detected any irregularities that would suggest an attack.

Any other suggestions as to what plugin is the most likely to be the instigator?
Plugin list:
Exclude Pages from Navigation
Hierarchical Pages
MapPress Easy Google Maps
MCE Table Buttons
Members
No Comments On Pages
Spectacu.la Page Widget
WordPress MU Domain Mapping

Having Munin or similar will at least graph your memory usage so you can identify around when you’re about to hit the memory ceiling. And it’s not necessarily an attack as much as a flood of requests from search engines. Identifying the trigger helps find the issue.

That’s not a large number of plugins and identifying if it’s one of those is a trial an error exercise. When I ran into this, it was a Flickr plugin for me.

What about the themes you have installed for your sites?

What Jan said. ??

We have a long term site which is pretty quiet now, yet needs to be on a dedicated server just to handle things like bots and google coming around looking for feeds that haven’t been updated in years.

@ Jan Dembowski
We only have one main theme with about 30-40 childthems on the server. They are all made with the “Thematic” template.

@ Andrea_r
I can’t see any evidence of bot activity in the logs. Do you know where I should be looking and what for?

Look for a bunch of gets, all in a row, like this:

/wp-content/themes/newsworld-1.0.0/thumb.php?src=/g0../0d1.
/wp-content/themes/DeepFocus/tools/timthumb.php?src=/g0../0
/wp-content/themes/TheProfessional/tools/timthumb.php?src=/

Ha! I totally forgot about that pseudo DoS. Until I did the .htaccess thing, I kept getting smashed too.

wpfranner if you do see those thumb.php GETs in your log, give this a read.

https://tech.ipstenu.org/2011/timthumb-pseudo-ddos-effect/

Do you know where I should be looking and what for?

anything that has the word “google” in it. ?? I’m not being facetious either – I’ve had to temporarily block it and force them to come by less often.

+1 Andrea :/ Bing Bonged me the other day.

I looked around for a plugin or script that could curse the memory error. I found something really strange. Somehow, the users on certain sites have started replicating almost endlessly. On one site I found that one of the Admin users was duplicated 25.000 times… see this image

The user only appears once on the list but in the _usermeta table in DB it is duplicated A LOT (90.176 lines of data). There are several of our users that this has happened to according to the DB table.
The usermeta table is now at 16MB in raw sql dump…. that doesn’t really seem right.

Does anyone have an idea on what is happening here?
We have the Members Version 0.1.1 By Justin Tadlock installed on the server. Could that plugin cause this?

I would deactivate it and other user plugins to rule them out at the very least.

(good hunting)

I have Members on a couple times without any issues, FWIW. But yeah, that looks way shady to me!

Solution
So I finally got the answer to the problems I described. It was indeed the _usermeta table that was causing the problems.
After emptying the entire _usermeta table, the serverload went down by almost 70 %! I had to remap all the users to the different sites which took almost 12 hours.

Now my sites are lightning fast and I get no errors at all.
As to what caused the _usermeta table to run amok in the first place is anyone’s guess. I have updatet all my plugins and WP and no user copying has taken place yet.
Thanks for the help all!

What plugins were/are you running?