high traffic crashing usmagazine.com

@willhines – Matt mentions Owen. Owen = ringmaster ??

Yeah, I’m not interested in Lightpress if only because its support community is not as evolved. And I’ve never heard of it before today. WordPress is great and FEELS right — I’m going to keep pushing to have it be our main tool, though if I can’t enable comments I will lose that battle.

Thanks, Matt and Owen and everyone for the fast response. I’ll look into getting more memory. I’m totally fine with getting a bigger box if I can convince mediatemple to let me(?). I’ll also ask them about installing a PHP opcode cache.

I shouldn’t have posted the link to the 1000/month server. I wasn’t suggesting you get that server, but was trying to make the point that if you want to be a safe, get the best decicated server you can afford. The next time you get a burst of activity, will your optimizations handle it? When you can answer that question with confidence, stop where you are. You’re not trying to handle your average traffic, you’re trying to handle your bursts.

I’ve gotten a mandate: we’re moving to Movable Type. Sucks! I first discovered WordPress two years ago when I got sick of waiting for MT to publish pages. But now the allure of flat HTML has won out. Not my decision. I guess if the ISP will support this, then it’ll be better for me, and I can keep using WP for other projects.

Thanks, everyone.

That’s just silly… but oh well.

This thread might indicate that the Codex needs some pages on VPS optimizations and performance tuning for large sites.

It’s infuriating.

Lists of optimizations are good. Even better would be a handy list of highly-trafficked, high-profile sites that use www.ads-software.com. One mention of “huge-site-you-have-heard-of.com uses this” is more powerful than a page of statistics.

Well, I guess it’s too late now, but you *did* verify that WP-Cache was creating cache files and serving them up, right?

I did confirm it, and it was a huge help. Brought the server load down from 8 to about 1 — thwarting our first burst of traffic.

But we had two more bursts since then which overwhelmed despite WP-Cache.

It’s when a post is heavily commented that seemed to kill us. The Us readers, when they latch onto a post, comment several times a minute. That, combined with the crush of traffic going to the post (“Secret dinner for Nick and Jessica”) — would do it. I had just pinpointed that as a stress point when I got the word that we had to switch. It’s still just a hunch, but by disabling comments on popular posts I’m keeping the site up.

I said “Ok. It’s $200.”
“It’s not free.”
“Nope.”
(hands over credit card).

I’m SURE WordPress could have worked. But the ISP, or whoever controls the server, is not on board — well, then it can’t work. I’m sure MT will have trouble, but since it was their suggestion I assume they’ll battle through whatever problems surface.

High Traffic:
https://codex.www.ads-software.com/User:Matt/Famous_Blogs

I hope you come back and read that list……

Hmmm, when I just visited the homepage, it was not served from the cache.

Also, the comment counts certainly mean every time there’s a comment, at the least the post and the homepage(possibly) need to be flushed. I haven’t looked into the guts of wp-cache to see when it force-flushes other than ever N hours.

However, that said, the 51 comments on that post were generally every few minutes, rather than a few per minute. That’s still MUCH less than the potential hit rate of say 20-50 hits per minute. But again, note, it is just as possible you are getting crushed at the server-resource end if you are spiking to hundreds of connections per minute (i.e., many per second). A slow VPS with low memory could just be sunk.

Certainly, if you aren’t getting static pages served up whenever possible, that’ll hurt if you’re being dugg or slashdotted, etc.

On a very heavy site, I’d consider delaying cache updates due to comments until maybe every few minutes (and comment-count updates at the same time).

But, we’re working with imperfect information here. Certainly, even a static HTML system is going to have to refresh stuff every time a comment is added (again, the post and the homepage if the post is on it…).

-d

You’re right — the commnent rate is not as bad as a few a minute (it was a few a minute on the Britney pregnant post, but that was just for a short burst).

I hate it when people use this forum as tech support, but I’ll put this here in case I can save someone else headaches:

I see this in the source on a single post page:
<!– Dynamic Page Served (once) in 13.895 seconds –>
<!– Cached page served by WP-Cache –>

And I see files being created in the wp-content/cache directory. So I think I have wp-cache working.

I guess since you’re moving to MT, it doesn’t matter, but I thought I’d give you some feedback anyway.

I’m not getting the first page from the cache. I entered https://usmagazine.com and let it do it’s thing. These two were done about 30 seconds apart. This is what was at the bottom of the html for each.

<!– Dynamic Page Served (once) in 15.743 seconds –>
<!– Dynamic Page Served (once) in 15.032 seconds –>

No mention of the Cached page, and the times are different.

And after typing the above, I went back. Still no cache, but somewhat faster:

<!– Dynamic Page Served (once) in 4.358 seconds –>

good luck in wherever you end up.

I have posted an article on optimizing WordPress for highly commented blogs. This will solve your problem.

Give it a try even if it is with simulated loads. Then maybe you can convince your management back.

Wish you best of luck.

My suggestions are pretty much the same as Owen’s.

1. If you’re dipping into swap, add RAM.
2. If Apache’s connections are maxed, up the limit
3. If MySQL is running slow, allocate more RAM to it (see number 1) and:

• Install the plugin that freezes $now, like Owen said
• enable MySQL’s Qcache
• Then, re-analyze your queries, and tackle any outliers

• APC is pretty much a free speed boost. Give it at least 32MB… maybe as much as 64MB or 96MB if you have a lot of pages.
• WP-Cache2 is a similar no-brainer, although do note the caveat that Owen linked
• Up the expiration on WP-Cache2 to at least double the default

I’ve done all this, plus relaced WP’s file-based object caching with in-memory caching using APC, and it all runs wicked fast. How much traffic it can handle depends on many things. If you’re getting a “slashdot” or “digg” type effect, meaning you have a lot of first-time viewers all hitting the same post, WP-Cache will handle millions of hits a day… as they’ll all be getting the same static cached file.

If your traffic is more spread out, say if someone big linked your front page and a majority is going on to click to other pages (this is rare… the vast majority will hit you once and go on to the next item) or if you simply have a large userbase, you might not to be able to handle quite as much, since many hits won’t be cached, especially if people have comment cookies, which means they can’t get a generic cached page, but the Qcache and the $now freezing and APC will do a lot to minimize that load. You may be able to handle this on a VPS, but a cheap dedicated server (expect to pay $90-$160 a month) will be able to handle it easily, especially with the modifications that I mentioned.

Note:

the site has more than 1.2MB of locally-stored images on the homepage… if the site has a daily visitorship >10K, and depending on user cache settings, that’s many gigs of overhead per day. on a low bandwidth, memory limited VPS, apache serving up that much content could be a real factor. I’m not speaking from experience, I’m speaking in general terms having done a ton of research on VPS’ at one point and deciding to stick with shared hosting until I outgrew it.

Maybe mark (or others) can speak to the issues of huge content like that; someone with lighttpd or alternate serving approaches would be helpful. Also be good if Will could provide us with the VPS specs, so we know exactly what the min and burst CPU, RAM, and pipe are, and what processes are running on the box aside from LAMP.

I noticed the homepage is now showing it is cached, with a dynamic-serve time of 0.320 seconds. Given other feedback above, seems like things weren’t getting and staying cached…

-d