All WP Sites Downed: Hacked But Doesn't Explain All

  • I have done a tremendous amount of searching here and on Google to figure out in full what caused 20+ installations to melt down.

    Tracking down whether it is truly one root cause or possibly three is important in how I take the next steps.

    Let me say up front that I am totally self-taught and am certain I have made a long list of mistakes but I’ve done the best that I can in 20 months to go from 2002 HTML to putting together a nice WordPress site. I know many of my mistakes already of course through what has occurred recently. Finally, all of the stuff I’ve done up to this point is for volunteer groups…no one is making any money on these sites, including me. (Making money is great! It’s just not the case here.)

    When I say the sites melted down, I mean that if one attempts to pull them up, they either:
    1) Show a “Fatal Error: ‘many varieties of text goes here’ ending with line XXX” / Parsing error
    2) Internal Server
    3) Blank White Screen
    4) Site will come up but going to wp-admin is a dicey situation (results in #1) OR can get into the dashboard but get a variety of either a line code / parse error OR can go one level in and THEN get a line code / parse code error.

    An examination of downloaded sites shows files scattered throughout that have Zero bytes (too many to list as examples). When one opens the files, of course they are entirely blank.

    Obviously, that would explain the line code errors on something citing “missing”, but that is not the exact problem stated in all errors.

    —–

    I am on shared unlimited hosting with a Godaddy reseller (didn’t know this when I first signed up).

    My first website – WordPress – was installed in the root directory. Later, many installs were added, so their folders sit in the root directory with the WP install. But all have their own databases. WordPress is by far the predominant type of installation, but there are two Joomlas, a Drupal, and a few other applications on my account (all are PHP based from what I can tell)

    This is important to know because I discovered some information that indicates Godaddy’s servers may not have adequate protection to prevent interactivity between users / files who have shared hosting on their servers. THIS post and the discussion comments have some information in that regard.

    The above linked post references massive hacking, but, from what I’m reading about a proper server configuration in a shared environment (suexec), it seems there are broader implications.

    Godaddy, I’ve discovered, have had repeated attacks to WordPress (and some other php based) sites, apparently dating back to April / May of this year. The post linked to below relates that there were attacks on September 18 and 21.
    GoDaddy Hacked Again: Another Way To Cure Your Site

    This, of course, leads to:

    CAUSE #1

    EVAL hacking code in most php file headers in the root directory wordpress installation

    I have not been able to look in every file from every WordPress install I downloaded, but I *did* find the EVAL base_64 code in a jsquery file in the wp-uploads folder of another install.

    Problem with this as entire explanation: I cannot find anything in any of my searches that connects the EVAL base_64 malicious hacking to the blanking out of files.

    CAUSE #2

    A plugin destabilized on of the sites and its subsequent manual deletion wreaked havoc on all the installations.

    I had a WP 3.0.1 install with Antisnews Theme that I had spent a lot of time developing over the past couple of months.
    There were a number of plugins I was experimenting with, but was doing so one at a time. I made sure every plugin was rated as 3.0.1 compatible because I was asking a lot of the install by using a number of them.

    The install was entirely stable with these plugins:
    Wordpress.com Stats
    Askimet
    Contact Form 7
    SEO All In One Pack
    Add to Any
    FD Feedburner
    Twitter Tools

    Was still stable having added, one at a time:
    FD Footnotes
    Citation Manager
    Greg’s High Performance SEO (removed All in One)
    WP SlimStat
    List Category Posts

    These were added in a short period, but one at a time:
    W3 Total Cache
    Simple Music
    MP3 Player FX

    I started to see some instability when I reached the last three, specifically in a couple of line code errors above the Dashboard. A refresh seemed to clear the problem and the site was otherwise functional.

    I disabled MP3 Player FX, Citation Manager, and SlimStat to see if that helped.

    The next day – September 21 – I discovered that although I could get into the dashboard, I could not access any of the menus.

    I pulled up some of my other websites to see if it was a possible server issue, starting with the wordpress install in the root directory. It and others seemed fine (it pulled up, I could go to the dashboard, click links there and make my destination)

    I did a bit of Googling quickly on the last few plugins I installed. I noticed the W3 Total Cache had just had a number of error reports and in the few days prior a major upgrade to correct issues.

    Believing the plugin was the source of the problem, I felt I had no choice but to go into my c-panel and delete it. I have done that before with a couple of other plugins and it was fine.

    After I deleted it, my site seemed improved, but not “perfect” – could go into the dashboard and minimally click around.

    I refreshed the page on the root directory install and that’s when I got the first line code error. I started checking other sites and saw the same thing.

    I discovered that every wordpress site I went to, besides the one for which I had deleted the plugin was done. I checked most (but not all) of my installations and all were throwing errors.

    I called my hosting provider who said that my deleting the plugin should NOT cause an account wide melt down, but then again…it could. (???)

    After spending two hours on the phone with them, they had no answers and said the server was fine after they said they ran tests.

    Fast forwarding several days down the road, I discovered the EVAL base_64 embeds in the files as noted above.

    I have spent hours on the phone with my hosting provider who says they have run multiple testings, including for malware. They said some of my databases sit on separate database servers. They have had no outages, server problems or otherwise.

    They don’t what happened, can’t explain how my deleting the plugin could have caused account wide failure, etc., etc.

    In efforts to sort this out, I found a thread here on the WordPress forums regarding questions some people were having about whether or not W3 Total Cache was changing the configuration of the Apache server. (LINK HERE)

    I believe the W3 was simply interacting with other plugins – not that it is poorly coded, etc. Again, I was demanding a lot from my installation.

    CAUSE #3

    There was server damage, corruption, failure.

    Based on the complete lack of assistance from hosting service and an overall degrading level of service from them over the last 9 months or so, it is very difficult to trust their competence, and unfortunately, even their honesty.

    A programming developer took a look at a number of files from several sites and the overall situation and it is his opinion that there is something very wrong with the server.

    It is hard to know the answer to this in some respects, since information from the provider would have to be totally rejected (they ran tests, my databases are on separate servers, etc.)

    CURRENT STATUS / REMEDIES ATTEMPTED:

    On the third call to my provider, they informed I had the ability to restore my accounts by rolling them back in time. The restoration was haphazard as despite using two different browsers, I kept getting a message: “Notice: Action Was Not Able To Complete Or Timed Out”. Examining the files after the process on all my installations, I saw updated file dates, and over the course of a few hours SOME of the sites did come back up, in various states of stability.

    The programmer went through much of the code on one of the sites and essentially laid over fresh WordPress files throughout. After that, we deactivated, deleted, and reinstalled all the plugins and theme. The site seemed “back to normal”.

    The following day, that site was down again. The programmer examined the files and noted many had been changed all at a time when no one had access to the site or back end.

    All sites are down again, even those which had some form of functionality following a restoration.

    It is true that some of these installations were out of date – the majority are either not currently having traffic driven to them or are for testing purposes. Of course I am well aware of the hazards of having them sitting online and out of date. Practices will change in the future.

    QUESTIONS:

    Has anyone any information regarding whether the EVAL Base64 hacking causes the wiping out of a number of file contents in WordPress?

    If it does not, there are additional issues.

    I have read the removing Malware and “Has WordPress Been Hacked” links here and elsewhere. I am already resigned to the fact that the databases and wordpress installs are “shot” due to the malware.

    The core question is whether or not our problem is “limited” to the malware hack – and the key to that is the consideration that the files were wiped out.

    We’re going to be moving to a new provider, but if the blanking out of the code is a server fault OR if they have poor configurations on the servers that allows files to interact (i.e. the W3 plugin), before I notify them of the move, I am going to attempt to get some kind of recompense from them.

    Finally, it will assist us in the moving process to know what we should be looking for in the exports from the site contents in case the most recent backups of those have any possible corruption or damage, especially if it goes beyond the EVAL hacking problem.

    I theorize that the servers are poorly configured, which allowed not only the massive hacking across GoDaddy, but a very sloppy interactivity of user files on the server. The plugin is not configured to do damage, but it seems that it may likely did, only because of the poor configuration.

    I have tried to provide absolutely everything I thought relevant, but if anyone has questions, I will be checking this thread. Thank you for your attention.

    Giving site links is not incredibly helpful at this stage, since none will pull up, but for purposes of examples of line code errors, here are two: (It is possible, due to the incredibly random behavior we’ve witnessed that either of these MAY come up and go back down. If the sites are up, clicking on things will likely provoke an error eventually…unless a miracle occurs.)

    https://grassrootsne.com – where the W3 cache plugin was installed.
    https://clibertyc.com – the site that had been edited for errors, fresh wp installed, plugins/theme deactivated, deleted, and reinstalled and was functional until “hit” again.

Viewing 11 replies - 1 through 11 (of 11 total)

  • what a nightmare.

    I got 20+ sites ..similiar to your situation and also self-thought.

    If i were you, I’d start from scratch
    -get a nice VPS or dedi at hostgator or mediatemplate.
    -Install one copy of WordPress, enable network, install the domain mapping plugin.
    -go over every line of themes’ codes.
    -install fresh copies of the plugins
    -import your cleaned databases manually via phpmyadmin.

    If you want to go this route, I’d help.

    Moderator James Huff

    (@macmanx)

    Have you asked your hosting provider if they have any recent backups that they can restore your account from?

    Or, do you have any resent backups that you can restore from?

    Thread Starter Stubborn_Facts

    (@stubborn_facts)

    @klark0 How very kind of you to offer. I can see that you appreciate the frustrating situation here having a similar setup yourself. I do see a lot of people recommending Host Gator. It does seem for good reason; the whole server configuration seems to be at the core of it. Which one do you use?

    I don’t know as I “need” virtual or dedicated. While I had a lot of installs, none get anything like the traffic necessary to trigger that need. That may become necessary in the future, but we weren’t there yet. The only reason to go virtual or dedicated would be that shared hosting is hazardous to the health of multiple sites at once. From what I’ve read, if that is possible, it’s poor server configuration. But I could very well be incorrect.

    I believe you’re correct regarding fresh copies. The reading I have done regarding EVAL base64 hacking recommends not trying to import the databases from sites hit, but to focus in on the Exported content file as this is by far the most difficult / impossible aspect of one’s site to replace.
    Fortunately, on none of the sites was there such significant CSS editing on the themes that it would be prohibitive to just upload fresh copies. Apparently themes are one of the “favorite” targets of the hackers.

    Ultimately, if I can find any kind of scanning software to go through the export files and ensure those are clean, that would cut the work down a great deal. I’ve already done a preliminary text string search on one of the exported files and cannot find any of the very well known strings for the EVAL base64 code.

    I’d be interested in your answer re: hosting. Thanks again for the interest.

    @james There are backups available from the hosting provider – I’ve tried three times to do restores, backing up a little each time. As noted above, it is murky as to whether the restoration was actually truly successful on each occasion because I always get an error message that the process could not complete. The reason it is murky, is because after the first attempt, some of the sites were back up and the folders inside c-panel showed changes consistent with the restores.
    From my reading, restore won’t solve the problem with the hacking because either 1) the database tables are corrupted or 2) the server is getting hit with another round of attacks.
    Restoring from my backups apparently won’t work either according to what I’ve read. All recommendations point to scrapping the wordpress and database files totally.

    The core issue again here is: Is the EVAL hack known to actually blank out whole file contents? I can’t find anything on that anywhere.

    Stubborn,

    before you slip into suicide mode maybe we can offer some ‘comfort’. It is a very long story but due to the fact we are in aviation we don’t have the pleasure (and luxury) to tell it all. So here in a nutshell: Recently we’ve installed W3-total cache and it messed up bad. And when we say bad, we actually mean catastrophic – We got the 505 internal error, the 404, you name it we got a screen-shot of it. Our bounce-back rate increased to 65%!! Our Apache server was bogging and sputtering. In 76 hours since installment W3 almost crashed our database. We have not contacted the ‘author’ of this product since all and everything is grass-root-development and we are not into blaming games.

    From this point on we are not going to install ANY cache program until it is proven in a test environment. We are not developers and/or code-monkeys (but we learned our lesson).

    Here was our strategy to get out of this flat spin (after a nerve wrecking investigation): we de-activated W3, we deleted (yes, DELETED) .htaccess in your public_html folder to cure the 505 Internal Server Error, gave your server a couple of minutes to recover, logged-in as admin, went into our permalinks and saved our options again (to cure the 404 Not Found Error). Since 48 hours our server runs nice (not perfect) – all links and functions work. Eventually we had to go through all our settings and re-save them. 99% of our options show up the way we had them set up, but we needed to SAVE them AGAIN to re-write .htaccess and all plug-in specific config files.

    We don’t know about any hacking and/or manipulation regarding our database. We definitely know that our trouble was caused by W3 describing behavior you mentioned.

    Hope that helps.

    I wish someone would have contacted me because there has never been a case like this and I would like to know specifically what transpired and how the plugin was used to cause this outcome.

    Thread Starter Stubborn_Facts

    (@stubborn_facts)

    @p3air I laughed outloud when I read your reply. I HOPE I wasn’t coming across as nearly suicidal! It’s frustrating, to be sure, but in the big picture, life will go on.

    I really appreciate your post here and the information about what you’ve done to recover. I’m going to review what we’ve done and see how it lines up. As I noted in my posts, it may well be that there were a couple of things that happened on top of each other here. For us, it comes down to the configuration of the server(s). There’s just no getting around the fact that we had an account wide problem regardless of the hacking or effects of the plugin.

    But that doesn’t sound like your issue – if I am following what you are saying this is a server you are running yourself. If I detected that correctly, you simply don’t have that same issue, which is great.

    We know we had hacking – I’ve now seen the code myself.

    Again, I appreciate the information and hope that your whole situation stabilizes.

    —-
    @frederick Townes – The reason I didn’t contact you is because I had already resolved the issue regarding the W3 plugin by doing a Google search on how to properly thoroughly delete it. One of the restores I mentioned took the site temporarily to a point that the W3 was back in the WP Plugins list but not activated.

    I know you said that there has never been a case like this, but in THIS thread a number of people are specifically discussing server configuration issues.

    Do note that I said the following:

    The plugin is not configured to do damage, but it seems that it may likely did, only because of the poor configuration.

    I am not blaming W3 – the server should be configured in such a way that the obvious interactivity of files on the one on which my sites site is happening should never occur.

    I can see you work hard to communicate with people who are having issues and are genuinely interested in learning about any trouble people are having. This is a free plug-in…I greatly appreciate the amazing hard work of the many coders who do these projects.

    I firmly believe – as stated above – I was really taxing my setup with all of the plugins I was trying to use and there was simply some issue with bad interactivity. Despite the fact that there was interactivity…my site DID speed up while it was running your plugin.

    Server configuration is huge – as I’ve learned the hard way. It does seem like there is some evidence pointing to the idea that depending upon the server configuration, W3 may be having an impact outside of the expected locations of install.

    We may simply never know precisely all that occurred here. The programmer who went thoroughly through another one of our sites and got it back up, said the following after hearing that it went back down again:

    I had replaced all of the WP files so unless there was a backdoor in a
    plugin, it was no different than starting from scratch. Without
    detailed server logs I have no idea why seemingly random files are going
    to 0 bytes and getting wrong permissions set on them.

    This again, points to server configuration issues. But in reading the post I linked to here, I do see that people were mentioning permissions issues.

    In any case, my real purpose in posting this was to get to the bottom of whether the EVAL hacking is known to cause files to zero out.

    It didn’t even occur to me to contact you and you can imagine how insane it has been with this many downed sites.

    I had replaced all of the WP files so unless there was a backdoor in a plugin, it was no different than starting from scratch.

    Just a quick note… that statement is technically not correct. Unless you delete everything first there is always the chance that a back door was left behind somewhere else in the installation. Also plugin files themselves are often targets for infection.

    A good portion of what I do these days is clean WordPress installations for people (I wrote the article about GoDaddy you linked to in the original post above, and I wrote a guide on how to completely clean WordPress) and to be honest I have never seen one where there were files getting emptied out.

    I have seen files that were overwritten with malicious code, however, and I know of at least one host (MediaTemple) that is performing periodic cleanings on client websites without even telling them that they are infected in the first place. If GoDaddy has started emulating this practice as well (a practice which I personally feel is unethical), then it is possible that the files are getting infected, and then automated sweeps are cleaning out the infected code leaving blank files behind.

    My suggestion is to restore from GoDaddy’s backups, but restore to a directory other than where the sites actually are, and as soon as the restore has been completed, download the whole thing and move it to a new location. You don’t need the WordPress or the plugin files/directories at all. The only thing you need are what is in the uploads directory and the theme, and non-Wordpress directories that might be at the root (such as independent /images directory, or uploaded movies, whatever), the sitemap (if there is one), favicon, etc. Everything else can be replaced with fresh files.

    If you need a hand scanning the non-replaceable files and folders for infection then hit me up by email, I can probably help.

    -Michael

    Thread Starter Stubborn_Facts

    (@stubborn_facts)

    @michael
    Thank you very much for your reply.

    I just re-read your article – I had read so many in researching this whole issue I wanted to make sure I had the correct one in mind. It’s frustrating to review. I cannot tell you how insanely illogical the conversations with my hosting provider have been. Obviously they have the same mind set of the folks you dealt with, only they know even less. I have learned a number of lessons from this, not the least of which is when you notice a degrading level of service it’s time to bail out. I saw the occasional red flag for the past few months.

    I’ve been trying to decide whether it is even worth bothering to take the time in calling them back and very bluntly saying “the jig is up”.

    If these folks had any real principles, they would dig up known working copies of my files and reload somewhere – anywhere.

    You have provided the only plausible theory regarding the blanking out of those files I’ve heard so far. It unfortunately fits the SOP we’ve both observed. Your suggestion about restoration has inspired me to try a couple of things that may not only help me get at some of the files that we’ve lost, it may tell me more about what is going on here regarding the server.

    I agree with you regarding the effort to repair undertaken by the programmer. He’s a great programmer but he’s not incredibly familiar with WordPress. He didn’t take into account buried files in plugins or even the database when he did the cleanup. Simply installing fresh wordpress core files will not do the job. I wasn’t clear on that when he proposed attempting to do so.

    The very frustrating thing here is that XML imports in WordPress 3.0 don’t work very well. I did a search for some of the coding from the EVAL injection and visually scanned and can’t see anything “obvious” in the xml file, yet it simply won’t import all of my content. I did some searching to see if I am alone in this problem and I find a lot of information that leans towards widespread difficulty.

    I would have been satisfied with the ability to do that since I have full copies of the items you mentioned such as images, etc.

    In any case, I’m going to “experiment” with the restoration thing a bit in a couple of respects.

    I am also going to keep in mind that you’ve been specializing in this area lately. There are a couple of sites that are going to be real nightmares if we can’t get to a place where we can restore the posts via some automated means.

    Thanks again.

    The very frustrating thing here is that XML imports in WordPress 3.0 don’t work very well. I did a search for some of the coding from the EVAL injection and visually scanned and can’t see anything “obvious” in the xml file, yet it simply won’t import all of my content.

    Have you tried importing the data into some installation other than the corrupt one? Meaning, an entirely different account altogether, fresh install, fresh database?

    It’s hard to say for sure without seeing the details of the file itself and what it is exactly that isn’t importing. Is it posts that are not coming in correctly? And did you verify that the post content was in fact intact inside the xml export?

    My email is michael at endlesspoetry dot com, feel free to drop me a line if you want to get into details. That way we can post here if we come up with a solution and hopefully help someone else without them having to read the whole conversation.

    Thread Starter Stubborn_Facts

    (@stubborn_facts)

    Michael,
    I’ll be sending you a message – mainly to clarify so this thread doesn’t become even longer. I’m hoping to provide some information a “solution” but it’s probably prudent to have some “supervision” since I am clearly learning as I go.

    Does anyone here need any input from me? The W3TC tag was added here, so I’m trying to be sensitive to any issues surrounding this topic.

Viewing 11 replies - 1 through 11 (of 11 total)
  • The topic ‘All WP Sites Downed: Hacked But Doesn't Explain All’ is closed to new replies.