Haunted by a ghost (index.php mystery)

  • A site that was running five days ago (and possibly even more recently) started choking early this morning (I was notified by email from New Relic), with an error saying that index.php was trying to require inclusion of a file that doesn’t exist. Since the site had not been changed at all in a couple months, I first tried restoring from a month-old files backup, but the error was not resolved. I don’t normally read WP code files, but I decided to take a look at index.php… and I discovered the weirdest thing: It wasn’t a WordPress file at all, but a Joomla bootstrap! This site was running on Joomla in the past, but when it got hacked about a year ago, I wiped the files and database completely (at least I thought I did) and started over in WordPress.

    Since the restore didn’t replace that file, I decided to look inside the backup. The backup was done by UpdraftPlus, which separates the files into four categories: Plugins, Themes, Uploads, and Others. I never thought about it until now, but even the Others backup is only files in wp-content – there is no attempt to backup the files in the webroot, like index.php, wp-config.php, etc., nor the wp-admin or wp-includes directories. I’m not very impressed with UpdraftPlus right now.

    I grabbed a copy of a WP bootstrap from another website I manage, and the site appears to be running now, but I’m concerned that something hinky is going on that I can’t see. As I said, I had not touched the site in a couple months, so my hand didn’t cause this. My hoster did some server maintenance a couple days ago, but it’s an unmanaged VPS, so they wouldn’t have messed with my files. The only way to get on the server by SSH or FTP is with a private key, there is only one WP user, and WP says there is only one active login at the moment (me).

    Does anyone have any thoughts about what might have happened, or what sort of hinky things I should look for? I’ve now upgraded WP to 4.8 and all plugins to their current versions, but if extra files have crept in somehow, upgrades won’t get rid of them. I feel like my site is haunted, since I got a visitation by the ghost of bootstraps past…

Viewing 2 replies - 1 through 2 (of 2 total)

  • Have you checked all of the server logs? There’s normally *something* that can be found in there, especially since you have a pretty good approximation of when this happened.

    There’s also the chance that what ever maintenance was performed did inadvertantly mess with the file system. Again, without all of the logs you won’t know.

    That’s what external backups are always important. Backup plugins are good most times, but they are no substiture for a good verified FTP’d down copy of all of the websites files and a good SQL dump.

    Thread Starter OsakaWebbie

    (@osakawebbie)

    I’m trying to read the logs, but I’m not making much sense of them. The nonexistent file being required by the Joomla bootstrap was defines.php, but there is no mention of it in Apache’s access_log or error_log. I also looked through them manually for anything that looks strange, but it’s really hard to know what to look for – I get very few real visitors, but all those bots out there keep it busy.

    I’m sure it would have been in a PHP log, but it looks like PHP wasn’t logging – in phpinfo(), here are all the variables that looked potentially relevant:

    fastcgi.logging = 1
define_syslog_variables = Off
log_errors = Off
error_log = "no value"

    Except for fastcgi.logging, those values are not very encouraging. [Please don’t shoot me – I am a lone developer forced by necessity to manage my server; I’m not very good at server admin.] I have now turned on logging to syslog, but that’s little help for finding out what happened earlier this week. Does fastcgi.logging=1 mean there is a record somewhere? If so, where should I look? And if you have any suggestions about what to look for in the Apache logs, let me know.

    I learned a little more in New Relic. It turns out that the problem did not start this morning – that’s just when New Relic decided to flag it and send me an email. For some reason I can’t coax the New Relic error chart to go back any farther than 12:01 a.m. on 7/26, but the error was already happening then. On 7/22 at about 8 p.m. it was working, because I showed it to someone. I have created a support ticket that asks my hoster if they did anything during maintenance. It seems unlikely, but I hope it was them, because any other prospect is more worrisome about my site’s security.

    As for backups, I’ve now learned my lesson. But that problem will solve itself soon, as I’m currently working on moving all the stuff on this old host to a Linode server with brand-new software, on which I am paying for regular backups not just of the web stuff but the entire server. (The website is easy to move, but I also have a couple hand-crafted web apps that are more complicated.)

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Haunted by a ghost (index.php mystery)’ is closed to new replies.