This whole debacle could not have come at a worse time as nearly the entire team was off doing family things or otherwise unavailable. As some of you already know later in the day a couple of our sites were yet again defaced. Again, no one was really around or available so things lingered in a state that they should not have. Thirteen hours after Rob and I started driving we arrived at Rob's parents house. So at 1am, we log in to check on things and find out that things have been defaced again and the “fires are raging”.
After noticing that things were still changing on the sites we took the server offline and immediately started scanning the logs and filesystem. The intruders managed to compromise every site on one of our servers. The server compromised houses the main site, the developer site, the help site and last but not least the shop. When scanning through the filesystem we found cracker shell files in the shop site filesystem. These files are most often placed using a remote file inclusion. The most common way of achieving this is by attacking the site with a request that modifies the $mosConfig_absolute_path variable.
Since our sites are traditionally locked down pretty hard it was hard for us to imagine how this could have happened. We had an issue a couple of weeks ago with a vulnerable 1.5 demo site that was running an old beta2 copy; but that had been since rectified and we were pretty sure that wasn't the case. After drilling into the access logs for the shop site we found just as expected lots of remote file inclusion requests.
As it turns out, we got caught “with our pants down” today. Of all of our sites, there was one that still had register globals emulation on. Of all of our sites there was one that had the htaccess file missing and most importantly ... that one site has a remote file inclusion vulnerability. The Joomla! Shop site runs a custom component that was written to connect to print mojo who manages our online shop. A simple and as we have all seen common mistake was made and the print mojo component had a vulnerability to remote file inclusion.
We are going to takes things a step at a time and make sure that each and every site is as secure as it can be before we bring them back up. You can all expect the shop site to be offline for the immediate future until we are certain that it is secure and ready to be brought back online.
I would like to thank all of you out there for your patience. This has been a long and hard day for everyone involved and I do have a great appreciation for the worry that you must have had. Just for the record, the vulnerability that caused this embarrassing problem was NOT the Joomla! core, or even any available third party extension. As far as I know the component that runs our shop has never been publicly released (and good thing for that
). The issue was not with our hosting provider either, they have been wonderfully understanding and helpful throughout this process. The entirety of any blame rests squarely on the shoulders of the entire Joomla! core team.Again, thank you all for your patience and understanding.
Discussion thread: http://forum.joomla.org/index.php/topic,203291.0.html
