Thank you S & I Team

[brian]

Thank you to everyone on the S&I team for getting the forum back online so quickly after the hard disk crash
http://dev.joomla.org/component/option, ... ,33/p,296/

Looks like the backup procedures you have in place have worked admirably

From what I can tell all posts from 05:30GMT till the disk crash (around 17:00GMT) have been lost but I can see people already reposting their answers etc

Well done Team

[brad]

Thanks.. we're still working on things, but wanted to put the forum online asap.

Extensions site will be back soon, and then I'll get the forum search going again. In the meantime, try using: http://www.google.com/coop/cse?cx=00762 ... igrppcfurc for forum searches.

[brian]

When you recover and have had some sleep after working through the night could you blog about the backup procedures you have in place and how you do the backups. I'm sure we could all learn from it.

[Rochen]

Thank to Tonie for making that blog post to help keep people in the loop as to the situation. We have been working for the past few hours to get this restored. Swapping the drive didn't take long as we keep cold spares of every component on-site - what took the time was reloading the Operating System and then restoring the huge volume of data - the Joomla Forum database is by no means small. We are nearly there though.

I am sure Brad can get something written up for the blog. In simplistic terms though we capture backups to a secondary drive in each server every night and then we push those backups to an off-site NAS location using rsync over a secure tunnel for additional safety.

We are also going to be rolling out additional things like NAS imaging backups soon for further protection. Taking image backups will remove the need to reload the OS in the event of a drive failure and means we can push any previous image out to a server over a secure 1GBit/sec VPN tunnel in a short space of time. Very cool stuff... but more on that later.

Heading back to the restore work now. There will be one further server reboot before everything is done as we need to roll back a few core configuration files etc.

- Chris