Cleaning up after hasn’t been fun, but I am pleased with what I’ve managed to recover from the worst crash.

I wanted to get the latest data from the dead webserver. It was in MySQL, and stored in /var/lib/mysql. Unfortunately, the /var/lib directory no longer existed.

I didn’t want to try to recover it in place — with so many bad blocks, things can only get worse — so I copied the entire partition to a file on my laptop (the one with the shiny new disk): ssh deadserver dd if=/dev/hda1 conv=noerror > hda1.img (You need the conv=noerror or else dd will stop when it hits the first bad block.)

So, then I had most of a corrupt filesystem image. To make it useful I used the loop driver: losetup /dev/loop0 hda1.img

Now I could try reiserfsck to see what I could recover. I started with reiserfsck –rebuild-sb /dev/loop0 to rebuild the superblock: even it if hadn’t been affected by the physical disk corruption, it would certainly be confused by it new home in a looped image that probably wasn’t the same size as the original partition. Next step was reiserfsck –rebuild-tree /dev/loop0 to try to find the contents of the missing directories. I finished it off with reiserfsck –check /dev/loop0 to make sure it was happy.

Now I can just mount /dev/loop0 /mnt and have a look in /mnt/lost+found. The data is there!