We like to talk about backups a lot here at Small Dog. Sometimes it’s advising customers on the best backup solution for them, while other times it’s making sure our own employee data is safe. I know I’ve even written a few articles for newsletters about backing up. There are just so many options and so many things to consider.
With all that time spent talking about how to back up the other half of the equation often gets far less attention and discussion. What happens when a drive crashes, or data is deleted by mistake, or a planet-destroying battle station takes out your Alderaan-based server farm? Recovery. Last week I experienced what going through a major data recovery was like for the first time.
First, it should go without saying that if you don’t have backups at all, or have poor backups, there won’t be much “recovery” going on. In my case, I have 4 machines: My Linux desktop, Mac mini, MacBook Air and my server virtual machine that runs on the Mac mini. Each of them uses CrashPlan to back up data to the cloud. I also used a local 80 GB USB drive to backup a small amount of larger files that were impractical to upload to the internet.
So what happened? Normally the Mac mini, and the virtual machine in it, run 24/7. Occasionally I like to take snapshots of the entire virtual disk file. This is nice to have in case I change some settings, or the virtual disk becomes corrupted. This is not part of my automatic backup because the virtual disk cannot be copied while the virtual machine is running. Last Thursday, I shut down the virtual machine to download one of those snapshots. As I started it, I noticed there were actually two large virtual disk files coming in instead of just one. The second one had a weird name on it, and appeared to be the result of a test I had run months ago. I canceled the download, found the file on the Mac mini, and deleted it permanently since it was so large, and I wanted to reclaim the space (~20 GB).
I finished downloading the other file and then went back to the Mac mini to restart the virtual machine. It booted, but I could tell immediately something was wrong. Some services I had recently installed were not running. Further investigation made my heart skip a beat; apparently what had happened was some time back in March of 2014 the virtual machine had forked and started using the second weirdly named virtual disk file. Yes…the very same one I had just deleted permanently.
At a moment like this, I have to say, the first thing I did was start analyzing potential losses. How much was on that server? Would it be ok if I lost it? There really wasn’t too much on the server, but it was hosting a blog which had some extremely important things I had written in September. It was some of the most important writing I’d ever done. Absolutely irreplaceable.
Fortunately my story has a happy ending. After fighting with the CrashPlan web restore interface, and then switching to their desktop application (which works much better), I was able to find all the backups. I needed to rebuild a new virtual machine server, copy all the data back, and cross my fingers that my MySQL database backups could be successfully imported. The whole process took about 8 hours and I’m happy to report everything is back to normal now, with zero data loss.
What did I learn here? First, while I knew that writing was important, this incident showed me it was even more important than that. Never underestimate how important your data is. The day after I restored the server, I downloaded all of the writing and took it to the print shop. It’s now safely stored in a binder. I also learned that the slower cloud-based backup had pushed me into the habit of excluding as much as I could. To increase the robustness of my backups, I now use the cloud plus two 1TB Seagate Backup Plus Slim Portable drives in a RAID1 configuration. RAID1 means that even if one of the drives fails, I can swap in a second one without data loss. If each drive has a 5% failure rate, the chance of the entire RAID failing is now 0.25%. That helps me sleep a lot better at night!