Power Surges

It’s been something of a disaster here in the last 10 days. Things were going swimmingly until last Friday when the power flickered on and off. Normal the quality of power supply here is excellent and this is the first time the power had gone off in a while. All of my hardware (or so I thought) is plugged into surge protected power boards so I wasn’t too worried. My main PC came back up no problems as did the laptop, modem, network switch and everything else. But I couldn’t get a connection to the internet from any computer.

Dead Router

Several ADSL router reboots later and with the help of an old laptop I’d isolated the problem to being in my ADSL Modem/Router. It just failed to connect to the internet. OK, dead modem, At that point I was a bit baffled why it would be dead because the modem is plugged into a high quality surge stopping power board. Off to the computer shop I go and 30 minutes later come back with a Billion 7800 NXL router. I picked it because of the dual band WiFi and the capability of running an external USB device. Got that home, connected it to the ‘net without issue and then plugged the network cable from my network switch to the modem. And bang, down went the modem. Uh-oh. Well it turns out the network switch wasn’t plugged into a surge protector. So it was dead and doing weird things to anything connected to it. So, I pulled the switch out of the network and connected the various PC’s directly to the modem and everything came back online. Yay. Or not.

Dead Mac Mini

At this point I’d plugged in the Mac Mini Server with dual RAID disks which acts as a simple network file server and also runs a series of CRON jobs to perform rsync driven backups from various servers around the internet that I look after. The Mac had appeared to boot up OK. I say appeared because I don’t actually run this machine with a monitor. In fact the last time I had a monitor plugged into it was two years ago. The only way I can tell it’s alive is by a single LED on the front and whether or not I could access the shared directories we use every day. It turns out I couldn’t access anything from it. I couldn’t open a PuTTy session with it either. Anyway, to cut a long story short I had to find a spare monitor and keyboard and mouse to plug into it and see if I could log into the OS via the GUI. Turns out the Mac wouldn’t boot up reliably, would occasionally let me log in and then freeze, or would just freeze and the opening login screen. Bugger that’s dead too.

Emergency Remote Backups

At this stage I was getting nervous because there were no backups being done of anything anymore. I quickly installed an Ubuntu virtual machine in Virtual Box and got some jobs going to rsync my files down from various remote servers in the USA. There was a bit of a hassle getting the Ubuntu client to connect to shared drives on the Windows 7 Host but found this web page to be very helpful. Basically I created a directory on the Ubuntu client using:

sudo mkdir /media/share-name

And then mounted the VirtualBox shared folder using:

sudo mount -f vboxsf shared-folder-name /media/shared-name

I wanted the mount to be persistent and I couldn’t make heads or tails of solutions I’d found that suggested adding entries to /etc/fstab. So I ended up creating a shell script with this in it:

sudo mount -f vboxsf shared-folder-name /media/shared-name
return 0

Then I made the script executable using:

sudo chmod +x script.sh

Then added an entry to my /etc/rs.local file with something like this:

/path/to/script.sh

Once that was done my lovely VirtualBox shared folders were available to my various cron jobs when I started up my Ubuntu VM. Once the CRON jobs were run at least I’d have up-to-date backups on a disk in my office.

A New PC

If there’s been one bit of luck in this whole episode it’s that I happened to have the makings of a new PC sitting in boxes in my office when the power surge happened. I’d been planning on upgrading my main PC and had bought the components in preparation of a leisurely build and test cycle before I transitioned to the new machine. Clearly there was a rush on and I needed the PC to act as a primary shared file server and to be running back scripts rather than my main development PC. So one very late Friday night later I had the machine built and Windows 7×64 Pro installed on it.

I don’t really want to talk to much about the specs of the PC because, well, it bores me. But if you want to know it’s an i5-3570 (my first Intel machine) on a ASUS motherboard with 16GB of ram, a Gigabyte 7850 video card, a 256GB SSD boot drive and a WD Green 2TB hard disk. I was primarily interested in keeping the machine quiet so there’s a high end modular PSU in the machine, a Noctua HSF combination with a low speed 150mm fan and it’s all sitting in a huge well ventilated Corsair Carbide 500R case. The machine seems to work well, is very quiet, and operates much cooler than the AMD based machine I’m using right now. I’ve installed VirtualBox on the new PC too and have Ubuntu running in it performing all of the required backup tasks that I had running on my main PC for just a day or two. The new PC is also acting as a central file repository on my network and is keeping synced copies of all critical files on several external USB drives. This setup will have to tide me over until the Mac returns.

Lessons Learned

This whole disaster could have been avoided if I’d just taken care with how I’d had the network switch had been plugged into the power. I suspect it wiped out the Mac Mini and the old ADSL modem via network cables. I’m just lucky it didn’t take out my main PC and laptop as well. We had minimal data loss because my last backup was just a few hours old. All backups here are automatic and the effort I put into the system a while back paid off in spades. Most of my time has been lost building the new PC, getting a grip on Ubuntu and how it works with VirtualBox, and running backwards and forwards to computer shops and repairers. I’ve also realised I needed a real UPS to allow me to gracefully shut down my computers when the power goes off next time.