Virtual Machines – taking the pain out of major upgrades

If your computers are physical machines, where each piece of hardware runs a single OS image, then upgrading that OS image puts your services at risk or makes them unavailable for a period of time.

Sure, you have a development and test environment, where you can prove the process, but those machines cost money.  So processes develop to either ensure you have a good backout, or you can make changes you know will work.

Virtual Machines have changed the game.  I have a couple of Linux (Debian) based VM’s.  They’re piddly little things that run some websites and a news server.  They’re basically vanity VM’s, I don’t need them.  I could get away with shared hosting, but I like having servers I can play with.  It keeps my UNIX skills sharp, and let’s me learn new skills.

Debian have just released v6 (Squeeze).  Debian’s release schedule is slow, but very controlled and it leads to hopefully, very stable servers.  Rather than constantly update packages like you might find with other Linux distributions, Debian restricts updates to security patches only, and then every few years a new major release is made.

This is excellent, but it does introduce a lot of change in one go when you move from one release of Debian to the next.  A lot of new features arrive, configuration files change in significant ways and you have to be careful with the upgrade process as a result.

For matrix (the VM that runs my news server), I took the plunge and ran through the upgrade.  It mostly worked fine, although services were out for a couple of hours.  I had to recompile some additional stuff not included in Debian, and had to learn a little bit about new options and features in some applications.  Because the service is down, you’re doing that kind of thing in a reasonably pressured environment.  But in the end, the upgrade was a success.

However, the tidy neat freak inside me knows that spread over that server are config files missing default options, or old copies of config files lying round I need to clean up; legacy stuff that is supported but depreciated sitting around just waiting to bite me in obscure ways later on.

So I decided to take a different approach with yoda (the server that runs most of the websites).  I don’t need any additional hardware to run another server, it’s a VM.  Gandi can provision one in about 8 minutes.  So, I ordered a new clean Debian 6 VM.  I set about installing the packages I needed, and making the config changes to support my web sites.

All told, that took about 4 hours.  That’s still less time than the effort required to do an upgrade.

I structure the data on the web server in such a way that it’s easy to migrate (after lessons learned moving from Gradwell to 1and1 and then finally to Gandi), so I can migrate an entire website from one server to another in about 5 minutes, plus the time it takes for the DNS changes to propagate.

Now I have a nice clean server, running a fresh copy of Debian Squeeze without any of the confusion or trouble that can come from upgrades.  I can migrate services across at my leisure, in a controlled way, and learn anything I need to about new features as I go (for example, I’ve switched away from Apache’s worker MPM and back to the prefork MPM).

Once the migration is done, I can shut down the old VM.  I only pay Gandi for the hours or days that I have the extra VM running.  There’s no risk to the services, if they fail on the new server I can just revert to providing them from the old.

Virtual Machines mean I don’t have to do upgrades in place, but equally I don’t have to have a lot of hardware assets knocking around just to support infrequent upgrades like this.

There are issues of course, one of the reasons I didn’t do this with matrix is that it has a lot of data present, and no trivial way to migrate it.  Additionally, other servers using matrix are likely to have cached IP details beyond the content of DNS, which makes it less easy to move to a new image.  But overall, I think the flexibility of VM’s certainly brings another aspect to major upgrades.

Hosting Changes

Hopefully no one’s noticed that I’ve moved my websites (including this one) to a new hosting provider (well, new for this site, but not new to me in general).  When I first decided to move away from shared hosting at Gradwell, I wanted to find a VPS solution to give me more control over the sites.  At the time, I was hosting an EverQuest guild website which got a reasonable amount of traffic, and so I chose a VPS solution that would definitely have enough oomph and bandwidth to deliver that.  Not a huge amount, but enough.  It was my first time in the VPS market and I wasn’t really sure who were best, what was good value and what the different offerings resulted in.

The 1and1 service I went with was pretty good, but not what I would call cheap.  Over the 18 months or so I’ve had the service it’s been pretty reliable, a few unexplained outages, and a couple of periods of downtime that were longer than I would have liked.  But the VPS was powerful, had plenty of memory and lots of network capacity and easily delivered the 7 or so domains I hosted on it.

Not long after I moved the sites though, the EverQuest guild site dropped off dramatically in terms of load (lots of people left the game), and after 18 months it’s become apparent that the 5 or 6 vanity domains I host really don’t justify the cost and performance of the 1and1 VPS.

I looked again at shared hosting, because I’m a pretty good example of who should use it.  I tried tsohosts and while they’re excellent value and I have nothing bad to say about them, I really don’t get on with cPanel and the shared hosting mentality, especially after running my own VPS for so long.  I wanted to get into the config files and set things up ‘just so’, and after only 2 or 3 days fighting with cPanel I gave up and bought another VPS from Gandi.

I love the Gandi system (I already have another VPS from them hosting a usenet server), and although a single share VPS is pretty low resource you can deliver quite a lot from a Linux machine with not much in the way of power these days, especially when what you’re delivering are vanity domains with almost no traffic.

So I’ve got a single share (256MB memory) VPS from Gandi and over the last couple of days I’ve moved everything over, and I’m pleased at how easy it was.  The 1and1 VPS was running CentOS but I’ve gone for Debian with Gandi (I prefer Debian) so not everything could be simply copied over, but the content (static and mysql data) was easy enough to transfer, so now it’s just a case of making sure the VPS is secure and managed properly.

If you’re looking for some domains or a VPS, I really can’t recommend Gandi enough, my only gripe is there’s no easy way to pay monthly (no direct debit facility, so you have to top up a pre-pay account when you remember), but it’s a small issue when the service is so excellent.