Update: For those of you who are looking for a drop-in replacement for Amazon, you should consider Server Central. I’ve had the pleasure of doing a very detailed performance study of various cloud providers (many of the enterprise providers) recently and the performance was nothing short of amazing. A special mention about their customer support. Honestly, I don’t think I’ve ever experienced any better from an Enterprise Provider. If you’re looking for an alternative to Amazon and really have no desire to “Roll Your Own” (why would you in the first place), you should give them a look. Tell them Percy sent you.. seriously.
I have this mantra, it goes like this:
“The Best Linux Admins are the Laziest.. Why? We want to do as little as possible so we automate everything.”
It pretty much describes me to a tee. I’m the guy that actually figured out that by typing 30% fewer keystrokes in a day, every day, I would actually gain a few extra days of efficiency a year. So I’m the guy that will spend a few hours automating everything or creating a bunch of shortcuts so I don’t have to type as many keystrokes.
This applies to my current endeavor. Migrating a live Mail Server. It’s not your average mailserver, mind you.. It’s customized to the gills! I’ve got special processes that will do everything from mitigate spammers to auto-train SPAM / HAM, special rules, etc.
For me to set up a new mailserver (the entire thing is backed by MySQL), I would have to effectively do a complete rebuild. I never created a chef recipe, because it’s a one-off for me. Why bother right? just back up the scripts, etc.
Well, for the last few years, I’ve been living in Amazon as a host for my email. It’s served me well, but it also doesn’t serve me well as my bills slowly crept up to $238.00 per month for an email server. Personally, it’s ridiculous. Now, as many of you know, I go back and forth between Amazon and Linode depending on the need, etc.
It’s all about price point, features, etc. I do a lot of research and calculation on everything before I make a choice on where to put any one particular server. I don’t like surprises. Over the years, I’ve watched the monthly fees on my mailserver creep up, but like I said, “I’m lazy.” So why move something that isn’t broken right?
Well.. I finally decided to do something about it today.
I bit the bullet and spun up a Linode, logged into the ssh prompt on the shiny new server and thought to myself, “Ugh.. I don’t even feel like configuring the hostname.” Much less spend the next 6 hours migrating mail, settings, doing a cutover, etc.. Too much trouble.
So I just decided to do a “quick cut-over” as I’ll call it.
I know the mailserver runs fine (It’s mine, it’s configured PERFECTLY).
It’s backed by Mysql and has Spamassassin, Courier, Squirrelmail, DCC, Pyzor, Fail2ban, SSL, etc. All with hand-rolled RPMs from scratch. So you can definitely understand why I have absolutely no desire whatsoever to build a new one.. especially from scratch.
The nice thing about doing it intelligently and my way is the permissions and all account information are preserved.
Here was my solution. Yes, it applies to pretty much any other server, although, with MySQL databases that aren’t static, you’ll just set up a Master-Master replica pair and just break replication once you’re ready to cut-over a live site.
1. I need to make sure both servers are running the same version of the OS. I’m running CentOS 6.4 so it was easy to just spin up a new Linode with CentOS 6.x and just upgrade it.
2. Rather than Tar, SCP, Transfer, I just decided to do a bunch of rsync(s).
Here’s the procedure:
Spin up the new server, log into the new server and copy your /etc/fstab file to your home directory. (You’ll need this).
Now, on the old server, rsync everything except /proc, /dev/, /srv, /sys, /boot
After you do this, copy the fstab file in your home directory back to /etc overwriting the one that was rsynced. Also check to make sure your /etc/passwd files match.
Reboot the server and log in using a user / password combination from the old server… Remember we rsync’d /etc? You’re now operating on the old server’s user / passwd files.
Re-establish your ssh keys on the new server, because the keys will now be identical to the one(s) on the old server.
Cut over DNS
Wait for DNS to take over, check that the transactions are indeed going to the new server. Once it is and there is no more traffic going to the old server, issue a couple of rsyncs again to resync your data directories (for me it was my maildirs in this case).
Restart the services for good measure and you’re done.
Total time to migrate a 50 GB ever-changing mailserver? uhm.. about 2 hours. (most of it waiting for the rsync to complete).
Total Data Loss and Downtime for a Service outage? Zero.
This technique can be applied to pretty much anything. As long as you’re smart about it.
Here are a few Gotchas:
DOUBLE CHECK YOUR /etc directory on the “new” server. Make sure you haven’t hard-coded ANYTHING by IP unless it’s loopback. (This is why you use hostnames people).
Make sure your network interface scripts are updated to reflect the new ip.
Make sure your fstab is the one from the new server. If you don’t, you’re going to hose the server and you’ll have to start over.
DO NOT UNDER ANY CIRCUMSTANCES sync the PROC or DEV file systems!
Well, that’s it… Total actual work time? About 3-4 minutes of actual typing.. the rest of the time was spent babysitting a bunch of rsync processes.
How’s that for efficiency and zero downtime? And they said migrating out of Amazon was hard.. meh.. not if you think about it.
OK PEOPLE: I KNOW THIS TIP ACTUALLY SAVED A BUNCH OF YOU A LOT OF WORK AND HEADACHE. BUY MY DAUGHTER A BAG OF LOLLIPOPS! (Link on Right)