Do you use let's encrypt?





8944 votes ~ 30 comments

 

Migrating from one machine to another

Posted by Steve on Sat 25 Jun 2005 at 19:30

When it comes to migrating a server machine, which handles webserving for several domains along with email, DNS, and several other services there is a lot to remember. Since I've just migrated the machine handling this domain I'm going to describe how I did it.

Whilst there's nothing intrinsically difficult about migrating a group of services to a new IP address, and location, there are often a lot of very small steps to take which need concentration.

Writing a plan is a reasonable thing to do - especially if you're planning on upgrading services as part of the migration.

My general plan for the migration was:

  1. Copy all data across to the new host, web trees, database contents, etc.
  2. Update all DNS entries such that www.* point correctly to the new host.
  3. Wait until this propogated, making sure that things were working correctly, and there were no missing Perl modules, etc.
  4. Migrate all the MX records to point to the new host - so that mail deliveries go to the new machine, instead of the old.
  5. Migrate user accounts. (Passwords + groups, rather than the data).
  6. Disable all services on the old host, and take a full backup of the system state.

If you're in a hurry you could migrate both the www, and MX records to the new host all in one step. This would save waiting for two sets of DNS changes to propogate. But the attraction of doing it in two stages is that you get to catch any potential problems without risking mail, which is often more important than anything else.

It's also means that any existing users don't need to be bothered by email changes until the migration is half-done.

In my case the services being migrated were:

None of these services are terribly difficult to move individually, or even en masse, but there are some potential issues which can cause problems. Almost all the problems are caused by two things:

  • Waiting for registrars to update nameserver entries.
  • Delays in DNS propogation.
  • Hardcoded IP addresses in suprising locations.

The single most important thing in planning a migration is to account for DNS propogation delays.

DNS really is the backbone of the internet without working DNS a lot of things become more difficult to use. If you're not familiar with DNS it's very straightforward in purpose: When you wish to connect to a webserver, or mailserver, you tend to do so by name, eg "www.debian.org". For your computer to connect it needs to know the IP address though, and DNS is nothing more than a mapping of hostnames to IP addresses, and vice-versa.

Because the DNS system is designed to scale and provide reduncency entries aren't updated and available immediately globally. There are caches along the way. (Simplifying things here ....)

This means making updates potentially difficult, when you make a change some clients will resolve your hostname to the old address and some will get the new address. Generally it takes a couple of days for the new IP address to be resolved by all hosts.

For static hosts this doesn't matter too much. You can simply have the two machines, old and new, serving the website.

However when you're running a database-backed dynamic server you can soon run into problems, as updates will be made to two distinct databases.

There are a couple of solutions to this problem:

  • If you're worried about data inconsistancy you can switch the old server to connecting to a database on the new host, so there's two front-ends, but only one back-end database.
  • If you're worried about there being multiple disparate servers around you can redirect all connections from the old host to the new one via a firewall rule, proxy server, or other means.
  • If you have access to the DNS server(s) which are responsible for the domain name in question you can plan for the migration by dropping the TTL field before updating the address.

Migrating the mail handling is similarly problematic, it's easy to end up in a sitation where different hosts are trying to deliver mail to your new and old machine.

Similarly when people connect to your server to receive their mail they may connect to the old one, and not see freshly delivered mail which was addressed to the new host.

My solution to this problem was to use netcat to redirect traffic from the old host to the new one.

The simple introduction to netcat describes how this is done. It's a fairly simple process:

  • On the old host disable apache, your mailserver, etc.
  • Setup inetd to forward connections.

As a sample the following snippet from my /etc/xinetd.d/smtp file shows how I forwarded incoming SMTP connections to the new host:

service smtp
{
        disable = no
        socket_type     = stream
        protocol        = tcp
        user            = nobody
        wait            = no
        server          = /bin/nc
        server_args     = -w 2 xx.xx.xx.xx 25
}                                               

This causes all incoming connections on port 25 to be seemlessly redirected to the new host - making sure I didn't have to worry about the delay in DNS propogation.

Once you're sure that everything has been migrated properly you can relax for a day or two to make sure things are stable and you've not forgotten anything.

(Note: Don't forget to check that you've migrated cronjobs for all users too - I forgot this for a couple of days when moving things to this server.)

One important thing that you must check is that you don't have configuration files copied across from the old host which have hardcoded references to the old IP address.

A simple way to test for this is a recursive grep:

steve@skx:~$ rgrep xx.xx.xx.xx /etc

For my server I discovered that some of the Apache setup had rules in place which allowed access to particular directories from only the old address - updating this was simple once I'd noticed it.

Similarly sometimes services allow you to specify which addresses they should listen upon. In my case I'd setup a caching proxy server, and it's configuration file /etc/squid/squid.conf included this entry:

http_port xx.xx.xx.xx:8080

Updating this to match the new IP address was a simple job, and allowed the proxy to continue working.

 

 


Redirecting connections
Posted by dom (62.254.xx.xx) on Sun 26 Jun 2005 at 18:01
xinetd provides the "redirect" configuration directive, which is probably a better way of doing the smtp redirect you mention. Instead of having a "server" and "server_args" entry in the configuration block for your service, just have "redirect = xx.xx.xx.xx 25".

[ Parent | Reply to this comment ]

Re: Redirecting connections
Posted by Steve (82.41.xx.xx) on Mon 27 Jun 2005 at 11:17
[ View Steve's Scratchpad | View Weblogs ]

Neat, thanks for the tip.

(I tend to use inetd, and other than the rate-limitting I've never really explored the additional features of xinetd).

Steve
-- Steve.org.uk

[ Parent | Reply to this comment ]

Re: Migrating from one machine to another
Posted by adumont (80.58.xx.xx) on Sun 26 Jun 2005 at 19:52
Based on my experience in host migrations, what i always tend to propose is:

prepare the new machine, with a new temporal IP address (probably local, i mean not a publicly accesible IP). You install there all you want to migrate. You try it works as you want it.

Then you can do the switch: exchange the IP addresses between the new and old machines.

No need to waut for DNS propagation,...

Should you detect any problem and/or strange behaviour with the new box, you can easily do a "roolback" (please always have an easy rollback solution prepared), switching back again the IP addresses.

I prefer this way of doing things,

Regards,
Alex

[ Parent | Reply to this comment ]

Re: Migrating from one machine to another
Posted by Steve (82.41.xx.xx) on Mon 27 Jun 2005 at 11:18
[ View Steve's Scratchpad | View Weblogs ]

If I were in charge of the machines that's what I'd do.

But in this case I couldn't, I asked the ISP if we could just move the old IP address over to the new box - but they have different ranges allocated to their dedicated hosts, and their UML hosts.

So doing it the 'easy' way wasn't an option. Kinda annoying,but understandable from their point of view.

Steve
-- Steve.org.uk

[ Parent | Reply to this comment ]

Re: Migrating from one machine to another
Posted by dom (163.1.xx.xx) on Mon 27 Jun 2005 at 11:19
This only works if both machines are on the same physical network, of course. In many cases it is necessary to migrate services to hosts on completely different networks.

[ Parent | Reply to this comment ]

Re: Migrating from one machine to another
Posted by hackman (77.70.xx.xx) on Sun 29 Nov 2009 at 10:44
You can migrate the traffic of every tcp based service to the new IP with iptables & iproute2.

You can find a detailed example here: http://hydra.azilian.net/blog/2009/11/migrating-traffic-to-a-new- server/

[ Parent | Reply to this comment ]

Re: Migrating from one machine to another
Posted by Anonymous (82.119.xx.xx) on Mon 27 Jun 2005 at 08:40
How to migrate SMTP server (postfix)? I want to put it on separate machine. I think the only problem is users not changing their settings in Outlook. Most of them have mail.domain.tld there, so I will change that in DNS, but some can have other hostname or even IP address.

So what to do? I can't use port redirect (rinetd or iptables rule) because that would cause all this mail to come from IP of old server which would break realy and anti-spam restrictions. Hmm can I setup postfix on old machine to accept mail as a relay host and deliver it to new one? How to setup postfix to "know" users to which relay? (I have users in mysql)

[ Parent | Reply to this comment ]

Re: Migrating from one machine to another
Posted by Anonymous (213.164.xx.xx) on Mon 27 Jun 2005 at 08:59
You need your users to use mail.domain or you'll have even more trouble later.

Perhaps your logs will show who is not using the mail.domain address, then you could e-mail them, and some time later make the switch.

As a short term measure you could use port forwarding but you really want to fix the problem instead, not work around it.

Depends how many users you have of course..

[ Parent | Reply to this comment ]

Re: Migrating from one machine to another
Posted by cschmidt (65.48.xx.xx) on Mon 27 Jun 2005 at 18:31
I'm going to be doing this in the next couple of days so thanks for the useful article!

I just wanted to ask though, since DNS replication can take so long, is there an easy way to just redirect all the traffic to the new host? I know I can pass traffic on specific ports through my firewall(iptables) to a machine behind it, but could something similar be done per IP?

Doing this could (potentially) eliminate the DNS propagation delays as all the traffic hitting the old box would just get redirected to the new box. If you can leave both machines up for a week concurrently you wouldn't have to worry about people not being able to access the new server.

Would this work? If so, has anyone done this before? How?

Thanks!

[ Parent | Reply to this comment ]

Re: Migrating from one machine to another
Posted by Steve (82.41.xx.xx) on Wed 29 Jun 2005 at 00:53
[ View Steve's Scratchpad | View Weblogs ]

If the machines are both on the same subnet you can give the new host the IP address of the old one easily enough - see virtual addresses for details.

Beyond that you can redirect services to the new host, either piecemeal like I did with the netcat and xinetd example - or wholesale using the iptables firewall.

Personally I was happy to just redirect two services, www + SMTP.

Redirecting everything would make it impossible for me to login to the old host via ssh for example - because the connection would be forwarded to the new host ...

Steve
-- Steve.org.uk

[ Parent | Reply to this comment ]

Redirecting SSL services
Posted by Anonymous (81.57.xx.xx) on Tue 28 Jun 2005 at 23:16

To redirect services running over SSL, one could use stunnel (iptables, netcat or xinetd wouldn't do the job here).

Also, for a LAMP (dynamic web content), it's possible to make the transition invisible to user (no downtime) and to ease rollover (in case something goes wrong) by using a database replication mechanism, traffic redirection, and updating DNS only when everything is validated :


  • setup the sql replication betwheen oldserv (master) to newserv (slave)
  • When you decide to do the switch, just forward the traffic from oldserv to newserv using netcat and/or stunnel. The result is instantanly.
  • Invert the sql replication direction (newserv become master) so your data in oldserv keeps up to date in realtime.
  • If your new setup fits well, update your DNS and wait for the propagation, then remove the netcat/stunnel redirection. If things goes bad, you can switch back to oldserv by just removing the redirection


This make the switch very fast (I mean, not noticeable by users). MySQL and PostgreSQL both offers nice replications mechanisms that allows this setup.

By the way, to complete this description, I used to synchronise files (mostly php sessions in /tmp and cvs repositories) with unison, a sort of bidirectional rsync, very adapted to this uses (switch and/or rollover): it always update both servers files to the newer version.
An evidence: don't forget to backup your data !

[ Parent | Reply to this comment ]

DNS Tip
Posted by seadawg (142.162.xx.xx) on Fri 7 Apr 2006 at 23:34
When you're unable to move your IP address to the new box, a good thing to do is use a bit of planning.
A couple of days before the migration, edit your zone files and change the TTL values to about 5 minutes. Give this change time to propagate, then the dns records shouldn't be cached anywhere for more than 5 minutes.

[ Parent | Reply to this comment ]