This site is now 100% read-only, and retired.

Preparing a machine to instantly deploy in case of a failure?

Posted by bacardi on Tue 22 May 2007 at 09:58

This must be a common problem, but I can't find the canonical solution. We have two identical machines running a few medium size websites. One is a production machine, the other is an identically setup machine used for development. We want to be able to deploy the development machine quickly in case of problems with the primary server - how should we go about that?

Currently, when features on the development server are adequately tested, they are pushed into production.

Due to a tight budget, we do not have a live redundant backup to our production server (other than its RAID 1). One thing we do have is hard drive space - so we've been trying to setup a process where live production data (mysql, apache) is backed up nightly onto the development machine, so if the production server goes down, we can flip the development server into production mode.

This seems to be an extremely dirty method, requiring manually changing a lot of configuration files, questions about whether the development server can actually acquire the production IP, and many other quirks.

I have not completed this process, but there must be a more efficient (standard?) way. I've only been involved with Debian for a little over a year, but have done enough Linux to pick things up quickly.. and after some browsing, haven't run into many cases of easily converting a development machine into production mode.

Any suggestions would be appreciated!

 

 


Re: Preparing a machine to instantly deploy in case of a failure?
Posted by Anonymous (213.156.xx.xx) on Tue 22 May 2007 at 10:56
My suggestion:
Use rsync to syncronize the following directory on the development server
/etc2/apache2
/var/www2

with the following directory on the production server:
/etc/apache2
/var/www or whatever your Document root is

in case of disaster:
copy from /etc2 to /etc the needed directory
rename /var/www in /var/www.dev and /var/www2 in /var/www
assign the ip of the production server to an alias of eth0


[ Parent ]

Re: Preparing a machine to instantly deploy in case of a failure?
Posted by Anonymous (62.172.xx.xx) on Tue 22 May 2007 at 12:55
Worth mentioning is that rsync does an incremental update, comparing the differences of the local and remote file/directory structure. This means it only transfers across any differences. You can set rsync to run in a cron job every 5 mins, and it will result in a negligible increase in bandwidth.

You can also limit the bandwith to e.g. 120kb/s with the command line option --bwlimit=120

[ Parent ]

Re: Preparing a machine to instantly deploy in case of a failure?
Posted by marki (15.195.xx.xx) on Wed 23 May 2007 at 18:04
Yes,
but when you have thousands of files, rsync needs to compare them all... and that can take some minutes...

[ Parent ]

Re: Preparing a machine to instantly deploy in case of a failure?
Posted by neofpo (200.185.xx.xx) on Wed 30 May 2007 at 02:07
[ View Weblogs ]
http://packages.debian.org/unstable/admin/incron

This tool may help sync only the changed files, when they change. You may even get some e-mail triggers to important files, such as configuration files. It is on unstable, but I believe it can be compiled under Etch.

I had similar demands at my work. At that moment, bringing up some high level solution would not suit the time / resources available. A couple of rsyncs and cron scripts to sync MySQL did the job and proved stable and functional.

[ Parent ]

Re: Preparing a machine to instantly deploy in case of a failure?
Posted by Xeeper (213.134.xx.xx) on Tue 22 May 2007 at 11:04
How silly it might sound. Check the Linux Virtual Server (http://www.linuxvirtualserver.org) project. The solution, just create a server pool with just one server (the production). For the sorry server configure your development server.

However, you need to make a few adjustments on your development server to make this work. For instance, the night database backup has to be restored every night on the development database server (preferable under the same name as your live database) and in apache you need to make an extra VirtualHost defintion for the production website. We (I) use Ultra Monkey (http://www.ultramonkey) as the director.

This is how it works: Ultra Monkey periodically checks your server on IP bases and a special configured webpage which needs to return a predefined response. Using this page you can check depended services such as your database connection, SOAP services or whatever you need. As long the IP is connectable and the webpage returns the predefined response it is considered 'alive'. When a check fails the director will remove the particular (real) server from the pool. When all real servers are removed from the pool, the director will use the sorry (your development server) until (one of) the real server(s) is 'up' again.

This technique is called High Availability.

[ Parent ]

Re: Preparing a machine to instantly deploy in case of a failure?
Posted by Anonymous (62.172.xx.xx) on Tue 22 May 2007 at 12:50
I would suggest keepalived over Ultra Monkey, as it's maintained and is in debian apt :-)

[ Parent ]

Re: Preparing a machine to instantly deploy in case of a failure?
Posted by Xeeper (213.134.xx.xx) on Tue 22 May 2007 at 13:26
Ultra Monkey provide their own deb packages outside the official debian archive.
I'm not sure if they have Etch packages, but this lines come from my sources.list:
deb http://www.ultramonkey.org/download/3/ sarge main
deb-src http://www.ultramonkey.org/download/3/ sarge main

To prevent a Emacs vs VI discussion, just choose the director which suits your environment the most. Both are very good.

[ Parent ]

Re: Preparing a machine to instantly deploy in case of a failure?
Posted by thoger (62.168.xx.xx) on Tue 22 May 2007 at 20:07
AFAIK UltraMonkey 3 is not a lot more than (now quite old) HeartBeat (http://linux-ha.org/, includes ldirectord) packages built for several distributions. Official Debian archive contain more up-to-date packages, both 1.x and 2.x branch. I don't think there is any reason to use UltraMonkey packages on current Etch.

[ Parent ]

Re: Preparing a machine to instantly deploy in case of a failure?
Posted by Anonymous (87.78.xx.xx) on Wed 23 May 2007 at 10:08
Maybe a simpler approach might be to put a small linux router in front of the production and development servers, with a few netfilter rules forwarding all traffic to the production server. Switching to the development server would just need to change forwarding to this server (easily scriptable).

This does not tackle synchronization problems; but for the connectivity it might be a less involved approach than LinuxVirtualServer (which redirects network traffic on a lower level?) which probably also needs a front node(?). An additional benefit could be to automatically route requests from your development client to your development server.

[ Parent ]

Re: Preparing a machine to instantly deploy in case of a failure?
Posted by Anonymous (88.198.xx.xx) on Tue 22 May 2007 at 12:07
If the OS is not broken, pull the (remaining) disks out of the production machine and push them in the dev box.
Perhaps you need to finish of with a network change or two, but then you're done.


Other than that, there are quite a few teorethical solutions like distributed/network block devices, but none of them proves practical.
(those who disagree: have you really tried this in production?)

One of the promising options is to use a distributed filesystem; but afaik only commercial ones are "usable" on linux right now.

[ Parent ]

Re: Preparing a machine to instantly deploy in case of a failure?
Posted by Anonymous (62.140.xx.xx) on Tue 22 May 2007 at 13:41
If you count 16TB DRBD'd across to a co-lo over a 100Mb link, then "Yes", I have used it in production.

Cheers.

[ Parent ]

Xen
Posted by Anonymous (69.17.xx.xx) on Tue 22 May 2007 at 21:52
The traditional way is to use rsync, but the best way is to use Xen. You get automatic failover and no downtime, or at least so I've heard.

[ Parent ]

Re: Xen
Posted by Anonymous (71.90.xx.xx) on Tue 22 May 2007 at 23:43
Xen handles the virtualization, but as far as I know, you would still need something to handle filesystem synchronization and system failover. If you want a simple use manual approach, just write an script that uses rsync to sync the files, and set the IP address for the websites as an alias while just using different individual IPs for managing the systems. That way if one goes down, you just bring up the alias, which is easier than trying to switch the main system IP and all the necessary configs in a hurry. To automate this process, some type of HA application like Heartbeat would be in order. A simple setup to just monitor if a machine goes down and then move the ip/services over the other, is called an active/passive setup and is very easy to get going.

[ Parent ]

FS synchronization
Posted by Anonymous (69.17.xx.xx) on Wed 23 May 2007 at 04:01
In theory you could do filesystem synchronization "live" by doing raid 1 where one of the mirrored disks was on the backup system mounted via a dedicated gigabit ethernet link, a crossover cable, and iscsi. Don't know how well that would perform in practice but it would be easy to benchmark with bonnie after you'd set it up. Performance would depend on how well the raid1 code deals with having one disk slower than the other. Obviously if you're doing a large amount of disk writing then your total bandwidth will be limited by how fast the data can get to the remote disk. But if your I/O is bursty or you're just doing reads then there's no reason the system needs to wait on the remote disk.

Surely somebody's already done this.... Ah, Linux-HA.org recommends DRBD.

While you're at it don't forget to use lvm2.

[ Parent ]

Re: FS synchronization
Posted by Anonymous (82.225.xx.xx) on Wed 23 May 2007 at 10:31
I confirm : I used heartbeat + drbd in production, it works great.
What I have done :
- heartbeat to monitor the server availability
- rsync script in a cronjob to sync system files, since you can't use drbd on a root part.
- drbd partition(s) to sync data in real-time. Works great with firewire network ;)
What I have not done but would be cool :
- monitor services with something like mon, since heartbeat basically monitors only network connections.

Hope this help,
cheers,

Luc

[ Parent ]

Re: FS synchronization
Posted by matej (213.160.xx.xx) on Sat 26 May 2007 at 23:07
...but you can symlink /etc/apache2 => /www/apache2 (or whatever directory layout you use).

[ Parent ]

Re: Preparing a machine to instantly deploy in case of a failure?
Posted by Anonymous (83.204.xx.xx) on Mon 28 May 2007 at 21:53
If you have a spare DVD writer, a good (and straightforward solution) is to use you script for backing up live data and then use mondo!

Mondo will create a bootable DVD that will reinstall your system anywhere you want.

In fact i use this solution in two stage:
* build nightly daily changing data backup using simple genisoimage + wodim on DVD
* build from time to time a system backup without daily changing data using mondo

This quite efficient, if you have 2 DVD+RW and a DVD writer.

[ Parent ]

Re: Preparing a machine to instantly deploy in case of a failure?
Posted by Anonymous (67.48.xx.xx) on Wed 30 May 2007 at 03:59
Problem of 'nightly' and the script that moves the DVD to the other machine and powers it on comes to mind.

[ Parent ]

Re: Preparing a machine to instantly deploy in case of a failure?
Posted by Anonymous (66.68.xx.xx) on Fri 8 Jun 2007 at 05:40
Heartbeat is clearly the only real enterprise class solution here out of the ones mentioned. We run mail cluster of two ibm x306's and we move a million messages in a month. When one goes down, 3 - 7 seconds later the services are back up. We never get a phone call. If you use drbd in conjunction with heartbeat, you can have a double fault tolerant system with failover: Hardware Raid1->DRBD DistribFS Mirroring->Heartbeart (Failover).

[ Parent ]

Re: Preparing a machine to instantly deploy in case of a failure?
Posted by Anonymous (66.65.xx.xx) on Tue 19 Jun 2007 at 06:37
A vserver with your lamp/whatever environment inside of it with drbd is a varient I like, it simplifies implementation and allows for quick migration of machines.
http://oldwiki.linux-vserver.org/Vserver+DRBD

ucarp is another toolkit no one has mentioned
http://xn.pinkhamster.net/blog/tech/mail/high-availability_smtp_w ith_ucarp_on_debian.html

[ Parent ]

Re: Preparing a machine to instantly deploy in case of a failure?
Posted by sbox (208.60.xx.xx) on Mon 25 Jun 2007 at 15:18
rsync is great. I've never tried Unison, but it is an option. I've fallen in love with virtualization though. I'd recommend going that route. Alternatively, MySQL replication is not that difficult to configure, even using SSL to encrypt the traffic is easy using stunnel or ssh. I'm not sure what your webserver configuration is like but setting up another set of virtual hosts or a second instance of the system should not be terribly difficult. Virtualize the backup of the production if you want.

[ Parent ]