This site is now 100% read-only, and retired.

Question: Best tool for bare metal restore of Debian servers?

Posted by paulgear on Wed 4 Jun 2008 at 11:46

I've been doing a bit of searching through the Debian Administration archives and one thing that doesn't seem to have been discussed very much is full system recovery. There are plenty of discussions on different backup options, but nothing targeted at what seems to me the simplest possible backup scenario: protecting a single machine (specifically a server) so that if it is compromised it can be rolled back to a previous state.

My current disaster recovery plan for my (Internet accessible) server is: rebuild from scratch and copy in my rsnapshot backups as necessary. I'm very happy with the way rsnapshot gives me multiple point-in-time backups at very little cost in terms of disk space, but i'm a little concerned about the time to recover that my current plan would involve (especially since I'm self-employed, and time is money).

I've done some web searches and the product that seems to keep popping up is mondo. However, the last D-A article on mondo is a little old. Is it a good choice? Are there other good options?

My expected recovery scenario is having a recovery DVD or USB hard disk drive sitting off-site somewhere (if it's a DVD, i'd probably make a few copies and keep one on site), and recover from that and then restore my latest rsnapshot files from a USB hard disk. How many people have actually tested a disaster recovery plan similar to this and found it to be suitable in terms of quality of recovery and time taken to recover?

 

 


Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Anonymous (85.12.xx.xx) on Wed 4 Jun 2008 at 13:09
Set up a base system, configure it and "clone" it e.g. with dd or similar. Recover the rest from regular backups which you've made after that.

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by ymerlin (213.61.xx.xx) on Wed 4 Jun 2008 at 13:55
When strolling around at LinuxTag2008 I took a look at "Relax&Recover" http://rear.sourceforge.net/

Intro:
"Relax and Recover (abbreviated ReaR) is a highly modular disaster recovery framework for GNU/Linux based systems, but can be easily extended to other UNIX alike systems. The disaster recovery information (and maybe the backups) can be stored on the network, USB devices and DVD/CD-R. The result is bootable rescue system that can be booted via PXE, DVD/CD and USB media."

Sounds like what you need.

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Anonymous (84.255.xx.xx) on Fri 6 Jun 2008 at 00:16
i don't like it being a shell script...

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Anonymous (198.74.xx.xx) on Wed 4 Jun 2008 at 20:32
SystemImager?

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Thorsten (84.58.xx.xx) on Wed 4 Jun 2008 at 21:02
As I have 45 identical Servers n different locations on different hardware, I use preseeding and auto=true on the debian-installer.
Afterwards cfengine automagically configures the server and I´m done.

It took a long time to setup this, but now it runs fine :)

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Anonymous (218.185.xx.xx) on Fri 28 Nov 2008 at 01:50
I also use this solution but go one step further. Once the system has been installed I just give it a hostname and then run 'invoke-rc.d puppet restart' which will automatically download and install configuration files and packages. Then there is just the data to recover (database files etc) which I configure to always be stored in a /data partition.

Note that puppet uses public key encryption and this will have to be fixed up before the new server can download and install it's configuration files.

[ Parent ]

Why don't you use rsnapshot?
Posted by burke3gd (88.112.xx.xx) on Wed 4 Jun 2008 at 21:35
If you already use rsnapshot, why don't you take a backup of the whole filesystem with it? I do it this way and it works well.

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by suspended user mpapet (208.179.xx.xx) on Wed 4 Jun 2008 at 21:54
Logical Volumes are the only way to fly. The debian installer makes it pretty easy to do from the beginning. It's only a couple of steps more than formattinga (sp?) a vanilla ext3 disk. Storing snap shots and restoring backups is pretty easy. Makes it easy to add storage too.

I also use Bacula. Very nice back up solution. It's easier to restore from LVM's but Bacula is great for the average folder/partition restore.

I've been meaning to see if I can restore the logical volume with a live CD. Anyone have experience restoring a logical volume with a live cd?

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Anonymous (12.41.xx.xx) on Wed 4 Jun 2008 at 22:05
I like DAR, although (like tar) it doesn't capture anything below the file level (i.e. no partition info is saved). I wouldn't mind re-doing that stuff manually, but then I only have to worry about two simple servers at home.

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by jlawler (38.98.xx.xx) on Wed 4 Jun 2008 at 22:55
I think the key counter question is how many machines are you intending to implement this for? Some sort of auto-install or imaging system for new machines in a cluster is a pretty industrial strength solution, but it's the easiest way to go when you have large number of machines and don't want to waste time. If you've got 5 or 6 machines, each with different package sets, then you've got an entirely different problem.

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by paulgear (124.171.xx.xx) on Wed 4 Jun 2008 at 23:04

In the question, i asked about a single machine. I have more than that, but it certainly fits your "5 or 6 machines, each with different package sets" scenario better than the "industrial strength imaging solution" scenario. I would want to be able to deploy the same solution for my laptop, which sees a lot of hostile environments in my work as an independent consultant.

Most of the answers above didn't address how to get the data back onto the machine. It's all well and good to say "take a dd", or "use LVM snapshots", or "make full backups with rsnapshot", but it's the restore process i'm concerned about. How do you get a machine to the state where you can actually do the restores? Boot from a Knoppix disk and set up all the partitions & stuff manually? Reinstall? Those are the sort of solutions i want to avoid due to the time they take.

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Steve (82.41.xx.xx) on Wed 4 Jun 2008 at 23:38
[ View Weblogs ]

Really if the backup is offsite, via rsnapshot you've got two problems for a speedy recovery:

  • Bootstrapping a working system so that you can transfer the files back.
  • Transferring the data back - ie. actually restoring.

Me? I take backups of 4-5 machines every few hours (including this host!) via a combination of rsnapshot & backuppc.

For recovery, thus far, I've never had to do a bare metal restoration - usually just restore specific files on request from users, or to repair my own mistakes. Those "repairs" are almost instant.

However the most interesting part of your problem is the bare-metal nature of the restoration. Whilst it is true that booting into a Debian Installation CD-ROM, and running through the partitioning + minimal setup would take time I think that you have to assume that a full restoration is an infrequant excercise, and that the most timeconsuming part of a full restore is going to be the network transfer of all the data back onto the "new" machine.

So I'd probably go that route - either boot into a knoppix CD-ROM via a PXE bootserver, or run through a basic console-only installation of Debian from a CD-ROM/PXE boot. (Search the site for "pxe" to find instructions on how to setup a PXE-environment - trivial to do, and very handy).

If you find that you have too many machines, or that failures are common then I guess systemimager would be a better path.

(IMHO using LVM snapshots can be useful; but it doesn't solve the problem that a restore, over a network, is going to take a lot longer than any other part of the job - realising that is probably key ..)

Steve

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by paulgear (124.171.xx.xx) on Thu 5 Jun 2008 at 00:05

Really if the backup is offsite, via rsnapshot you've got two problems for a speedy recovery:

  • Bootstrapping a working system so that you can transfer the files back.
  • Transferring the data back - ie. actually restoring.

I wasn't planning on leaving the backup off site to do the restore. :-)

...
For recovery, thus far, I've never had to do a bare metal restoration - usually just restore specific files on request from users, or to repair my own mistakes. Those "repairs" are almost instant.

However the most interesting part of your problem is the bare-metal nature of the restoration. Whilst it is true that booting into a Debian Installation CD-ROM, and running through the partitioning + minimal setup would take time I think that you have to assume that a full restoration is an infrequant excercise, and that the most timeconsuming part of a full restore is going to be the network transfer of all the data back onto the "new" machine.

I think the most "interesting" (by which i mean difficult and annoying) part is what to restore from the full backup and what to leave on the restored machine. This page from a previous D-A article has a few hints, but i'd rather use a recovery system that has worked out the exact mechanism without requiring additional thought.

I guess that's the bottom line for me: what is the most bullet-proof recovery system? The big opportunity for mistakes is when i put my fat fingers on the keyboard and start messing with things. I want something that takes the human factor out of it as much as possible.

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Steve (82.41.xx.xx) on Thu 5 Jun 2008 at 11:34
[ View Weblogs ]

If you can be on-site, as you suggest, then I would imagine restoring the complete contents of the backup would be

  • Simple.
  • Less confusing.
  • Lower risk

With onsite access you're literally talking about booting with a knoppix CD, and copying over the contents of your backup data-store to the new system. The only steps you should need are:

  • Partition the disk(s).
  • Do the copy.
  • Install grub.
  • Make sure that /etc/fstab is correct.
  • Reboot.

Still it sounds like you'd want a "one click" restoration system, and to the best of my knowledge nothing like that exists just yet. You need, in some way, to have a working system to accept a backup via rsync/scp/ssh. Failing that you need physical access if you're just going to do a straight copy via dd.

Regardless of what you use to restore the data there will almost always be some fiddling one complete - things like grub, and ensuring that partition names & etc are correct.

Steve

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Lennie (82.75.xx.xx) on Wed 25 Jun 2008 at 09:25
There is a way: automated install & central configuration. So automated install would be something like FAI. Central configuration could be any of these: http://en.wikipedia.org/wiki/Comparison_of_open_source_configurat ion_management_software

It really payes off at large installations.

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Anonymous (140.110.xx.xx) on Thu 5 Jun 2008 at 02:42
There are many full system recovery software like partimage [1], ntfsclone [2] (for windows partition), etc. You can also find partimge-ng[3] for more file system support. My personal suggestion is Clonezilla[4]. Clonezilla provide integration interface (currently only command-line) for full system backup and recovery. Clonezilla is debian friendly, there are apt repositories provided by developers.[5]

Check out Clonezilla LiveCD [6] if you want to give it a try.
BTW, if you need massive full system recovery for PC classroom or PC Cluster, you can check out DRBL[7] for multicast support of Clonezilla.

[1] partimage - http://www.partimage.org/
Linux.com :: Backup and Restore Linux Partitions Using Partimag
http://www.linux.com/feed/59730
[2] ntfsclone - http://www.linux-ntfs.org/doku.php?id=ntfsclone
[3] partimage-ng - http://partimage-ng.org/
[4] Clonezilla - http://clonezilla.org
Linux.com :: Manage partitions and disks with GParted-Clonezilla live CD
http://www.linux.com/feature/115208
[5] Clonezilla/DRBL apt repository -
deb http://drbl.nchc.org.tw/drbl-core drbl stable
[6] Clonezilla Live CD - http://www.clonezilla.org/clonezilla-live/
gparted-clonezilla / clonezilla-sysresccd - http://www.clonezilla.org/related-live-cd/
[7] DRBL(Diskless Remote Boot Linux) - http://drbl.sf.net
http://www.clonezilla.org/clonezilla-server-edition/

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by ashland (203.91.xx.xx) on Sun 8 Jun 2008 at 06:42
I use clonezilla for my Debian, Red Hat and Windows servers. I send the backup to an external drive or through ssh to another server.
Recovery time is usually within minutes.
I also use it to clone identical PCs or servers. It's pretty bullet proof.

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Anonymous (217.91.xx.xx) on Thu 5 Jun 2008 at 10:35

I also use DAR for doing backups and restoring them.

To answer your question on what to restore from the backup: Depends on what your backup contains. I restore everything, but I dont backup everything. My (very shortened) DAR-command goes something like this:
dar -R / -X "*.dar" -X "*.iso" -X "ibdata1" -X "ib_logfile?" -P cdrom -P mnt -P media -P lost+found -P proc -P sys -P dev/pts -P tmp -P var/log/mysql -P var/tmp -D -c foobar
-X means exclude file, -P exclude directory.
The mount-command will give you an idea which directories on your system are on a tmpfs and thus get wiped on shutdown. You generally dont need to backup them. Also I dont backup my MySQL-databases with DAR, there are special tools for doing that.

For the bare-metal-restore:
Boot from a Live-CD (preferably a Debian/Ubuntu-CD), enable the repositories in sources.list and apt-get install dar (or use any other method to get DAR running in your live system). Then partition your disks (dont forget the swap-partition), and mount them. Now restore your backups (dar -x mybackup -R /where/to/restore), first the full backups, then the differential/incremental ones if you have any).
If the partition layout of the new system is different from the old backed up system, you will maybe have to modify /???/boot/grub/device.map, /???/boot/grub/menu.lst and /???/etc/fstab (replace ??? with the path where you restored the files). The same is true if you use UUIDs to identify your partitions (vol_id gives you the UUIDs of your new partitions).
If your NICs changed too, check /???/etc/network/interfaces and (if you use udev) /???/etc/udev/rules.d/70-persistent-net.rules

Last step: Install grub with "grub-install --directory /??? /dev/sdxx" and "chroot /???/ update-grub" and reboot into your restored system.

I dont think its worth to automate these steps because with five servers youll have to do that very rarely.

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Anonymous (67.88.xx.xx) on Thu 5 Jun 2008 at 17:29
We usually have physical access and we use Mondo. First we create a basic image with Mondo and create the server. If something goes wrong then we restore with mondo and use our rsync (custom) backup system to put all the files that change regularly back on. If you wanted a brain dead option and not have to do the second step, you could just run mondo on a nightly basis and move the image to a USB drive. When the time comes you can use the mondo images to do a full restore and walk away. This works great when imaging on the same hardware but might take some extra steps if you go to different hardware.

This only gives you a single day of backup unless you save the images, which can get very large. I think that is why you see most people use a mix of setting up a base system and then recovering from backup because they want snapshots of data for other reasons besides disaster recovery. You know the famous, oops I need the old version of this file I just overwrote :).

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Anonymous (67.88.xx.xx) on Thu 5 Jun 2008 at 17:38
Also, the only reason why we wrote our own backup system in perl was because it was early 2004 and not a whole lot of other options existed. When I upgrade to a better NAS I think we will probably switch to another option. The three I am considering are rsnapshot, backuppc and duplicity. We will probably replace Mondo with clonezilla. Not because mondo is bad but just because we use clonezilla for all of our workstations. The extra feature of imaging across the network and it's incredibly fast compared to other solutions we have used.

We had to do some hacking to get mondo to work correctly with Sarge and Raid I think more because of Debian than Mondo. When doing software raid everything can get a bit more interesting.

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Anonymous (210.54.xx.xx) on Thu 5 Jun 2008 at 22:59
We use Rdiff-Backup for our environment, we run it every night in to central backup server. In terms of disaster recovery we have customized Debian LIVE CD to run on different hardware which has rdiff-backup installed, we boot from it and restore system from the any incremental backups we wish.

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Anonymous (158.110.xx.xx) on Fri 6 Jun 2008 at 09:11
You get the point: bare metal restore.

IMHO the best practice is planning bare metal restore from the install fase of a machine, and the best tool I found for this is FAI[1] (FAI the fully automated installation framework for linux).

I don't use it yet, but my plans are to use it to install all my servers.

You can read an articole here[2] on Debian Administration.

With network boot you can restore a server in minutes, or you can boot from a fai-cd.

You can have different versions of you configs via SVN or CVS.

For live data backup / restore you can use any solution.

[1] http://www.informatik.uni-koeln.de/fai/
[2] http://www.debian-administration.org/articles/240

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by atrixnet (24.27.xx.xx) on Fri 6 Jun 2008 at 10:16
[ View Weblogs ]
Oh, system imager + flamethrower all the way, baby. What an awesome set of tools. Wizards included (guided configuration, creation, and deployment of system images)...you can't mess it up! You get your system recreated right down to the disk partitions, LVM, etc. You can't lose with this. It's so easy and the docs are straight forward, though you shouldn't really need them. Just call the various programs in the SI suite and you're off and rolling in no time. It is so fast too. I love it. Very happy with this solution for purposes of disaster recovery and quick system deployment over a network (network boot, install OS, configure it, etc). Updating images you've already created is also a snap. What a winner, I tell you.

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by stoffell (81.164.xx.xx) on Fri 6 Jun 2008 at 20:24
We always try to install an openvz kernel and keep our machines in the VE. Even if it's just 1 machine. (rarely happens since most machines these days have dual or quad cores and way too much memory to just perform 1 task :-))

Backup and restore with openvz and VE's is soooo sweet and easy..


---
stoffell

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Anonymous (72.24.xx.xx) on Sat 7 Jun 2008 at 21:43
I have been testing Mondo in combination with rsnapshot for our disaster recovery solution. So far it looks really good.

Here's how I have my systems set up. I have separate physical partitions for our database storage, virtual Apache web servers, and /home. I do an bare metal backup of the / and /boot partitions, which encompass all but the three partitions I have already mentioned, with Mondo. I then do daily backups of the other three partitions with rsnapshot.

I create dvd .iso images with Mondo and have tested them successfully. Mondo does a very good job in my estimation. The only problem I have run into so far is with the 2.6.24 amd64 kernel. I have to boot the systems into another kernel to do the Mondo backups, but they successfully restore to a working amd64 kernel.

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Anonymous (75.85.xx.xx) on Fri 13 Jun 2008 at 05:59
Has anyone ever tried to rsync a new machine back to life?

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Anonymous (62.41.xx.xx) on Fri 13 Jun 2008 at 09:05
Hi, i use Bru Server, http://www.tolisgroup.com/products/bruserver/
nice tools, agents etc, but $$ kosts money.
Im using it now for about 1 year and i works very well.
i backup 6 servers with it.

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Lennie (82.75.xx.xx) on Tue 24 Jun 2008 at 09:08
Yes and there is nothing really complicated about it.

We keep a full-backup (excluding /proc, /sys and maybe /dev) of all the systems on a backup-server.

We keep a file with the output of df -h in /etc/, so combined with /etc/fstab we know the layout and size of the different paritions (there is also /var/log/dmesg if you need more hardware information).

When you want to do a restore:
pop in a live-system (cd, usb whatever)
ssh into the backup-server to look at the df and fstab
create the filesystems and mountpoints for example in /target
create a /proc, /sys.
And rsync them back.

Then login to the newly create system.

There are 2 ways:
1. chroot:
chroot /target/bin/login
2. use the live-cd to boot from the newly created system:
specify the right kernel-line on the live-system boot

when you are logged into the system, run grub or lilo (or silo or whatever) with options if needed. And then you can boot in the newly restored system.

So we haven't automated it, but it doesn't happen like every day or something.

And it's just a few commands.

Although with /sys and /dev and udev added in later releases it has become more work, for example this file also needs to be kept an eye on:

/etc/udev/rules.d/z25_persistent-net.rules

It specifies the name (ethN) of the network interface.

We use it for doing upgrade-testing as well (usually we create only one filesystem and maybe leave out some data).

[ Parent ]

Some extra tips
Posted by Lennie (82.75.xx.xx) on Tue 24 Jun 2008 at 09:43
some live-systems have a newer mkfs, they might create a filesystem with newer features. So after the filesystem is created, you might need to use tune2fs/debugfs to disable some of them.

As older Debian-kernels might otherwise maybe not understand them and refuse to mount the partition.

The chroot-methode allows you to do a lot of other things as well, as install a different kernel because it's different hardware or restoring on.

I started doing this a long time a go, when servers still had the possibility to use floppy disks, and used tomrtbt-disk and an extra disk with a compressed statically linked rsync.

I still keep a statically linked rsync (available from the backupserver by ftp, http, scp, sftp).

You statically link it this way: CCFLAGS=-static make

You can see if it's statically linked with: file rsync

To make the rsync-file smaller you can use strip rsync (removes debug-symbols)

You can also use: ldd -v rsync

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Anonymous (66.167.xx.xx) on Fri 13 Jun 2008 at 22:10
If you're considering commercial products. Look at Storix. It will do bare metal recovery to different hardware and supports recovery to different sofware raid and LVM configurations.

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Anonymous (15.219.xx.xx) on Wed 18 Jun 2008 at 03:06
I have a few servers which have a lot of common packages and then some customisations on top.

What I have done is place all the configs and package dependancies in my own repo & packages. I can just build server from disk point it to my repo and install the package for that server which is linked to say custom-env and custom-env-vim and slapd-domain.name.here.

The data well I have my own scripts that rsync/ssh/zip the stuff around the place to a couple of different servers in a couple of different areas

Making my own packages which are kept in cvs gives me a lot of control. And yep I have 2 working repo's as well just in case

Alex

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Anonymous (62.12.xx.xx) on Mon 14 Jul 2008 at 10:22
Why don't you combine your rsnapshots with a true image?
Make a new image after major changes and incremental rsnapshots after that?

You can store the image off-site for additional reliability or make several copies. I know some friends who use Ghost4Linux:
http://www.howtoforge.com/back_up_restore_harddrives_partitions_w ith_ghost4linux

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Anonymous (92.73.xx.xx) on Thu 27 Nov 2008 at 13:24
There is no easy question to that answer.
As always, that heavily depends on your setup.
How much data are we talking about? How volatile is it?
Can you spend money on the backup solution? (translates to: How much would the downtime cost you?). How is your harddisk setup?
Do you have a remote management console or do you have to pay hands-on?

If the data is highly volatile:

In gerneral, if we are not talking of a non-harddisk failure I'd suggest investing into an RAID with hot spares. This is usually the solution with the best cost/performance ratio, since the failure of a harddisk does not affect the system at all (in terms of downtime). You can use bacula to return your system to arbitrary states then.

If we are talking of a worst case scenario (hardware failure of unknown kind, including data corruption), we have to talk about the money the downtime will cost you. On calculating that, take into account : Your work, the loss of confidence of your customers, costs for travelling, your loss of service level and such.
As a rule of thumb: if it reaches the cost of the server (including maintenance and collocation for the period you expect between 2 crashes) within the time you need to recover it using any other solution, maybe because it is colocated far away, the best solution is a hot standby server. This is because it will cost you the same amount of money but does not affect your service level.

If the data is not that volatile and none of the above conditions is met, I'd make an image using partimage and use either xfs for my data partition (you can make nifty incremental dumps with xfs) or use bacula.

Hope this helps.

[ Parent ]

Re: Question: Best tool for bare metal restore of Debian servers?
Posted by Anonymous (68.197.xx.xx) on Wed 10 Feb 2010 at 21:53
I know I'm posting to a really old post -- but then this topic never get's old <grin>

We just released a tool for doing Bare Metal Backups we call Kleo. It's a graphical wrapper around partimage. Partimage is fantastic imaging tool, but the user interface was always a challenge. We created a wizard style interface to make using this excellant tool a bit easier.

We bundle Kleo with our Carroll-Net Server Recovery Kit. An Unbuntu 9.10 remix packed with hundreds of specialized server recovery tools.

Both Kleo & the CnSRK are free. http://www.kleobackups.net

[ Parent ]