This site is now 100% read-only, and retired.

Question: Making efficient backups on Debian?

Posted by Serge on Fri 10 Dec 2004 at 10:05

I was wondering what needs to be backed-up on a Debian system, without backing up too much, considering packages can be re-downloaded. Until now, I've been using this various pieces (amongst others) in my backup script, on a server that performs typical LAMP, DNS en email services.
dpkg --get-selections > /var/backups/apt-backup-selections
mysqldump -u root --password=$MYSQLROOTPW --all-databases --opt > "/var/backups/mysql-$DATE.sql"
tar --absolute-names --create -j --file $MOUNTDIR/$BACKUPDIR/backup-$DATE.tbz --exclude-from=$EXCLUDE $INCLUDE
After this I make remote stored backups of
/etc 
/root 
/var/cache/bind 
/var/lib/dpkg 
/var/backups 
/var/www 
/usr/local

I somehow am in doubt whether this is enough, at least to the extent of the base system and the configurations. More specially, a good backup of the apt system seems important to be able to easily recover the host.
I should actually take the time to test a disaster recovery with this setup, but as I don't have a second Bytemark host handy, I'm keeping using this as a (bad) excuse :)

So what are other people here doing about their Debian host backups?


Serge

 

 


Re: Question: Making efficient backups on Debian?
Posted by Steve (127.0.xx.xx) on Fri 10 Dec 2004 at 10:32
[ View Weblogs ]

You seem to have all the important things covered, with the possible exclusion of any user home areas, and any mail spools.

I have a script which I wrote which backs up the following directories:

  • /etc/
  • /home
  • /usr/local
  • /var/lib/dpkg
  • /var/spool
  • /var/log

Storing the output of dpkg --get-selections is handy, but it's redundent as that information can be rebuilt from the /var/lib/dpkg/status file, so I don't bother with that step.

I only keep five days of backups by putting everything in a .tar.gz file, and rotating it that keeps the space usage down, but gives me a week to realise I've deleted something I wanted to keep!

I do find that keeping logfiles is useful, especially if you're worried about a system compromise ..

-- Steve.org.uk

[ Parent ]

Re: Question: Making efficient backups on Debian?
Posted by Anonymous (68.63.xx.xx) on Mon 30 May 2005 at 23:37
Instead of backing up /var/lib/dpkg, which is big and static depending on you package selection, I use dpkg --get-selections > package-list.
I also backup /var/cache/debconf which keeps all of your debconf configuration intact. I install the .dat files right after debootstrap and right before setting the package list and installing the other packages.

[ Parent ]

Re: Question: Making efficient backups on Debian?
Posted by Anonymous (203.167.xx.xx) on Thu 18 May 2006 at 02:01
If you apt-get install debconf-utils, you can use a similar util for debconf, "debconf-get-selections".

[ Parent ]

Re: Question: Making efficient backups on Debian?
Posted by asg (127.0.xx.xx) on Fri 10 Dec 2004 at 13:30

Another, though somewhat tangential, item that one might keep at least a local copy of (generated from a daily cron job) is a list of all files including owner/permissions for those pebcak moments when you get creative with chmod -R :)

We get a lot of people on #debian asking things like:

I just typed

chown -Rf root:root /
and went for a coffee! Help.

A simple recovery for such cases would be having a local copy of the system's ownership and permissions.

[ Parent ]

Re: Question: Making efficient backups on Debian?
Posted by Serge (127.0.xx.xx) on Fri 10 Dec 2004 at 14:22
[ View Weblogs ]
a list of all files including owner/permissions

I guess dumping a file list doesn't seem to difficult with ls, but at first sight, I can't think of a system to automate recovery of the permissions.
Would you care to enlighten us? :)

debian/rulez

[ Parent ]

Re: Question: Making efficient backups on Debian?
Posted by asg (127.0.xx.xx) on Fri 10 Dec 2004 at 14:50

Well, I expect some grep/awk/sed fu or a perl script could do the job easily, though I don't have one written (since I've never had to personally recover from such a situation.)

I expect one could also generate a cfengine ruleset to monitor such things automatically but I don't know about the scalability of it (my laptop currently has 140k files which would likely be very taxing for cfengine.

The list is often more helpful if one accidentally frobes a few files and would like to reset them to defaults (without having to reinstall the constituent packages, etc.)

[ Parent ]

Re: Question: Making efficient backups on Debian?
Posted by Serge (127.0.xx.xx) on Fri 10 Dec 2004 at 20:29
[ View Weblogs ]

I have been thinking some more...(ouch!)
As root:

apt-get install acl
cd /
getfacl -R / > /acl.txt
setfacl --restore=/acl.txt

should do the trick, even if you don't have acl's in your kernel and/or in your fs.

Serge van Ginderachter

[ Parent ]

Re: Question: Making efficient backups on Debian?
Posted by Steve (127.0.xx.xx) on Fri 10 Dec 2004 at 14:48
[ View Weblogs ]

Installing something like aide or tripwire would allow you to build up such a database easily.

However it might be hard to recover from..

-- Steve.org.uk

[ Parent ]

Re: Question: Making efficient backups on Debian?
Posted by asg (127.0.xx.xx) on Fri 10 Dec 2004 at 15:15

Well,

 find / | grep -v -e ^/proc -e ^/tmp -e ^/sys | xargs ls -ld > /var/backups/system-file.lst
generates the desired information.

[ Parent ]

Improvement on grep
Posted by docelic (213.202.xx.xx) on Mon 3 Jan 2005 at 18:56
find / | grep -vEe '^/(proc|tmp|sys)' | xargs ls -ld > /var/backups/system-file.lst

[ Parent ]

Re: Question: Making efficient backups on Debian?
Posted by Steve (127.0.xx.xx) on Mon 20 Dec 2004 at 08:52
[ View Weblogs ]

It's worth noting that some packages stick their data in strange locations - for example the Gallery package places all its albums in /usr/share/gallery/albums.

It could be worth making sure that the packages you have installed don't store date in non-obvious directories like that. Or you might miss them in the backup.

Steve
-- Steve.org.uk

[ Parent ]

have it easyer
Posted by Anonymous (127.0.xx.xx) on Fri 24 Dec 2004 at 07:42
I keep all my backupworthy files in a svn-repository.

[ Parent ]

Posted by Anonymous (68.77.xx.xx) on Mon 3 Jan 2005 at 04:46
Doing any one of these would be a good start: apt-get install hdup apt-get install dar apt-get install rsnapshot apt-get install rdiff-backup For people with more complex needs like backing up many different machines, then try: apt-get install bacula

[ Parent ]

Mysql Backups
Posted by TokenGoth (193.195.xx.xx) on Fri 7 Jan 2005 at 12:34
mysqldump -u root --password=$MYSQLROOTPW --all-databases --opt > "/var/backups/mysql-$DATE.sql"
Would you honestly ever like to be in a position to have to restore a single database out of what could be a massive .sql file? Heres a snippit of the script I use:
#Options:
CACHE=/var/backup/
MYSQL_USER=root
MYSQL_PASSWD=rootpwd


  test -d $CACHE/mysql && rm -rf $CACHE/mysql/* || mkdir -p $CACHE/mysql

  MYSQL_DATA_DIR=$(grep datadir /etc/mysql/my.cnf | sed -e "s/.*\=//")

  echo "Starting Mysql backup"
  COUNT=0
  for db in $(find $MYSQL_DATA_DIR/ -type d | sed -e "s/\/var\/lib\/mysql\///");
  do
   #echo "[$(hostname)]  Dumping Mysql DB: $db"
   mysqldump $db -a --add-drop-table \
        --allow-keywords  -q --user=$MYSQL_USER -p$MYSQL_PASSWD\
        | bzip2 -c \
        > $CACHE/mysql/$db.sql.bz2
   if [ $? = "0" ]
   then
    echo "  Dumped $db to $CACHE/mysql/$db.sql.bz2"
    COUNT=$(($COUNT+1))
   fi
   if [ $COUNT -eq 0 ]
   then
     echo "No databases dumped!"
   fi
  done

Probably a few crap things about that, but it Works For Me™

[ Parent ]

Re: Mysql Backups
Posted by Simon (89.243.xx.xx) on Tue 9 Oct 2007 at 11:43
That's really handy, thanks!

[ Parent ]

Re: Question: Making efficient backups on Debian?
Posted by gonad (203.118.xx.xx) on Sun 1 May 2005 at 04:04
rdiff-backup is an awesome backup utility

[ Parent ]

Re: Question: Making efficient backups on Debian?
Posted by Anonymous (192.38.xx.xx) on Tue 10 May 2005 at 11:15
dirvish is also a good backup utility - it is based on rsync and the use of hardlinks.

[ Parent ]

Re: Question: Making efficient backups on Debian?
Posted by Anonymous (66.73.xx.xx) on Wed 9 Nov 2005 at 08:16
Dirvish is indeed nice.

I prefer rsnapshot, a simpler backup utility also based on rsync and hardlinks.

Both dirvish and rsnapshot are in the Debian package system.

Another, more secure backup utility called rlbackup 2.20 is not yet available as a Debian package. I believe bnl.gov uses rlbackup on some of their Debian Sarge servers. Lack of Debian package is the reason I am not using it.

[ Parent ]

Re: Question: Making efficient backups on Debian?
Posted by Anonymous (212.80.xx.xx) on Sat 22 Apr 2006 at 14:22
I also moved from custom .tar.gz-scripts to dirvish, because it may take a long time to restore single files from archives or even find the correct version before the dumb change happened.

[ Parent ]

Re: Question: Making efficient backups on Debian?
Posted by garo (213.119.xx.xx) on Fri 22 Sep 2006 at 08:32
Instead of listing what to backup, isn't it safer to list the things you don't want to backup and backup the rest ?
What would you list in your 'don't-care-list' ?

[ Parent ]

Re: Question: Making efficient backups on Debian?
Posted by ericnils (66.31.xx.xx) on Sat 21 Oct 2006 at 05:52
Rebuilding a system is much easier if you back up any service configuration files. If you do not work with a configuration file on a regular basis then customizing it can be time consuming. This is especially frustrating if you are rebuilding a configuration file to match restored data.

I regularly back up the following configuration files:
Apache2: /etc/apache2/apache.conf, /etc/apache2/sites-available/, and /etc/apache2/mods-enabled/
PHP: php.ini
Samba: /etc/samba/smb.conf
MySQL: /etc/mysql/my.cnf

If you are using a locally stored backup script to back up to a remote or removable device then you should also back up your backup script.

[ Parent ]

Re: Question: Making efficient backups on Debian?
Posted by Serge (84.197.xx.xx) on Sat 21 Oct 2006 at 10:39
[ View Weblogs ]
You obviously want to backup the whole /etc directory IMHO, amongst others.

--
Serge van Ginderachter

[ Parent ]

Re: Question: Making efficient backups on Debian?
Posted by endecotp (62.253.xx.xx) on Fri 9 Mar 2007 at 14:19
[ View Weblogs ]
I knew I'd find the answer here :-)

I backup my home computers over the net to an online server using rsync. Because of the limited bandwidth and quota on the server I can't just backup everything, so I copy things like /home (carefully avoiding things like Iceweasel's cache), some of /var and /usr/local, and /etc. At one time I tried to prune what I copied from /etc (e.g. the 3 MBytes in /etc/X11/xkb and the ridiculously-overhuge gconf) but that was too painful to maintain.

To aid recovering the Debain packages, I had been doing "dpkg -l", but I was aware that that could be truncating the package names! The "dpkg --get-selections" suggested above avoids that problem.

One thing that is not included in the --get-selections output is the package version. This raises the interesting question of what to do about package versions when you restore. On the one hand, you could install the latest versions of each package. This has the advantage that the packages will all be available "from the usual places", but the disadvantage that your configuration files may need to be updated because the package has changed, or new dependencies may need to be installed, or functionality may have unexpectedly changed. The package might not even exist any more! If the aim of your backup/restore policy is to get back to your position immediately before the failure then this isn't what you want. So on the other hand you could (how?) store the package versions, and then (how?) install those same versions when restoring. What do other people do about this?

One of the comments above notes that you should store your debconf data. Good point. Would it be possible to semi-automate the restore by generating, either at backup-time or restore-time, a "pre-seeded" debian installer thingy based on the saved debconf data? I know next to nothing about this, but if it's a one-liner then it would be good to know. Google suggests "debconf-get-selections --installer > file ; debconf-get-selections >> file".

Pre-seeding deals with the debconf configuration, but restoring the /etc config files when the packages that they correspond to may have been upgraded is another challenge. Is there some way to run the packages' "here are the differences between your version and the maintainer's version of the conffile" dialog after restoring your old /etc contents? Storing your /etc files in Subversion and using its merge features would also help; I really must get around to trying that.


Phil.

[ Parent ]

Re: Question: Making efficient backups on Debian?
Posted by linulin (91.202.xx.xx) on Tue 29 Nov 2011 at 00:05

The loss of package version information is the right concern. 'dpkg --get-selections' is good for up-to-date systems installed from a single supported release, (e.g. stable). But if you mix stable + testing + experimental for whatever reason, then automatic restoration to the same state is not possible, even if all required package versions are still available in the repositories of interest.

Could anyone propose some clever tricks to work this problem around?

--
...Bye..Dmitry.

[ Parent ]

Re: Question: Making efficient backups on Debian?
Posted by clown (84.204.xx.xx) on Fri 12 Oct 2007 at 12:35

It is quite unsecure to expose your credentials on command line, especially, if you do your backups automatically! Reconsider not to calling mysqldump this way:

mysqldump -u root --password=$MYSQLROOTPW ...

There is a user debian-sys-maint created by the installation scripts who perfectly fits for doing backups: it has All PRIVILEGES on all databases and the system stores it's credentials in /etc/mysql/debian.cnf ini-style configuration file readable by the root user only. The problem is how to get credentials from the configuration file.

I'm using this little script named parse-ini to parse ini-file:

#!/bin/sh
[ -z "$1" ] || [ -z "$2" ] && exit 1
sed -e 's/[[:space:]]*\=[[:space:]]*/=/g' \
    -e 's/;.*$//' \
    -e 's/[[:space:]]*$//' \
    -e 's/^[[:space:]]*//' \
    -e "s/^\(.*\)=\([^\"']*\)$/\1=\"\2\"/" \
   < $1 \
   | sed -n -e "/^\[$2\]/,/^\s*\[/{/^[^;].*\=.*/p;}"

It returns a sourceable list of variable definitions from $2 section of $1 ini-file. Now, my backup script looks like:

touch $mysqldump
chmod 600 $mysqldump
eval $(/usr/local/bin/parse-ini /etc/mysql/debian.cnf client)
mysqldump -A -u $user -p$password > /srv/backup/mysqldump_all.sql

First two line are also important for security, don't forget them!

[ Parent ]

Re: Question: Making efficient backups on Debian?
Posted by clown (84.204.xx.xx) on Fri 12 Oct 2007 at 12:38
touch /srv/backup/mysqldump_all.sql
chmod 600 /srv/backup/mysqldump_all.sql
eval $(/usr/local/bin/parse-ini /etc/mysql/debian.cnf client)
mysqldump -A -u $user -p$password > /srv/backup/mysqldump_all.sql

Sorry for this mistake!

[ Parent ]