Weblog entry #16 for kumanna

A backup workflow with bup
Posted by kumanna on Wed 3 Oct 2012 at 02:09
Tags: none.

It's important to back things up. You should also make your life easier by using some good software. I use obnam for some backups, but I wanted to try out bup for a new backup workflow I needed. I will describe the workflow here.

The situation: I have a laptop with things to be backed up. I also have a Raspberry Pi which I use as a network attached storage (it has a 1 TB external drive attached to it) and runs Raspbian Wheezy on it. Naturally, you can do this on any NAS running some GNU/Linux variant. In addition, I also have an external portable USB hard disk where another backup copy will be stored. How do I automate this?

Here is how I went about this process. First, for the Raspberry Pi:

  1. First, identify the directory or directories to be backed up. To keep this simple, the directory I am backing up is /home/kumar/Work.
  2. On the Raspberry Pi, install bup. If you want to build it from the git clone, you would need python-dev, and pandoc for the documentation.
    sudo aptitude install pandoc python-dev
    git clone git://github.com/apenwarr/bup.git
    cd bup;make;sudo make install
    
  3. Initialize the backup directory on the Raspberry Pi:
    BUP_DIR=/home/kumar/BACKUPS bup init
    BACKUPS is the place where bup is asked to store the backups (much like it's analogue of .git). You can store all your backups within this directory, and you can still classify backup sets within this directory with different names (much like git branches).
  4. On the local machine, initialize the backup directory.
    bup index -uv /home/kumar/Work
  5. Back it up. This will take time the first time, if the Work directory has several contents within it.
    bup save -r kumar@<raspberry_pi_ip_address>:BACKUPS -n workbackup /home/kumar/Work
    
    workbackup is the name of the backup set. This can be paperbackup or databackup or notesbackup so that several classes can be backed up within the BACKUPS directory.
  6. Now, you have backed things up. Make changes in the Work directory. For instance, add/edit/remove/move some files.
  7. Rerun the index and save commands from steps 4 and 5 above. Enjoy the speedup and ability to rewind to the previous state.
  8. Test restoration of ONE directory of backup (just to save time) on Pi itself directly on the Pi:
    bup -d /home/kumar/BACKUPS/ restore -C /tmp/ /workbackup/latest/home/kumar/Work/offset-coupling-params
    
    Then scp it back if you need it.
  9. Alternative: Use fuse. On the Pi:
    sudo aptitude install python-fuse
    sudo adduser kumar fuse
    mkdir Mount
    bup -d /home/kumar/BACKUPS fuse Mount
    Then cd into Mount and cp/scp the correct files wherever.
    fusermount -u Mount
    
  10. Once happy, write the bup index/bup save commands into a script and run it periodically to back up (check this at the end of the post). Have fun.

Next, to prepare the external hard disk, I went with the old-style autofs automounting solution:

  1. To automount the disk, in /etc/auto.master, add
    /auto			/etc/auto.misc	--timeout=60
    
  2. In /etc/auto.misc add:
    backupdrive	-fstype=btrfs	:UUID="7927a7ab-8c3f-4198-bd73-d3cc519a9ac3"
    
    fstype may need changing for your file system, and UUID can be found using blkid on the exact partition of the hard disk you wish to back up to.
  3. Initialize the backup directory:
    sudo mkdir /auto/backupdrive/BACKUPS
    sudo chown kumar.kumar /auto/backupdrive/BACKUPS
    bup -d /auto/backupdrive/BACKUPS/ init
    
  4. Index and back up:
    bup -d /auto/backupdrive/BACKUPS/ index -uv /home/kumar/Work
    bup -d /auto/backupdrive/BACKUPS/ save -n workbackup /home/kumar/Work
    
  5. Make changes in the Work directory. Add/edit/remove/move files.
  6. Rerun index and save commands from step 5. Enjoy the speed.
  7. Test restoration of ONE directory of backup from removable disk:
    bup -d /auto/backupdrive/BACKUPS/ restore -C /tmp/ /workbackup/latest/home/kumar/Work/offset-coupling-params
    
  8. Once happy, write the bup index/bup save commands into a script and run it periodically to back up. Have fun.

Finally, a quick script to automate all of the above.

#!/bin/sh

bup index -u /home/kumar/Work
bup save -r kumar@<raspberry_pi_ip_address>:BACKUPS -n workbackup /home/kumar/Work
if ls /auto/backupdrive/ 2> /dev/null;then
    echo Backing up to the removable disk...
    bup -d /auto/backupdrive/BACKUPS/ index -u /home/kumar/Work
    bup -d /auto/backupdrive/BACKUPS/ save -n workbackup /home/kumar/Work
    echo Backed up to the removable disk! Yay!
else
    echo WARNING: Not backing up to the removable disk
fi

What this script does it, it backs up to the NAS, and then checks if the external disk is connected. If the disk is mounted, then it backs up to that disk as well. It might be a good idea to cron or anacron this script, or have someone remind you to run this periodically. The only limitation I have is that you can't use bup restore to restore from a remote repository yet, but that isn't a huge issue for me now. In addition, metadata is not stored by the bup save/restore commands. You would have to use split and join with tar to get that. However, bup is fast moving, so I'd expect these features soon. Till then, this workflow works all right for data only backups.

Although this is just a mental dump of the procedure I made for myself, suggestions for improvement are welcome. Thanks.

 

Comments on this Entry

Posted by Anonymous (91.92.xx.xx) on Wed 3 Oct 2012 at 04:50
according to bup docks:

'bup save' and 'bup restore' don't know about file metadata.
That means bup aren't saving file attributes, mtimes, ownership, hard links, MacOS resource forks, etc.

call to metastore must be included in script (packages.debian.org/metastore)

another stopper is 'bup currently has no features that prune away old backups'

seem unsuitable for backups :(

Djadala

[ Parent | Reply to this comment ]

Posted by kumanna (72.179.xx.xx) on Wed 3 Oct 2012 at 05:30
[ Send Message | View Weblogs ]
I am aware of these limitations. These don't really affect me now, since all backups are my own and I care more about the actual contents, as opposed to the metadata.

Kumar

[ Parent | Reply to this comment ]

Posted by Anonymous (88.192.xx.xx) on Wed 10 Oct 2012 at 15:27
First I also thought about the inability to prune old backups as a stopper.

Now I think bup may well suit for 'incremental only' backups in Amanda's terminology. 'Incremental only' has uses for mainly static file systems like video/photo/music library or archive folder, from where you rarely remove anything and mainly add. File system level metadata is more scarse in those file systems usually. So one just postpone the 'second initial copy' indefinitely until events like too big backup repository storage starts to affect, the bup begins to grawl due to some load issue or the likes. After which one just archives the repo, makes a new, determines how long to keep the archived backup repo and so on.

But I still think that the bup's limitation is real for file systems with higher rates of change in data like home folders and the like. Of course, what I can tell, usually many, from individuals to organizations, settle for one backup tool, which is fitted to suit everything and everybody from high change rate file systems to mainly static ones.

[ Parent | Reply to this comment ]

Posted by Anonymous (128.240.xx.xx) on Thu 4 Oct 2012 at 14:25
Hi, just to let you know - bup is packaged in Debian and has been for a while!

[ Parent | Reply to this comment ]

Posted by Anonymous (91.232.xx.xx) on Tue 23 Apr 2013 at 22:01
btxblww

[ Parent | Reply to this comment ]