This site is now 100% read-only, and retired.

Directory/File synchronization across systems

Posted by lloyd on Thu 21 Apr 2011 at 16:26

Tags: ,

Both Debian Lenny and the system I've been running it on are now long in the tooth. So I've just brought up a Debian Squeeze-based system on a new and more capable box. I'd like to keep files in sync across the two hosts and would look for suggestions on doing that.



For awhile, at least, I'd like to keep the home directories and files in sync across the two systems. E.g., if I edit or create a file on one, I'd like to sync it over to the other and vice-versa. I'm looking for two wins here: 1) home dir and file consistency while I wring out the new machine; another layer of back up for my home dir and files until I can get a more permanent back-up regime on the new machine.

It seems I have a number of possible options:

rsync - http://en.wikipedia.org/wiki/Rsync
Unison - http://www.cis.upenn.edu/~bcpierce/unison/
git sync - http://code.google.com/p/git-sync/
persy - http://persy.digitalkultur.net/

I'd be much grateful for user experiences and recommendations.

Many thanks and best wishes,

LRP

 

 


Re: Directory/File synchronization across systems
Posted by Anonymous (212.183.xx.xx) on Thu 21 Apr 2011 at 16:52
i would suggest that you should mount home on the new box on the old box over NFS and take a snapshot at appropriate intervals

[ Parent ]

Re: Directory/File synchronization across systems
Posted by Anonymous (217.87.xx.xx) on Thu 21 Apr 2011 at 17:35
I can recommend unison. I sync a huge part of my home directory between 4 different machines (3 Workstations, 1 notebook) several times a day, usually four times (home ws -> notebook, notebook -> office ws, office ws -> notebook, notebook -> home ws). It has been very handy for some years now. The main advantage over rsync is that it can deal with changes in a reasonable way. If you change a file on one side, it detects on which machine the file got changed last and transfers the file to the other one. If there are changes on both sides, it will not do anything until the user tells in which direction to sync.
The graphical user interface is good. The speed is not as good as rsync's though.

[ Parent ]

Re: Directory/File synchronization across systems
Posted by Anonymous (212.202.xx.xx) on Fri 22 Apr 2011 at 22:58
I would like to second this. Unison is a mature piece of software and serves me well since years. As long as you don't get conflicts, it is a real fire-and-forget solution for bidirectional synchronization. If you need to solve conflicts, the GUI version unison-gtk comes in handy.

[ Parent ]

Re: Directory/File synchronization across systems
Posted by Anonymous (198.188.xx.xx) on Thu 28 Apr 2011 at 03:29
Unison is the real deal. Another Debian dev asked about this a few months ago. Just remember that you have to use the same version of Unison on both machines, which is easy because Debian packages the older versions in both Lenny and Squeeze. I've synced my work machine, cloud machine, and home machine for years.

[ Parent ]

Re: Directory/File synchronization across systems
Posted by Anonymous (157.88.xx.xx) on Wed 18 May 2011 at 12:37
I'm also for unison: ssh traffic; different OSes; handle USB stick sync; easy configuration; very solid. No a single file lost in two years.

[ Parent ]

Re: Directory/File synchronization across systems
Posted by Anonymous (91.19.xx.xx) on Thu 21 Apr 2011 at 17:46
Did you already try csync2 ?

Best regards!
morphium

[ Parent ]

Re: Directory/File synchronization across systems
Posted by Anonymous (82.235.xx.xx) on Thu 21 Apr 2011 at 18:34
Minor food for though:

* avoid major boo-boos by synchronizing your boxes synchronized: use some NTP daemon.

* any solution copying date (instead of mounting an unique filesystem through some network) implies some latency (during copy) BUT somewhat backups the data.

[ Parent ]

Re: Directory/File synchronization across systems
Posted by Anonymous (38.96.xx.xx) on Fri 13 Jan 2012 at 15:33
Csync2 and a daemonized inotifywait script are a fantastic solution: -------------- inotifywait -mrq -format '%f' \ -e close_write -e create -e delete -e move /var/www | while read file; do /usr/local/sbin/csync2 -x >/dev/null 2>&1 done --------------

[ Parent ]

Re: Directory/File synchronization across systems
Posted by Anonymous (82.17.xx.xx) on Thu 21 Apr 2011 at 18:38
At work we use HA for main IT servers and rsync for data between master and slave nodes. Syncs are handled by rsync in a cron. DRBD is probably worth looking at but might be overkill for your needs. Wasn't aware that csync2 existed.

We used to use amanda for backups but now use RBME which is a tool that uses rsync, again in a cron to sync to a luks crypt encrypted esata hd.

This works well and after moving to non 5 year old scsi raid array to nice sas disks with lots of cores - pretty fast. See about 50MB/s on syncs which is quite nice.

sno

[ Parent ]

Re: Directory/File synchronization across systems
Posted by Steve (82.41.xx.xx) on Sat 23 Apr 2011 at 10:35
[ View Weblogs ]

DRBD is well known, well proven, and simple to use. But I suspect it gets little use because of the limitation:

  • It may be only mounted by one of the hosts at a time.

So yes, you get an up to date copy on the remote side, but you cannot use it unless you demote the previous master.

Steve

[ Parent ]

Re: Directory/File synchronization across systems
Posted by Anonymous (195.24.xx.xx) on Wed 27 Apr 2011 at 11:40
That's not a required limitation. I run several DRBD systems with both ends mounted and writable. Bring it up primary/primary:-


1: cs:Connected st:Primary/Primary ds:UpToDate/UpToDate C r---
ns:1811561548 nr:99569519 dw:1911131067 dr:25270137 al:3356 bm:5273 lo:0 pe:0 ua:0 ap:0
resync: used:0/61 hits:5394183 misses:5273 starving:0 dirty:0 changed:5273
act_log: used:0/127 hits:1546741620 misses:3356 starving:0 dirty:0 changed:3356

Then use an appropriate filesystem - I've gone with OCFS2. at that point, you can mount on both sides, and write to both sides.

[ Parent ]

Re: Directory/File synchronization across systems
Posted by Anonymous (212.15.xx.xx) on Thu 21 Apr 2011 at 20:31

[ Parent ]

Re: Directory/File synchronization across systems
Posted by Anonymous (70.33.xx.xx) on Thu 21 Apr 2011 at 22:02
I've used unison for this successfully in the past. It handles bidirectional sync well.

[ Parent ]

Re: Directory/File synchronization across systems
Posted by Anonymous (209.160.xx.xx) on Thu 21 Apr 2011 at 22:10
What is wrong with the automounter + nfs?

[ Parent ]

Re: Directory/File synchronization across systems
Posted by Anonymous (128.111.xx.xx) on Fri 22 Apr 2011 at 02:45
Wuala? It's an encrypted distributed storage. Google tech talk: www.wuala.com/en/learn/techtalk
Cons: it's in Java

www.wuala.com/

[ Parent ]

Re: Directory/File synchronization across systems
Posted by allan (173.162.xx.xx) on Fri 22 Apr 2011 at 17:02
git does not track meta data (permissions) and will not take kindly to large files (photos, video). This may not be an issue for you and git sync mayt address that.

If you are interested in sync replication (both hosts online and connected at the same time) then also check out sshfs in addition to nfs.

[ Parent ]

Re: Directory/File synchronization across systems
Posted by wabibito (192.94.xx.xx) on Fri 22 Apr 2011 at 20:15
This may be far off base but I got tired trying to rsync from one system to another.

Here is what I do.

It borrows from the ancient UNIX way of tarring up an entire system and restoring it.

For example, I go to the root of the source server.

Then I issue the command: tar cvpzf [name-of-archive-date-time-initials].tgz --exclude=/proc --exclude=/lost+found --exclude=/name-of-archive-date-time-initials.tgz --exclude=/mnt --exclude=/sys --exclude=/(others-you-deem-apply) /

Copy backup to an external drive or internal spare. Whatever you have available.

On the target server, format the drives in the same as the source.

Copy the name-of-archive-date-time-initials.tgz to the root of the target drive.

Then issue the following command: tar xvpfz name-of-archive-date-time-initials.tgz -C /

This will overlay the old over the new bare install.

Reboot when finished.

You may or may not need to once over the configuration files.

Depending on the size, speed, I/O of your hard drives on both the source and target servers if may or may not take time.

Hope this adds an option to your copying.

[ Parent ]

Re: Directory/File synchronization across systems
Posted by Anonymous (216.93.xx.xx) on Fri 22 Apr 2011 at 22:16
Google continusync, a shell script that uses rsync to do the work. It came recommended by an rsync developer.

[ Parent ]

Re: Directory/File synchronization across systems
Posted by Anonymous (189.252.xx.xx) on Sat 23 Apr 2011 at 18:48
Thanks to all for this incredible information. You've given me much to explore and think about.

All the best,

LRP

[ Parent ]

Re: Directory/File synchronization across systems
Posted by Anonymous (195.139.xx.xx) on Wed 27 Apr 2011 at 16:54
sgi fam is also an option. It can do stuffs based on changes, newfile or delete. Its very handy to just run a few lines of perl, to scp/rsync/whatever based on fileevents.

[ Parent ]

Re: Directory/File synchronization across systems
Posted by AJxn (2001:0xx:0xx:0xxx:0xxx:0xxx:xx) on Fri 27 Jul 2012 at 04:19
[ View Weblogs ]

Now you all should have a look at git-annex, as it looks like it will be a real good piece of software.

Have a look at Git Annex home page

[ Parent ]