There is a new version of this tutorial available for Debian 10 (Buster).

Setting up Unison File Synchronization between two Servers on Debian 8 (Jessie)

This tutorial shows how to set up file synchronization between two Debian 8 servers with Unison. Unison is a file-synchronization tool similar to rsync, the big difference is that it tracks/synchronizes changes in both directions, i.e., files changed on server1 will be replicated to server2 and vice versa.

 

1 Preliminary Note

In this tutorial I will use the following two Debian servers:

  • server1.example.com with the IP address 192.168.1.101
  • server2.example.com with the IP address 192.168.1.102

I want to synchronize the directory /var/www between the two servers. I will run Unison as the root user in this tutorial so that Unison has sufficient permissions to synchronize user and group permissions.

All commands in this tutorial are run as root user. Login to both servers on the shell as root and start with step 2 "Installing Unison".

 

2 Installing Unison

server1/server2:

Unison has to be installed on server1 and server2; since we connect from server1 to server2 using SSH, we also need the SSH packages and I'll install the nano editor for file editing on the shell. This can be achieved as follows:

apt-get -y install unison openssh-server ssh nano

 

3 Creating A Private/Public Key Pair On server1

server1:

Now we create a private/public key pair on server1.example.com:

ssh-keygen -t dsa

[email protected]:~# ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/root/.ssh/id_dsa):
 <-- ENTER
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
 <-- ENTER
Enter same passphrase again: <-- ENTER
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.
The key fingerprint is:
ba:82:e1:a1:42:9b:d4:c8:99:c8:bd:8b:7d:4d:d4:66 [email protected]
The key's randomart image is:
+---[DSA 1024]----+
| |
| |
| . |
| . E |
|+ * . S |
|.Ooo o |
|ooo+. + |
|oo=... o |
|.. oo.. |
+-----------------+
[email protected]:~#

It is important that you do not enter a passphrase otherwise the mirroring will not work without human interaction so simply hit ENTER!

Next, we copy our public key to server2.example.com:

ssh-copy-id -i $HOME/.ssh/id_dsa.pub [email protected]
ssh-copy-id -i $HOME/.ssh/id_dsa.pub [email protected]
The authenticity of host '192.168.1.102 (192.168.1.102)' can't be established.
ECDSA key fingerprint is 51:7f:b4:ed:bd:e3:fc:16:2f:55:5c:e1:2c:d7:3d:a9.
Are you sure you want to continue connecting (yes/no)? <-- yes (you will see this only if this is the first time you connect to server2)
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: <-- server2 root password
Number of key(s) added: 1
Now try logging into the machine, with: "ssh '[email protected]'"
and check to make sure that only the key(s) you wanted were added.

Now check on server2 if server1's public key has correctly been transferred:

server2:

cat $HOME/.ssh/authorized_keys
[email protected]:/home/administrator# cat $HOME/.ssh/authorized_keys
ssh-dss AAAAB3NzaC1kc3MAAACBAKHLdAztIr8muZIlQYuE/4f75kmgTwWqJRZJ1dTqHDnHWsy48emDU8v85hxAPg43k9aF7/zAwpA0MNNNk5T9Tx/DyUkK/KcyVP2f4p8tvovrkUvoxsZACkTUmFqKdq2x6/AGfjsCRmkpLhZuad7r5rKEXHRh8KYGHqD1Id8wcpy5AAAAFQCww3OekKcKMshMAwBK3XQmmYEGUwAAAIEAgjztlwh8OFYxwQve/RrhI2sceCXwS/yjQyH7q0zdWB9Fr4s/16T2PLBT+7M3vb+JlPDO3JRqgaYbel1kS2F2iKrY0EX0FI3/9fVDfWoz3mhCscPLriqy5AcsHitxQNfiZgA5wDiSjWpk1v+FbIC+VuqbKdQuE4MBKj19N9YALIUAAACABQ4NDsa2UBc8jsxvghjoLhUWF7HChaCksXQcL6i98VNRcemtPC6wpIri75iR4Uhv1666bDOBAdmIBX9Qf7A/+czPKPaj4CGI1hVy1pgYMa3btnEvoSnH/ONtjpOz9q+3up1OOOn+5fud7xjJn+Fq8WoGROgarBpCbQU3w2GUUnM= [email protected]

 

4 Running Unison

server1:

We can now run Unison for the first time to synchronize the /var/www directory on both servers. On server1 run:

unison /var/www ssh://192.168.1.102//var/www

Output will be similar to this one - you might have to answer a few questions as this is the first time Unison is being run:

[email protected]:/var/www# unison /var/www ssh://192.168.1.102//var/www
Contacting server...
Connected [//server1//var/www -> //server2//var/www]
Looking for changes
Warning: No archive files were found for these roots, whose canonical names are:
/var/www
//server2//var/www
This can happen either
because this is the first time you have synchronized these roots,
or because you have upgraded Unison to a new version with a different
archive format.
Update detection may take a while on this run if the replicas are
large.
Unison will assume that the 'last synchronized state' of both replicas
was completely empty. This means that any files that are different
will be reported as conflicts, and any files that exist only on one
replica will be judged as new and propagated to the other replica.
If the two replicas are identical, then no changes will be reported.
If you see this message repeatedly, it may be because one of your machines
is getting its address from DHCP, which is causing its host name to change
between synchronizations. See the documentation for the UNISONLOCALHOSTNAME
environment variable for advice on how to correct this.
Donations to the Unison project are gratefully accepted:
http://www.cis.upenn.edu/~bcpierce/unison
Press return to continue.[<spc>] <-- Press Enter
Waiting for changes from server
Reconciling changes
local server2
dir ----> example.com [f] <-- Press Enter
dir ----> example.de [f] <-- Press Enter
Proceed with propagating updates? [] <-- Enter "y"
Propagating updates

UNISON 2.40.102 started propagating changes at 10:17:17.94 on 25 Sep 2015
[BGN] Copying example.com from /var/www to //server2//var/www
[BGN] Copying example.de from /var/www to //server2//var/www
Shortcut: copied /var/www/example.de/web/index.html from local file /var/www/.unison.example.com.d3783bddaaf59b9ba4d2ed0433f9db63.unison.tmp/web/index.html
[END] Copying example.de
[END] Copying example.com
UNISON 2.40.102 finished propagating changes at 10:17:17.94 on 25 Sep 2015

Saving synchronizer state
Synchronization complete at 10:17:17 (2 items transferred, 0 skipped, 0 failed)

Check the /var/www directory on server1 and server2 now, and you should find that they are in sync now.

Of course, we don't want to run Unison interactively, therefore we can create a preferences file (/root/.unison/default.prf) that contains all settings that we otherwise would have to specify on the command line:

nano /root/.unison/default.prf
# Roots of the synchronization
root = /var/www
root = ssh://192.168.1.102//var/www

# Paths to synchronize
#path = current
#path = common
#path = .netscape/bookmarks.html

# Some regexps specifying names and paths to ignore
#ignore = Path stats    ## ignores /var/www/stats
#ignore = Path stats/*  ## ignores /var/www/stats/*
#ignore = Path */stats  ## ignores /var/www/somedir/stats, but not /var/www/a/b/c/stats
#ignore = Name *stats   ## ignores all files/directories that end with "stats"
#ignore = Name stats*   ## ignores all files/directories that begin with "stats"
#ignore = Name *.tmp    ## ignores all files with the extension .tmp

#          When set to true, this flag causes the user interface to skip
#          asking for confirmations on non-conflicting changes. (More
#          precisely, when the user interface is done setting the
#          propagation direction for one entry and is about to move to the
#          next, it will skip over all non-conflicting entries and go
#          directly to the next conflict.)
auto=true

#          When this is set to true, the user interface will ask no
#          questions at all. Non-conflicting changes will be propagated;
#          conflicts will be skipped.
batch=true

#          !When this is set to true, Unison will request an extra
#          confirmation if it appears that the entire replica has been
#          deleted, before propagating the change. If the batch flag is
#          also set, synchronization will be aborted. When the path
#          preference is used, the same confirmation will be requested for
#          top-level paths. (At the moment, this flag only affects the
#          text user interface.) See also the mountpoint preference.
confirmbigdel=true

#          When this preference is set to true, Unison will use the
#          modification time and length of a file as a `pseudo inode
#          number' when scanning replicas for updates, instead of reading
#          the full contents of every file. Under Windows, this may cause
#          Unison to miss propagating an update if the modification time
#          and length of the file are both unchanged by the update.
#          However, Unison will never overwrite such an update with a
#          change from the other replica, since it always does a safe
#          check for updates just before propagating a change. Thus, it is
#          reasonable to use this switch under Windows most of the time
#          and occasionally run Unison once with fastcheck set to false,
#          if you are worried that Unison may have overlooked an update.
#          The default value of the preference is auto, which causes
#          Unison to use fast checking on Unix replicas (where it is safe)
#          and slow checking on Windows replicas. For backward
#          compatibility, yes, no, and default can be used in place of
#          true, false, and auto. See the section "Fast Checking" for more
#          information.
fastcheck=true

#          When this flag is set to true, the group attributes of the
#          files are synchronized. Whether the group names or the group
#          identifiers are synchronizeddepends on the preference numerids.
group=true

#          When this flag is set to true, the owner attributes of the
#          files are synchronized. Whether the owner names or the owner
#          identifiers are synchronizeddepends on the preference
#          extttnumerids.
owner=true

#          Including the preference -prefer root causes Unison always to
#          resolve conflicts in favor of root, rather than asking for
#          guidance from the user. (The syntax of root is the same as for
#          the root preference, plus the special values newer and older.)
#          This preference is overridden by the preferpartial preference.
#          This preference should be used only if you are sure you know
#          what you are doing!
prefer=newer

#          When this preference is set to true, the textual user interface
#          will print nothing at all, except in the case of errors.
#          Setting silent to true automatically sets the batch preference
#          to true.
silent=true

#          When this flag is set to true, file modification times (but not
#          directory modtimes) are propagated.
times=true

The comments should make the file self-explaining, except for the path directives. If you specify no path directives, then the directories in the root directives will be synchronized. If you specify path directives, then the paths are relative to the root path (e.g. root = /var/www and path = current translates to /var/www/current), and only these subdirectories will be synchronized, not the whole directory specified in the root directive.

You can find out more about the available options by taking a look at Unison's man page:

man unison

Now that we have put all settings in a preferences file (especially the root (and optionally the path) directives), we can run Unison without any arguments:

unison

 

5 Creating a Cron Job for Unison

server1:

We want to automate synchronization, that is why we create a cron job for it on server1.example.com:

crontab -e
*/5 * * * * /usr/bin/unison &> /dev/null

This would run Unison every 5 minutes; adjust it to your needs (see

man 5 crontab

). I use the full path to unison here (/usr/bin/unison) just to go sure that cron knows where to find unison. Your unison location might differ. Run

which unison

to find out where yours is.

6 Test Unison

Now I will test the 2 way synchronisation of Unison to see if the setup is fully working.

Run the following command on server1 to create a test file with the content "Test 1":

Server1

echo "Test 1" > /var/www/test.txt

Now wait at least 5 Minutes (as we created a cronjob that runs once every 5 minutes). Then run on server2:

cat /var/www/test.txt

to show the content of the file test.txt on the screen. The Output should be similar to this screenshot.

Output of the cat command on Server 2.

Now run this command on server2 which updates the content of our test file to "Test 2":

Server2

echo "Test 2" > /var/www/test.txt

And wait at least 5 minutes. Then run the cat command on server1:

Server1

cat /var/www/test.txt

The output should be as shown on the screenshot.

Output of the cat command on Server 1.

 

Share this page:

10 Comment(s)

Add comment

Please register in our forum first to comment.

Comments

By: Luis Giovanny

Thanks you !! , Great post. 

I am set up load balancing with nginx LB (node 1)  and 2 node (Nginx web servers, PHP-FPM, Drupal 7, mysql cluster).

unison sync for Drupal's file beetwn two web servers. 

Thanks.

 

By: Rob

Does not work with "The Perfect Server - Debian 8 Jessie (Apache2, BIND, Dovecot, ISPConfig 3)"

By: Kevin Bodson

Hello,

i would like to know something. I just enter in the company and i'm new with unison_sync.

Unison was already set up in the company but they made some changes and now i need to update the unison sync for some folder.

 

So here is my problem :

I just would like to sync 2 folder like var/www/pdf on server 1 and var/www/pdf on server 2.

I don't know why but when i put a file e.g. (test1)  on the server 1 and then use unison_sync, he is working fine and put test1 into the /var/www/pdf on the server 2.

But if i put a file on the /var/www/pdf on the server 2, when i use unison, he doesn't copy my file from the server 2 to the server 1 ... he DELETE it!

 

Why ? there is a special parameter to do this ?

Into a configuration file, i have this line : # Use this command for displaying diffsdiff = diff -y -W 79 --suppress-common-lines

What are the parameters -y and -W and --suppress-common-lines ?

Thanks in advance.

Regards,

Kevin.

By: SamTzu

Almost perfect :)

You missed 2 things:

1) It should be rsa instead of dsa

2) /etc/init.d/cron restart

By: Fernando Garcia

Thanks! it works.

However, a few things.

1.      The key need to be RSA (at least in Ubuntu Server, which is what I was using).

2.      If the ssh port is different, one will need to use the -p as ssh-copy-id -i $HOME/.ssh/id_dsa.pub [email protected] -p 2222

3.      Also, add the port in: ssh-copy-id -i $HOME/.ssh/id_dsa.pub [email protected]:2222

 

4.      If you have a firewall or Fail2ban installed, make sure this services are not blocking your SSH transactions.

By: vlad

And what is the difference between unison and rsync?!

I preffer rsync:)

 

By: till

Rsync is a nice program and I use it regularly, but unison is different and has features that rsync does not provide. Unsion is a two way sync while rsync can only sync in one direction, so unison is used when you have situtaions where files get changed or added on both nodes.

By: Eric

Hello, I use my Raspberry Pi (Raspbian) as a NAS and want to synchronize it with my Linux Mint laptop. It works perfectly with unison and a RSA key when I call it through the GUI or the command editor. But it always fails when called by CRON : Fatal error: Lost connection with the server. Got any clue about it ?

By: Szczepan Ho?yszewski

Is it possible to configure unison to be boldly noninteractive for all current and future roots? I really want it to use it in a scripted scenario without having to provision unison configuration for each and every website directory!

By: till

Yes, that's what the above guide describes. See the second part of chapter 4 and chapter 5.