Install and Configure Lsyncd


When configured, lsyncd is a replication service that will run on the “master” server to ensure the same data exists on all “slave” servers. The lsyncd process monitors the web content directory and replicates changes every few seconds to clone nodes.

Lsyncd (Live Syncing Daemon) is especially useful to sync data from a secure area to a less secure area. To accomplish this lsync uses rsync to automate the replication of files from one “master” server to one or more “slave” servers. Lsyncd only transfers files that have changed.

Complete the following steps to install and configure Lsyncd.

Pre-reqs

  • 2 or more Linux servers
  • Ability to run the following commands while logged in as the root user, or from a super-user account.
  • Master server public and private IP addresses
  • Clone server public and private IP addresses

Set Up

  • On the master server, run the following:
    ssh-keygen -t rsa

Note: This will create 2 new files in ~/.ssh/ id_rsa and id_rsa.pub.

  • SFTP into your server, copy id_rsa.pub to into the folder /root/.ssh/ of each of your slave servers, and change the name of each copy to master.pub.
  • On your slave server run the following command:
   cat ~/.ssh/master.pub >> ~/.ssh/authorized_keys
  • From the master server, ssh into each slave server and choose Yes to accept the generated key. Going forward, this will allow you to sign in without a password.

Install Package Dependencies on the Master Server

For CentOS

  • Run this command to install all needed dependencies:
    yum -y install lua lua-devel pkgconfig gcc asciidoc

For Ubuntu

  • Run this command to install all needed dependencies:
    apt-get update && apt-get install -y lua5.1 liblua5.1-dev pkg-config rsync asciidoc

Install Lsyncd

Important : Be sure to manually check the destination directory on the slave server. If you have NFS or CloudFuse mounts, these could end up wiped by the lsync process.

Note : Lsyncd version 2.0.4 has a known issue with permissions, see the following bug reports. Always make sure you install either 2.0.7 (stable) or 2.1.5+ (latest).

To install the version of lsyncd required for this guide we will need to compile the code from source. If you are unfamiliar with this process simply copy and paste the commands below.

    cd /var/tmp
    wget http://lsyncd.googlecode.com/files/lsyncd-2.1.5.tar.gz
    tar xzvf lsyncd-2.1.5.tar.gz
    cd lsyncd-2.1.5
    ./configure && make && make install

With the completion of the make install command the lsyncd service will be installed on the master server. The compressed lsyncd package does not come with start up scripts so you will need to install your own.

Set Up the Startup Scripts

For Ubuntu:

  • Create the following file at /etc/init/lsyncd.conf
description "lsyncd file syncronizer"
    
    start on (starting network-interface
        or starting network-manager
        or starting networking)
                
        stop on runlevel [!2345]
        
        expect fork
        
        respawn
        respawn limit 10 5
               
        exec /usr/local/bin/lsyncd /etc/lsyncd.lua
  • Then create a symlink to preserve functionality when called as an old style init script.
    ln -s /lib/init/upstart-job /etc/init.d/lsyncd
                
  • If you want to disable lsyncd from starting automatically at bootup (I.E. customer scaled down to one server), you can create a file at /etc/init/lsyncd.override.
    /etc/init/lsyncd.override
    manual

For Centos:

  • Create the following file at /etc/init.d/lsyncd (skip if you did 'yum install lsyncd' above).
#!/bin/bash
    #
    # lsyncd: Starts the lsync Daemon
    #
    # chkconfig: 345 99 90
    # description: Lsyncd uses rsync to synchronize local directories with a remote
    # machine running rsyncd. Lsyncd watches multiple directories
    # trees through inotify. The first step after adding the watches
    # is to, rsync all directories with the remote host, and then sync
    # single file buy collecting the inotify events.
    # processname: lsyncd
    
    . /etc/rc.d/init.d/functions
    
    config="/etc/lsyncd.lua"
    lsyncd="/usr/local/bin/lsyncd"
    lockfile="/var/lock/subsys/lsyncd"
    pidfile="/var/run/lsyncd.pid"
    prog="lsyncd"
    RETVAL=0
    
    start() {
        if [ -f $lockfile ]; then
            echo -n $"$prog is already running: "
            echo
            else
            echo -n $"Starting $prog: "
            daemon $lsyncd -pidfile $pidfile $config
            RETVAL=$?
            echo
            [ $RETVAL = 0 ] && touch $lockfile
            return $RETVAL
        fi
    }
    stop() {
        echo -n $"Stopping $prog: "
        killproc $lsyncd
        RETVAL=$?
        echo
        [ $RETVAL = 0 ] && rm -f $lockfile
        return $RETVAL
    }
    case "$1" in
        start)
            start
            ;;
        stop)
            stop
            ;;
        restart)
            stop
            start
            ;;
        status)
            status $lsyncd
            ;;
        *)
            echo "Usage: lsyncd {start|stop|restart|status}"
            exit 1
        esac
        
        exit $?
  • Configure Lsyncd to start at boot:
chkconfig lsyncd on

Configure Log Rotation

Once lsyncd has an assigned log area, it will create a log event if an error occurs with replication. Log files can become large and unruly when not properly monitored, causing a harddisk to fill up. To ensure this does not happen add lsyncd to logrotate. Create a file at /etc/logrotate.d/lsyncd with the following shell script.

   cat << EOF > /etc/logrotate.d/lsyncd
   /var/log/lsyncd/*log {
        missingok
        notifempty
        sharedscripts
        postrotate
        if [ -f /var/lock/lsyncd ]; then
            /sbin/service lsyncd restart > /dev/null 2>/dev/null || true
        fi
        endscript
    }
    EOF

With this script enabled, the logs will be rotated instead of accumulating until they are manually deleted.

Create the config file

  • Create a file at /etc/lsyncd.lua and insert the following data with a text editor.

Note : Replace the target parameter with the private IP address of the clone server. This is found in the Rackspace control panel.

settings {
    logfile = "/var/log/lsyncd/lsyncd.log",
    statusFile = "/var/log/lsyncd/lsyncd-status.log", 
    statusInterval = 20
}
sync {
    default.rsync,
    source="/var/www/", 
    target="your.clone.private.ip.address:/var/www/",
    rsync = {
        compress = true,
        acls = true,
        verbose = true,
        rsh = "/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no" }
}

Take special note of the block of code beginning with sync. If you require more web servers, this block of code will need to be replicated with the IP address of each additional clone server. This is discussed further in the Continued Scaling section.

  • With configuration complete start the lsyncd service.
    service lsyncd start

Test the Configuraton

  • Before continuing, test the configuration. Create a file in the /var/www/ directory on the master server. This can be done with the touch command.
    touch /var/www/dummy_file
  • If everything is working correctly the file will be replicated to the /var/www/ directory on the clone server. You can test by SSHing into the clone server and using the ls command.
    ls /var/www


Was this content helpful?




© 2014 Rackspace US, Inc.

Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License


See license specifics and DISCLAIMER