• Sales: 1-800-961-2888
  • Support: 1-800-961-4454

Install and Configure Lsyncd


When configured, lsyncd is a replication service that will run on the "master" server to ensure the same data exists on all "slave" servers. The lsyncd process monitors the web content directory and replicates changes every few seconds to clone nodes.

Lsyncd (Live Syncing Daemon) is especially useful to sync data from a secure area to a less secure area. To accomplish this lsync uses rsync to automate the replication of files from one "master" server to one or more "slave" servers. Lsyncd only transfers files that have changed.

Complete the following steps to install and configure Lsyncd.

Pre-reqs

  • 2 or more Linux servers
  • Ability to run the following commands while logged in as the root user, or from a super-user account.
  • Master server public and private IP addresses
  • Clone server public and private IP addresses

Set Up

  • On the master server, run the following:
    ssh-keygen -t rsa

Note: This will create 2 new files in ~/.ssh/ id_rsa and id_rsa.pub.

  • SFTP into your server, copy id_rsa.pub to into the folder /root/.ssh/ of each of your slave servers, and change the name of each copy to master.pub.
  • On your slave server run the following command:
   cat ~/.ssh/master.pub >> ~/.ssh/authorized_keys
  • From the master server, ssh into each slave server and choose Yes to accept the generated key. Going forward, this will allow you to sign in without a password.

Install Package Dependencies on the Master Server

For CentOS

  • Run this command to install all needed dependencies:
    yum -y install lua lua-devel pkgconfig gcc asciidoc

For Ubuntu

  • Run this command to install all needed dependencies:
    apt-get update && apt-get install -y lua5.1 liblua5.1-dev pkg-config rsync asciidoc

Install Lsyncd

Important : Be sure to manually check the destination directory on the slave server. If you have NFS or CloudFuse mounts, these could end up wiped by the lsync process.

Note : Lsyncd version 2.0.4 has a known issue with permissions, see the following bug reports. Always make sure you install either 2.0.7 (stable) or 2.1.4+ (latest).

To install the version of lsyncd required for this guide we will need to compile the code from source. If you are unfamiliar with this process simply copy and paste the commands below.

    cd /var/tmp
    wget http://lsyncd.googlecode.com/files/lsyncd-2.1.4.tar.gz
    tar xzvf lsyncd-2.1.4.tar.gz
    cd lsyncd-2.1.4
    export CFLAGS="-march=native -O2"
    ./configure && make && make install

With the completion of the make install command the lsyncd service will be installed on the master server. The compressed lsyncd package does not come with start up scripts so you will need to install your own.

Set Up the Startup Scripts

For Ubuntu:

  • Create the following file at /etc/init/lsyncd.conf
description "lsyncd file syncronizer"
    
    start on (starting network-interface
        or starting network-manager
        or starting networking)
                
        stop on runlevel [!2345]
        
        expect fork
        
        respawn
        respawn limit 10 5
               
        exec /usr/local/bin/lsyncd /etc/lsyncd.lua
  • Then create a symlink to preserve functionality when called as an old style init script.
                   ln -s /lib/init/upstart-job /etc/init.d/lsyncd
                
  • If you want to disable lsyncd from starting automatically at bootup (I.E. customer scaled down to one server), you can create a file at /etc/init/lsyncd.override.
    /etc/init/lsyncd.override
    manual

For Centos:

  • Create the following file at /etc/init.d/lsyncd (skip if you did 'yum install lsyncd' above).
#!/bin/bash
    #
    # lsyncd: Starts the lsync Daemon
    #
    # chkconfig: 345 99 90
    # description: Lsyncd uses rsync to synchronize local directories with a remote
    # machine running rsyncd. Lsyncd watches multiple directories
    # trees through inotify. The first step after adding the watches
    # is to, rsync all directories with the remote host, and then sync
    # single file buy collecting the inotify events.
    # processname: lsyncd
    
    . /etc/rc.d/init.d/functions
    
    config="/etc/lsyncd.lua"
    lsyncd="/usr/local/bin/lsyncd"
    lockfile="/var/lock/subsys/lsyncd"
    pidfile="/var/run/lsyncd.pid"
    prog="lsyncd"
    RETVAL=0
    
    start() {
        if [ -f $lockfile ]; then
            echo -n $"$prog is already running: "
            echo
            else
            echo -n $"Starting $prog: "
            daemon $lsyncd -pidfile $pidfile $config
            RETVAL=$?
            echo
            [ $RETVAL = 0 ] && touch $lockfile
            return $RETVAL
        fi
    }
    stop() {
        echo -n $"Stopping $prog: "
        killproc $lsyncd
        RETVAL=$?
        echo
        [ $RETVAL = 0 ] && rm -f $lockfile
        return $RETVAL
    }
    case "$1" in
        start)
            start
            ;;
        stop)
            stop
            ;;
        restart)
            stop
            start
            ;;
        status)
            status $lsyncd
            ;;
        *)
            echo "Usage: lsyncd {start|stop|restart|status}"
            exit 1
        esac
        
        exit $?
  • Configure Lsyncd to start at boot:
chkconfig lsyncd on

Configure Log Rotation

  • Logging will assist in tracking down issues if they arise. Add this file at /etc/logrotate.d/lsyncd:
    /var/log/lsyncd/*log {
        missingok
        notifempty
        sharedscripts
        postrotate
        if [ -f /var/lock/lsyncd ]; then
            /sbin/service lsyncd restart > /dev/null 2>/dev/null || true
        fi
        endscript
    }
  • Once lsyncd has an assigned log area, it will create a log event if an error occurs with replication. Log files can become large and unruly when not properly monitored, causing a harddisk to fill up. To ensure this does not happen add lsyncd to logrotate. Create a file at /etc/logrotate.d/lsyncd with the following shell script.
   cat << EOF > /etc/logrotate.d/lsyncd 
   /var/log/lsyncd/*log {
        missingok
        notifempty
        sharedscripts
        postrotate
        if [ -f /var/lock/lsyncd ]; then
            /sbin/service lsyncd restart > /dev/null 2>/dev/null || true
        fi
        endscript
    }
    EOF

With this script enabled, the logs will rotate saving the harddisk from possible issue.

Create the config file

  • Create a file at /etc/lsyncd.lua and insert the following data with a text editor.

Note : Replace the target parameter with the private IP address of the clone server. This is found in the Rackspace control panel.

   cat << EOF > /etc/lsyncd.lua 
   settings {
        logfile = "/var/log/lsyncd/lsyncd.log",
        statusFile = "/var/log/lsyncd/lsyncd-status.log", 
        statusInterval = 20
    }
sync {
    default.rsync,
    source="/var/www/", 
    target="your.clone.private.ip.address:/var/www/",
    rsync = {
        compress = true,
        acls = true,
        verbose = true,
        rsh = "/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no" }
}
EOF

Take special note of the block of code beginning with sync. If you require more web servers, this block of code will need to be replicated with the IP address of each additional clone server. This is discussed further in the Continued Scaling section.

  • With configuration complete start the lsyncd service.
    start lsyncd

Test the Configuraton

  • Before continuing, test the configuration. Create a file in the /var/www/ directory on the master server. This can be done with the touch command.
    touch /var/www/dummy_file
  • If everything is working correctly the file will be replicated to the /var/www/ directory on the clone server. You can test by SSHing into the clone server and using the ls command.
    ..ls /var/www

 

Continued Scaling

As needed, we edit the /etc/lsyncd.lua configuration file. Inside the configuration file a block of code containing the private IP address of the clone server was present. This block needs to be replicated with the file for each new clone server created. For example, once two clone servers are present the file would look like the below block of code. Insert the private IP addresses of each clone server.

    sudo echo 'sync {
        default.rsync,
        source="/var/www/", 
        target="your.clone2.private.ip.address:/var/www/",
        rsync = {
            compress = true,
            acls = true,
            verbose = true,
            rsh = "/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no"
            }
        }'/etc/lsyncd.lua
  • Reload the configuration file for the changes to take effect.
restart lsyncd






© 2011-2013 Rackspace US, Inc.

Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License


See license specifics and DISCLAIMER