Optimizing speed in KVM image synchronization using rsync

An addition to my last post I think it might be useful to explain how to efficiently copy KVM images over network.

In https://gergap.wordpress.com/2013/08/10/rsync-and-sparse-files I explained how to efficiently handle sparse files. But what I didn’t mention is how to get the best transfer rates.

By default rsync uses SSH, which encrypts the whole traffic and gives you a good amount of security. If you have a local GigaBit-LAN the CPU becomes the bottleneck due to the encryption.

Rsync has also a solution for that problem. You can start the rsync daemon on one side and use the rsync tool on the other side with the “rsync://user@host/share” syntax to transfer the data. This way rsync uses its own efficient protocol (port 873) without encryption.
This way I’m able to achieve over 100MB/s transfer rate in my local LAN.

Rsync daemon configuration:
/etc/rsyncd.conf:
max connections = 1
log file = /var/log/rsync.log
timeout = 300

[backups]
path = /root/backups
comment = Backup images
read only = yes
list = yes
uid = root
gid = root
hosts allow = backupsserver
hosts deny = all

On my backupserver I extended my backup script to start the rsync daemon on the other side, start the rsync operation, and then kill the rsync process again.
This way the rsync port (873) is open only for a short time.
The “hosts allow” directive prevents connects from untrusted computers.
In addition it is possible to use “auth users = ” to setup password authentication. See man(1) rsync for details.

A small excerpt of my backup script:

...
echo "starting remote rsync daemon for faster backups" >> /tmp/rsync_script_tmpfile
ssh $HOSTNAME rsync --daemon >> /tmp/rsync_script_tmpfile
echo "backing up $BACKUPDIR" >> /tmp/rsync_script_tmpfile
rsync $OPTIONS rsync://$HOSTNAME/$BACKUPDIR $ARCHIVEROOT/$HOSTNAME >> /tmp/rsync_script_tmpfile
echo "stopping remote rsync daemon" >> /tmp/rsync_script_tmpfile
ssh $HOSTNAME killall rsync >> /tmp/rsync_script_tmpfile
...

Advertisements

0 Responses to “Optimizing speed in KVM image synchronization using rsync”



  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s





%d bloggers like this: