Limiting network file transfer throughput with rsync

Posted on by

Warning: count(): Parameter must be an array or an object that implements Countable in /nfs/c01/h05/mnt/6385/domains/uhleeka.com/html/blog/wp-content/plugins/uhleeka-codebox/uhleeka-codebox.php on line 65

To transfer a large amount of data (200GB) across a 100mbps network without saturating the connection, I used rsync with the –bwlimit=KBps flag.

Transferring from Windows to Ubuntu 9.04, I first mounted the windows share:

sudo mkdir /mnt/winshare
sudo mount -t smbfs -o username=user,password=pass //winserver/share /mnt/winshare

Next, run rsync with a specified bwlimit. Optionally, unmount the share

sudo rsync -vrR --delete --delete-excluded --bwlimit=6144 /mnt/winshare/* /destination/
sudo umount /mnt/winshare


Note that the bwlimit is in kilobytes not kilobits — 6144 KB ~= 48 Mbps. Also note that the bwlimit is limiting the I/O bandwidth…whatever that means.

    --bwlimit=KBPS          limit I/O bandwidth; KBytes per second

NTop (http://www.ntop.org) is great for tracking bandwidth usage:

sudo apt-get update
sudo apt-get install ntop

Set the administrative account password for ntop to run under:

sudo ntop -A

Start the service:

sudo /etc/init.d/ntop start

Browse the reports on the default port, 3000. http://yourserver:3000/

2 Comments

  1. I think you made a mistake here… unless both my math (admittedly horrible) rsync runs are fooling me too. From above:

    "Note that the bwlimit is in kilobytes not kilobits — 6144 KB ~= 48 Mbps. Also note that the bwlimit is limiting the I/O bandwidth…whatever that means."

    6144 is ~= 6mbps not 48.

    I'm trying to use no more than 30% 100Mb connection so I'm using

    –bwlimit=30000 (thirty thousand) and the end result is showing around 10Mb/s for a few hundred Mb.

    • @ezekieldas: 8 bits in a byte, so 6144 kiloBYTES ~= 6 megaBYTES ~= 48 megaBITS (bits per second is usually measured in round numbers; 1 kilobit is 1000 bits. 1 kilobyte is 1024 bytes, so there is often a slight ambiguity when converting bits to bytes.)

      So, if you are trying to limit your sync to 30 megabits per second, your number should be roughly (30 / 8) * 1024 = 3840 kilobytes.

      Unless my math is bad…

Comments are closed.