Date: Fri, 28 Jul 2006 17:50:26 +0100 From: Alex Zbyslaw <xfb52@dial.pipex.com> To: Ensel Sharon <user@dhp.com> Cc: freebsd-questions@freebsd.org Subject: Re: long, large rsync slows down over time ? Message-ID: <44CA4052.8060806@dial.pipex.com> In-Reply-To: <Pine.LNX.4.21.0607281209110.12027-100000@shell.dhp.com> References: <Pine.LNX.4.21.0607281209110.12027-100000@shell.dhp.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Ensel Sharon wrote: >I am moving a large number of files (about 40 GB, comprising almost 3.5 >million files) from one machine to another using rsync. > >I am noticing that the longer this rsync process runs, the slower it seems >to go, and the more CPU time it is taking on the destination (FreeBSD >6.1). > >Is there a reason for this ? Is there some aspect of rsync (or >FreeBSD) that just starts to bog down after transferring 1-2 million >files? > >When the job started, load on the destination was between .2 and .8, and >now, four days later, it is up to 4.0+ ... and as I say, although I have >not measured, throughput does seem to be slowing down. > > Are you using rsync with ssh? If so try openssh-portable port with hpn options enabled. If the machines are local you can even turn off ssh data encryption with -z which will make rsync go faster. If machines are local you could also try using rsh, or take the risk for remote machines. Though if it's been running for days, they don't sound local :-) --Alex
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?44CA4052.8060806>