From owner-freebsd-questions@FreeBSD.ORG  Fri Jul 28 16:50:29 2006
Return-Path: <owner-freebsd-questions@FreeBSD.ORG>
X-Original-To: freebsd-questions@freebsd.org
Delivered-To: freebsd-questions@freebsd.org
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id E84EF16A4E0
	for <freebsd-questions@freebsd.org>;
	Fri, 28 Jul 2006 16:50:29 +0000 (UTC)
	(envelope-from xfb52@dial.pipex.com)
Received: from smtp-out3.blueyonder.co.uk (smtp-out3.blueyonder.co.uk
	[195.188.213.6])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 74BC443D49
	for <freebsd-questions@freebsd.org>;
	Fri, 28 Jul 2006 16:50:29 +0000 (GMT)
	(envelope-from xfb52@dial.pipex.com)
Received: from [172.23.170.147] (helo=anti-virus03-10)
	by smtp-out3.blueyonder.co.uk with smtp (Exim 4.52)
	id 1G6VXg-0007Fk-8V; Fri, 28 Jul 2006 17:50:28 +0100
Received: from [82.41.35.166] (helo=[192.168.0.2])
	by asmtp-out1.blueyonder.co.uk with esmtp (Exim 4.52)
	id 1G6VXf-0003fi-Dz; Fri, 28 Jul 2006 17:50:27 +0100
Message-ID: <44CA4052.8060806@dial.pipex.com>
Date: Fri, 28 Jul 2006 17:50:26 +0100
From: Alex Zbyslaw <xfb52@dial.pipex.com>
User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-GB; rv:1.7.13) Gecko/20060515
X-Accept-Language: en
MIME-Version: 1.0
To: Ensel Sharon <user@dhp.com>
References: <Pine.LNX.4.21.0607281209110.12027-100000@shell.dhp.com>
In-Reply-To: <Pine.LNX.4.21.0607281209110.12027-100000@shell.dhp.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: freebsd-questions@freebsd.org
Subject: Re: long, large rsync slows down over time ?
X-BeenThere: freebsd-questions@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: User questions <freebsd-questions.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-questions>, 
	<mailto:freebsd-questions-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-questions>
List-Post: <mailto:freebsd-questions@freebsd.org>
List-Help: <mailto:freebsd-questions-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-questions>, 
	<mailto:freebsd-questions-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 28 Jul 2006 16:50:30 -0000

Ensel Sharon wrote:

>I am moving a large number of files (about 40 GB, comprising almost 3.5
>million files) from one machine to another using rsync.
>
>I am noticing that the longer this rsync process runs, the slower it seems
>to go, and the more CPU time it is taking on the destination (FreeBSD
>6.1).
>
>Is there a reason for this ?  Is there some aspect of rsync (or
>FreeBSD) that just starts to bog down after transferring 1-2 million
>files?
>
>When the job started, load on the destination was between .2 and .8, and
>now, four days later, it is up to 4.0+  ...  and as I say, although I have
>not measured, throughput does seem to be slowing down.
>  
>
Are you using rsync with ssh?  If so try openssh-portable port with hpn 
options enabled.  If the machines are local you can even turn off ssh 
data encryption with -z which will make rsync go faster.

If machines are local you could also try using rsh, or take the risk for 
remote machines.

Though if it's been running for days, they don't sound local :-)

--Alex