Date: Sat, 21 Jun 2003 11:26:28 -0400 (EDT) From: Andy Dills <andy@xecu.net> To: Chuck Swiger <cswiger@mac.com> Cc: freebsd-isp@freebsd.org Subject: Re: What would be the best way to copy lots of files from oneserver to another. Message-ID: <Pine.BSF.4.44.0306211113110.54979-100000@thunder.xecu.net> In-Reply-To: <3EF33D9D.2070703@mac.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, 20 Jun 2003, Chuck Swiger wrote: > Etienne Ledoux wrote: > > I need to copy lots of directories/files from one server to another. Approx > > 45Gb . These files are mailboxes stored in maildir format. What would be the > > best/safest/quickest way to do this ? > > Putting the disks onto one machine and copying locally is going to be a win > considering the amount of data you have, but if you have to copy over the > network, consider using something like rsync. I love rsync and use it for lots of things. A one time move of 45 GB is not something I would use rsync for. Rsync is good for just what is says it's for: keeping thing in sync. Rsync will build metadata information about the ENTIRE data set before moving a single bit. This is very costly, making rsync's utility plummet as the data set size increases. You'll find it preferrable to write a script to traverse and descend directories, spawning numerous rsync processes for smaller sized directories. Of course, considering that it's a one-time move, I would advise physically moving the drive, to remove any sort of network involvement. I've been in your position before, and I can promise you that even using NFS and spawning multiple moves (mv /var/mail/a* /mnt &; mv /var/mail/b* /mnt &; etc.) will quickly outpace rsync. The number of concurrent processes involved in the move can really only be determined by watching the aggregate traffic to your server. You'll see it level off at a certain point (for me it was X=7), after which additional processes don't yield any more optimization. Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.4.44.0306211113110.54979-100000>