Skip site navigation (1)Skip section navigation (2)
Date:      Sun, 11 Oct 2009 17:07:08 -0400
From:      Mike Tancsa <mike@sentex.net>
To:        net.help@m2k.com.tw, "freebsd-performance" <freebsd-performance@freebsd.org>
Subject:   Re: Migrate large amount of small files
Message-ID:  <200910112106.n9BL6p7G074295@lava.sentex.ca>
In-Reply-To: <1255189612.42461.net.help@m2k.com.tw>
References:  <1255189612.42461.net.help@m2k.com.tw>

next in thread | previous in thread | raw e-mail | index | archive | help
At 11:46 AM 10/10/2009, Ckcheng wrote:
>Hi all,
>Currently, I have a directory with over 5M small files (1~32K). Now,
>I want to transfer this directory to another machine and found
>that it's extremely slow and painful process. I tried the following
>method:

It might help if you mount all -onoatime as well as bump up 
vfs.ufs.dirhash_maxmem to 4x the default size

         ---Mike

>1. rsync
>2. tar via ssh
>3. tar via nc
>(all take hours and hours to finish)
>
>None of them is able to give me a reasonable migration time. So, I'm
>here for asking help. Any suggestion is extremely welcomed. Thank you.
>
>Btw, here is brief information of my server. (both machines are the same)
>
>OS: FreeBSD 6.4-Stable 64Bit
>CPU: 2 x Xeon L5420 2.50GHz
>RAM: 2 x 2G ECC DDR2-667 (full buffered)
>DISK: Seagate Barracuda ES 16MB (SATA 300)
>Network: 1Gbps (Broadcom BCM5708)
>Filesystem: UFS2
>
>Regards,
>_______________________________________________
>freebsd-performance@freebsd.org mailing list
>http://lists.freebsd.org/mailman/listinfo/freebsd-performance
>To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

--------------------------------------------------------------------
Mike Tancsa,                                      tel +1 519 651 3400
Sentex Communications,                            mike@sentex.net
Providing Internet since 1994                    www.sentex.net
Cambridge, Ontario Canada                         www.sentex.net/mike




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200910112106.n9BL6p7G074295>