Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 20 Aug 2013 08:32:48 +0100
From:      krad <kraduk@gmail.com>
To:        aurfalien <aurfalien@gmail.com>
Cc:        FreeBSD Questions <freebsd-questions@freebsd.org>
Subject:   Re: copying milllions of small files and millions of dirs
Message-ID:  <CALfReyeWxHjmqXhWiK4jbCvh3MktqKqnTBQjYgC0wDTgBcK5jg@mail.gmail.com>
In-Reply-To: <B629E9F8-C01A-4D09-8054-C63F69846F5C@gmail.com>
References:  <7E7AEB5A-7102-424E-8B1E-A33E0A2C8B2C@gmail.com> <20130816064612.GH1190@petole.demisel.net> <1376934082.25499.11612497.1C73C726@webmail.messagingengine.com> <B629E9F8-C01A-4D09-8054-C63F69846F5C@gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
When i migrated a large mailspool in maildir format from the old nfs server
to the new one in a previous job, I 1st generated a list of the top level
maildirs. I then generated the rsync commands + plus a few other bits and
pieces for each maildir to make a single transaction like function. I then
pumped all this auto generated scripts into xjobs and ran them in parallel.
This vastly speeded up the process as sequentially running the tree was far
to slow. THis was for about 15 million maildirs in a hashed structure btw
so a fair amount of files.


eg

find /maildir -type d -maxdepth 4 | while read d
do
r=$(($RANDOM*$RANDOM))
echo rsync -a $d/ /newpath/$d/ > /tmp/scripts/$r
echo some other stuff >> /tmp/scripts/$r
done

ls /tmp/scripts/| while read f
echo /tmp/scripts/$f
done | xjobs -j 20










On 19 August 2013 18:52, aurfalien <aurfalien@gmail.com> wrote:

>
> On Aug 19, 2013, at 10:41 AM, Mark Felder wrote:
>
> > On Fri, Aug 16, 2013, at 1:46, Nicolas KOWALSKI wrote:
> >> On Thu, Aug 15, 2013 at 11:13:25AM -0700, aurfalien wrote:
> >>> Is there a faster way to copy files over NFS?
> >>
> >> I would use find+cpio. This handles hard links, permissions, and in case
> >> of later runs, will not copy files if they already exist on the
> >> destination.
> >>
> >> # cd /source/dir
> >> # find . | cpio -pvdm /destination/dir
> >>
> >
> > I always found sysutils/cpdup to be faster than rsync.
>
> Ah, bookmarking this one.
>
> Many thanks.
>
> - aurf
> _______________________________________________
> freebsd-questions@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "
> freebsd-questions-unsubscribe@freebsd.org"
>



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CALfReyeWxHjmqXhWiK4jbCvh3MktqKqnTBQjYgC0wDTgBcK5jg>