Date: Tue, 20 Aug 2013 08:34:04 +0100 From: krad <kraduk@gmail.com> To: aurfalien <aurfalien@gmail.com> Cc: FreeBSD Questions <freebsd-questions@freebsd.org> Subject: Re: copying milllions of small files and millions of dirs Message-ID: <CALfReydQUEmWdjEmh45ZRit61RJq_00PTtR7ej0LVOvqS=ayNw@mail.gmail.com> In-Reply-To: <CALfReyeWxHjmqXhWiK4jbCvh3MktqKqnTBQjYgC0wDTgBcK5jg@mail.gmail.com> References: <7E7AEB5A-7102-424E-8B1E-A33E0A2C8B2C@gmail.com> <20130816064612.GH1190@petole.demisel.net> <1376934082.25499.11612497.1C73C726@webmail.messagingengine.com> <B629E9F8-C01A-4D09-8054-C63F69846F5C@gmail.com> <CALfReyeWxHjmqXhWiK4jbCvh3MktqKqnTBQjYgC0wDTgBcK5jg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
whops that should have been ls /tmp/scripts/| while read f echo sh /tmp/scripts/$f done | xjobs -j 20 On 20 August 2013 08:32, krad <kraduk@gmail.com> wrote: > When i migrated a large mailspool in maildir format from the old nfs > server to the new one in a previous job, I 1st generated a list of the top > level maildirs. I then generated the rsync commands + plus a few other bits > and pieces for each maildir to make a single transaction like function. I > then pumped all this auto generated scripts into xjobs and ran them in > parallel. This vastly speeded up the process as sequentially running the > tree was far to slow. THis was for about 15 million maildirs in a hashed > structure btw so a fair amount of files. > > > eg > > find /maildir -type d -maxdepth 4 | while read d > do > r=$(($RANDOM*$RANDOM)) > echo rsync -a $d/ /newpath/$d/ > /tmp/scripts/$r > echo some other stuff >> /tmp/scripts/$r > done > > ls /tmp/scripts/| while read f > echo /tmp/scripts/$f > done | xjobs -j 20 > > > > > > > > > > > On 19 August 2013 18:52, aurfalien <aurfalien@gmail.com> wrote: > >> >> On Aug 19, 2013, at 10:41 AM, Mark Felder wrote: >> >> > On Fri, Aug 16, 2013, at 1:46, Nicolas KOWALSKI wrote: >> >> On Thu, Aug 15, 2013 at 11:13:25AM -0700, aurfalien wrote: >> >>> Is there a faster way to copy files over NFS? >> >> >> >> I would use find+cpio. This handles hard links, permissions, and in >> case >> >> of later runs, will not copy files if they already exist on the >> >> destination. >> >> >> >> # cd /source/dir >> >> # find . | cpio -pvdm /destination/dir >> >> >> > >> > I always found sysutils/cpdup to be faster than rsync. >> >> Ah, bookmarking this one. >> >> Many thanks. >> >> - aurf >> _______________________________________________ >> freebsd-questions@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-questions >> To unsubscribe, send any mail to " >> freebsd-questions-unsubscribe@freebsd.org" >> > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CALfReydQUEmWdjEmh45ZRit61RJq_00PTtR7ej0LVOvqS=ayNw>