Date: Thu, 4 Apr 2013 17:52:16 -0700 From: Waitman Gobble <gobble.wa@gmail.com> To: Joar Jegleim <joar.jegleim@gmail.com> Cc: Terje Elde <terje@elde.net>, freebsd-questions@freebsd.org Subject: Re: Regarding zfs send / receive Message-ID: <CAFuo_fwtQnRFqpd%2BH=YNS8Ghk7nRf5mW7Ei9pz9c346PK0aOvQ@mail.gmail.com> In-Reply-To: <CAFfb-hoDYEdZGo5gfv=PbyCUKuDC6N0ECn=27YzUYEW=C%2BeaLA@mail.gmail.com> References: <CAFfb-hqTFH0oK9rOpWHo6wrodzuOm5oRbetqY3RSXvF7Gsa6PA@mail.gmail.com> <E019EA56-708A-481D-9FD8-5EB66D6B98AA@elde.net> <CAFfb-hoDYEdZGo5gfv=PbyCUKuDC6N0ECn=27YzUYEW=C%2BeaLA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Waitman Gobble San Jose California USA On Apr 4, 2013 2:07 PM, "Joar Jegleim" <joar.jegleim@gmail.com> wrote: > > Hi Terje ! > sorry for late reply, I've been checking my mail, forgetting that all my > mailing list mail are sorted into their own folders skipping inbox :p > > the zfs sync setup is a huge advantage over rsync simply because > incremental rsync of the volume takes ~12 hours, while the zfs differential > snapshot's usually take less than a minute . Though it's only ~1TB of data, > it's more than 2 million jpegs which rsync have to stat ... > I'm guessing my predecessor who chose this setup, over for instance HAST, > didn't feel confident enough regarding HAST in production ( I'm looking > into that for a future solution) . > > There's no legacy stuff on the receiving end, old pools are deleted for > every sync. I haven't got my script here but google pointed me too > https://github.com/hoopty/zfs-sync/blob/master/zfs-sync which look like a > script very similar to the one I'm using . > In fact, I'm gonna take a closer look at that script and see what differs > from my script (apart from it being much prettier :p ) > I didn't know about zpool.cache, gonna check that tomorrow, thanks. > > > > -- > ---------------------- > Joar > Jegleim > Homepage: http://cosmicb.no > Linkedin: http://no.linkedin.com/in/joarjegleim > fb: http://www.facebook.com/joar.jegleim > AKA: CosmicB @Freenode > > ---------------------- > > On 2 April 2013 14:40, Terje Elde <terje@elde.net> wrote: > > > On 2. apr. 2013, at 13.44, Joar Jegleim wrote: > > > So my question(s) to the list would be: > > > In my setup have I taken the use case for zfs send / receive too far > > > (?) as in, it's not meant for this kind of syncing and this often, so > > > there's actually nothing 'wrong'. > > > > I'm not sure if you've taken it too far, but I'm not entirely sure if > > you're getting any advantage over using rsync or similar for this kind of > > thing. > > > > First two things that spring to mind: > > > > Do you have any legacy stuff on the receiving machine? Things like > > physically removed old zpools, that are still in zpool.cache, seems to slow > > down various operations, including creation of new stuffs (such as the > > snapshots you receive). > > > > Also, you don't mention if you're deleting old snapshots on the receiving > > end? If you're doing an incremental run every 15 minutes, that's something > > like 3000 snapshots pr. month, pr. filesystem. > > > > Terje > > > > > hi, i have a similar situation. its better to only rsync new stuff in this case, because you should know when somebody ads something new. for example, a user uploads 200 new images, these are marked 'to sync' and are transferred to the other servers. letting rsync figure out what's new just isnt practical. an idea, works for me. hope it helps. Waitman Gobble San Jose California _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to " freebsd-questions-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFuo_fwtQnRFqpd%2BH=YNS8Ghk7nRf5mW7Ei9pz9c346PK0aOvQ>