Date: Tue, 31 May 2011 17:57:52 -0500 (CDT) From: Bob Friesenhahn <bfriesen@simple.dallas.tx.us> To: Per von Zweigbergk <pvz@itassistans.se> Cc: freebsd-fs@freebsd.org Subject: Re: Storing revisions of large files using ZFS snapshots Message-ID: <alpine.GSO.2.01.1105311749020.20825@freddy.simplesystems.org> In-Reply-To: <FFD730CF-2454-4A07-BA26-0C3B7257780E@itassistans.se> References: <FFD730CF-2454-4A07-BA26-0C3B7257780E@itassistans.se>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 31 May 2011, Per von Zweigbergk wrote: > > Basically what would happen is that every night after the backup > completes, rsync would be run, synchronizing over the differences > between the synthetic full backup from the previous day. Historic > copies of the full backup images as synchronized by rsync would be > kept using ZFS snapshots. After our retention window closes, I'd > just nuke the oldest snapshots from the server. This is feasable using rsync's "--inplace --no-whole-file" options, and is what I use as part of my daily backup strategy so that each backup cycle only consumes disk space for the changed blocks. It works fantastically. There is of course some risk associated with these rsync options since if rsync fails, the backup file may only be partially updated. Of course you can re-try the rsync a number of times during the backup interval if rsync fails. If the rsync previously failed (or perhaps always), it is necessary to also supply the expensive --checksum option so that rsync will definitely assure that the file content is correct. This option is expensive since it will force all blocks to be read on both sides. > First of all, will ZFS will do copy-on-write on a block level when > it comes to snapshots, or is copy-on-write on ZFS snapshots done on > a whole-file level? It would seem that block-level COW would be > required for this to even have a chance of working. Please note that > I'm not talking about deduplication in ZFS itself, but rather using > snapshots as a means to perform a crude kind of deduplication. ZFS does copy-on-write for file blocks when the file is updated. There is surely some copy-on-write of zfs metadata when a snapshot is taken, but surely vastly less than the amount of data consumed by the file. Remember that filesystem volumes look like files and they snapshot nicely. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.GSO.2.01.1105311749020.20825>