From owner-freebsd-stable@FreeBSD.ORG Thu Oct 30 15:04:39 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 235E91065672 for ; Thu, 30 Oct 2008 15:04:39 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from smtp.sd73.bc.ca (smtp.sd73.bc.ca [142.24.13.140]) by mx1.freebsd.org (Postfix) with ESMTP id ED1218FC20 for ; Thu, 30 Oct 2008 15:04:38 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from localhost (localhost [127.0.0.1]) by localhost.sd73.bc.ca (Postfix) with ESMTP id BDA8F1A000B31 for ; Thu, 30 Oct 2008 08:04:37 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at smtp.sd73.bc.ca Received: from smtp.sd73.bc.ca ([127.0.0.1]) by localhost (smtp.sd73.bc.ca [127.0.0.1]) (amavisd-new, port 10024) with LMTP id LKgclM7PmJCU for ; Thu, 30 Oct 2008 08:04:31 -0700 (PDT) Received: from coal (s10.sbo [192.168.0.10]) by smtp.sd73.bc.ca (Postfix) with ESMTP id 5687C1A000B18 for ; Thu, 30 Oct 2008 08:04:31 -0700 (PDT) From: Freddie Cash To: freebsd-stable@freebsd.org Date: Thu, 30 Oct 2008 08:04:31 -0700 User-Agent: KMail/1.9.9 References: <20081029231926.GA35188@0lsen.net> <95550BEC-DB92-4C68-8409-3DFF7C0B86C0@gmail.com> In-Reply-To: <95550BEC-DB92-4C68-8409-3DFF7C0B86C0@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200810300804.31186.fjwcash@gmail.com> Subject: Re: Anyone used rsync scriptology for incremental backup? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Oct 2008 15:04:39 -0000 On October 30, 2008 01:25 am Nikolay Denev wrote: > On 30 Oct, 2008, at 07:00 , Freddie Cash wrote: > > On Thu, Oct 30, 2008 at 1:50 AM, Andrew Snow > > > > wrote: > >> In this way, each day we generate a batch file that lets us step > >> back one > >> day. The diffs themselves, compressed with gzip, and extremely > >> space efficient. We can step back potentially hundreds of days, > >> though it seems > >> to throw errors sometimes when backing up Windows boxes, which I > >> haven't > >> tracked down yet. > >> > >> But to be honest, soon you can save yourself a lot of hassle by > >> simply using > >> ZFS and taking snapshots. It'll be faster, and with compression > >> very space > >> efficient. > > > > That's exactly what we do, use ZFS and RSync. We have a ZFS > > /storage/backup filesystem, with directories for each remote site, > > and sub-directories for each server to be backed up. > > > > Each night we snapshot the directory, then run rsync to backup each > > server. Snapshots are named with the current date. For 80 FreeBSD > > and Linux servers, we average 10 GB of changed data a night. > > > > No muss, no fuss. We've used it to restore entire servers (boot off > > Knoppix/Frenzy CD, format partitions, rsync back), individual files > > (no mounting required, just cd into the .zfs/snapshot/snapshotname > > directory and scp the file), and even once to restore the permissions > > on a pair of servers where a clueless admin "chmod -R user /home" and > > "chmod -R 777 /home". > > > > Our backup script is pretty much just a double-for loop that scans a > > set of site-name directories for server config files, and runs rsync > > in parallel (1 per remote site). > > > > We we looking into using variations on rsnapshot, custom > > squashfs/hardlink stuff, and other solutions, but once we started > > using ZFS, we stopped looking down those roads. We were able to do > > in 3 days of testing and scripting what we hadn't been able to do in > > almost a month of research and testing. > Do you experience problems with the snapshots? > Last time I tried something similiar for backups the bachine > began to spit errors after a few days of snapshots. > > http://lists.freebsd.org/pipermail/freebsd-fs/2008-February/004413.html We have 72 daily snapshots so far. Have had up to 30 of them mounted read-only while looking for the right version of a file to restore. These are ZFS snapshots, very different from UFS snapshots. -- Freddie Cash fjwcash@gmail.com