From owner-freebsd-questions@FreeBSD.ORG Wed May 20 07:31:21 2009 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8EE65106571B for ; Wed, 20 May 2009 07:31:21 +0000 (UTC) (envelope-from valentin.bud@gmail.com) Received: from mail-bw0-f165.google.com (mail-bw0-f165.google.com [209.85.218.165]) by mx1.freebsd.org (Postfix) with ESMTP id 693998FC17 for ; Wed, 20 May 2009 07:31:20 +0000 (UTC) (envelope-from valentin.bud@gmail.com) Received: by bwz9 with SMTP id 9so234489bwz.43 for ; Wed, 20 May 2009 00:31:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :from:date:message-id:subject:to:cc:content-type; bh=cE6YHGXjbt3PJIV/mPkm2JzjWNb4lUSLoHmdfxObuag=; b=Q5JJQNiaFRmBftePsJJ9czK9B8AjWdoUVymxHuH06ejtv0SeubwOEkHeWs6buvGlej 9txFPSWau7+KJzyCvWdxUK8MshdVI92YPDsK8R5owOt6jwkAMr1riAMWFPR8idAwTu3V z8Rl59fr4KENOBLnsjrBMpaKutpLEMTESpLVk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; b=kItKA9ZLdBIZX7P9LW/wdKEPBaENu7ZkI7Ex9BPx9zQdIyliuq/hdiryMW4U7iDB5A 6P+gzEQsQG0t2rCWOwhTzaylju0+OsZOb8FZnXF5/YzOpp+LCGvuB+dmF6U6MQzqmm1B DrNORAfXKUDlRBeF93B6QcEo2X5XHa1HIBSC0= MIME-Version: 1.0 Received: by 10.103.134.8 with SMTP id l8mr473170mun.116.1242802876093; Wed, 20 May 2009 00:01:16 -0700 (PDT) In-Reply-To: <20090518183829.D0E7BBEBB@kev.msw.wpafb.af.mil> References: <26face530905170912m3ca8b762nd0cfadc7db34da6f@mail.gmail.com> <20090518183829.D0E7BBEBB@kev.msw.wpafb.af.mil> From: Valentin Bud Date: Wed, 20 May 2009 10:00:56 +0300 Message-ID: <139b44430905200000l250c9ae6p1ddcb1a6ac10bef8@mail.gmail.com> To: vogelke+unix@pobox.com Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Kelly Jones , freebsd-questions@freebsd.org Subject: Re: Backing up FreeBSD and other Unix systems securely X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2009 07:31:22 -0000 On Mon, May 18, 2009 at 9:38 PM, Karl Vogel > wrote: > >> On Sun, 17 May 2009 09:12:57 -0700, > >> Kelly Jones said: > > K> I like this plan because it does versioned backups, and doesn't backup > K> identical files twice. I dislike it because I lose Mozy's unlimited disk > K> space. > > K> % Is there software that already does this? > > I have a 3-Tbyte server running FreeBSD-6.1 that does something very > similar. I don't bother with encrypting the filenames or hashes > because we control the box, and if I'm not at work, other admins > might need to restore something quickly. > > We have around 3.7 million files from 5 other servers backed up > under two 1.5-Tbyte filesystems, /mir01 and /mir02. My setup looks > like this: > > +-----mir01 > | +-----HASH > | | +-----00 > | | | +-----00 > | | | +-----01 > ... > | | +-----01 > ... > | | +-----fe > | | +-----ff > | +-----server1 > | +-----server2 > +-----mir02 > | +-----HASH > | +-----server3 > | +-----server4 > | +-----server5 > > The HASH directories have two levels of subdirectories 00-ff. > That's been more than sufficient to keep directories from getting > too big; I average around 25 files per directory. > > I do hourly backups on the other fileservers using something like the > find and timestamp method you mentioned, but I ignore 0-length files > because they always hash to the same value. The backup directories > for the second fileserver look like this for 5 May 2009: > > +-----mir01 > | +-----server2 > | | +-----2009 > | | | +-----0505 > | | | | +-----070700 > | | | | | +-----doc (filesystem) > | | | | | +-----home > | | | | +-----080700 > | | | | | +-----doc > | | | | | +-----home > ... > | | | | +-----190700 > | | | | | +-----home > > After the backups are rsynced to the backup server, I find any regular > files with only one link, compute the RMD160 hash of the contents, and > make a hardlink to the appropriate filename under the HASH directory. > People love to make copies of copies of files, so this really cuts down > on the disk space used. > > The hardlinks make it easy to avoid restoring things that aren't what > the user had in mind; if a file's been corrupted, I can tell when it > happened just by looking at the inode, so I don't restore an earlier > version that's also junk. I can also tell if there were duplicates > anywhere on the fileserver at the time the user lost the good version; > it's a lot faster for them to get a known good copy from somewhere > else on the fileserver than it is to restore over the network. > > The software is just a few scripts to do things like find files with > just one link, compute hashes, do hardlinks, etc. I can put up a tarball > if anyone's interested. > Hello Kelly, I am doing something similar at a company i work for. I would be interested to see your scripts to make a comparison. thanks, v > > -- > Karl Vogel I don't speak for the USAF or my company > > The best way for the Government to maintain its credit is to pay as it > goes-not by resorting to loans, but by keeping out of debt-through an > adequate income secured by a system of taxation, external or internal, > or both. --Pres. William McKinley's First Inaugural Address, March 4, 1897 > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to " > freebsd-questions-unsubscribe@freebsd.org" > -- network warrior since 2005