From owner-freebsd-fs@FreeBSD.ORG Wed Jul 18 05:08:03 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9D9CD106564A for ; Wed, 18 Jul 2012 05:08:03 +0000 (UTC) (envelope-from freebsd-fs@ch.pkts.ca) Received: from foster.nce.ubc.ca (cl-937.chi-02.us.sixxs.net [IPv6:2001:4978:f:3a8::2]) by mx1.freebsd.org (Postfix) with ESMTP id 2F71D8FC16 for ; Wed, 18 Jul 2012 05:08:03 +0000 (UTC) Received: from kirk.lan (S010600032d00065e.vc.shawcable.net [24.86.111.248]) (authenticated bits=0) by foster.nce.ubc.ca (8.14.4/8.14.4) with ESMTP id q6I57pqD012711 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Tue, 17 Jul 2012 22:07:52 -0700 Date: Tue, 17 Jul 2012 22:07:50 -0700 From: "CH" To: "Steven Hartland" Message-ID: <20120717220750.03a7b7c5@kirk.lan> In-Reply-To: <451BA6FA76214F37B2F271F91708E275@multiplay.co.uk> References: <20120717152629.42e0641e@fedora14-x86-64.shechinah.mi.microbiology.ubc.ca> <451BA6FA76214F37B2F271F91708E275@multiplay.co.uk> X-Mailer: Claws Mail 3.7.9 (GTK+ 2.22.0; i386-redhat-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.4 required=5.0 tests=ALL_TRUSTED,BAYES_00, DATE_IN_FUTURE_12_24,SARE_SUB_OBFU_OTHER autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on foster.nce.ubc.ca Cc: freebsd-fs@freebsd.org Subject: Re: Can you list internal checksums of a ZFS filesystem? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jul 2012 05:08:03 -0000 On Tue, 17 Jul 2012 23:54:38 +0100 "Steven Hartland" wrote: > ----- Original Message ----- > From: "CH" > To: > Sent: Tuesday, July 17, 2012 11:26 PM > Subject: Can you list internal checksums of a ZFS filesystem? > > > > Hello list, > > > > I'm moving data to a ZFS filesystem, and it's a ton of big files > > (more than 3 terabytes). I don't trust the network copy command > > completely, and so I'd like to compare checksums. I'm not looking > > forward to it, since it's going to be a slow process, especially if > > I can't run the command on the server. > > > > Since ZFS already has computed and stored checksums for every file > > on the destination machine, is there an easy way to extract that > > from the filesystem? Perhaps 'ls -lR --checksums' or similar? > > > > Note, checksums still have to be calculated on the source machine, > > and so the overall operation will still take the same length of time > > (assuming ZFS is faster than HFS+), but it does save a lot of cpu > > and disk reads on the destination machine, which can then be used > > for other things. > > IIRC it's block level checksums, not file level ones, so I don't think > this will be what you're looking for. > > Regards > Steve You're right, block-level checksums would be less efficient, and there probably isn't a handy program out there for generating the same block-level checksums on a non-ZFS system. I suspect it wouldn't (shouldn't?) be hard: for each file, while not eof, read block, checksum block, print checksum, repeat, compare list of checksums, do next file. As long as the checksum algorithm and the block size are the same, the thing should work. In any case, if the checksums are effectively unreachable from userspace, then I should just recalculate file-level checksums. Thanks CH