From owner-cvs-all@FreeBSD.ORG  Fri Apr 23 12:20:39 2004
Return-Path: <owner-cvs-all@FreeBSD.ORG>
Delivered-To: cvs-all@freebsd.org
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP
	id 40D3E16A4D0; Fri, 23 Apr 2004 12:20:39 -0700 (PDT)
Received: from rwcrmhc11.comcast.net (rwcrmhc11.comcast.net [204.127.198.35])
	by mx1.FreeBSD.org (Postfix) with ESMTP
	id CBB3543D5D; Fri, 23 Apr 2004 12:20:38 -0700 (PDT)
	(envelope-from julian@elischer.org)
Received: from interjet.elischer.org ([24.7.73.28])
          by comcast.net (rwcrmhc11) with ESMTP
          id <20040423192036013004oij1e>; Fri, 23 Apr 2004 19:20:36 +0000
Received: from localhost (localhost.elischer.org [127.0.0.1])
	by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id MAA08427;
	Fri, 23 Apr 2004 12:20:35 -0700 (PDT)
Date: Fri, 23 Apr 2004 12:20:34 -0700 (PDT)
From: Julian Elischer <julian@elischer.org>
To: Tim Kientzle <kientzle@FreeBSD.org>
In-Reply-To: <200404231627.i3NGRcVA096244@repoman.freebsd.org>
Message-ID: <Pine.BSF.4.21.0404231145150.6894-100000@InterJet.elischer.org>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
cc: cvs-src@FreeBSD.org
cc: src-committers@FreeBSD.org
cc: cvs-all@FreeBSD.org
Subject: bad news for bsdtar..
X-BeenThere: cvs-all@freebsd.org
X-Mailman-Version: 2.1.1
Precedence: list
List-Id: CVS commit messages for the entire tree <cvs-all.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/cvs-all>,
	<mailto:cvs-all-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/cvs-all>
List-Post: <mailto:cvs-all@freebsd.org>
List-Help: <mailto:cvs-all-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/cvs-all>,
	<mailto:cvs-all-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 23 Apr 2004 19:20:39 -0000

done on a small subdirectory of our data...
(only accounts starting with 849xxx)
total data as reported by 'du': 9394486 KB  (9GB)
(note du takes links into account and doesn't count the same data twice)
number of files with > 1 link: 
# find 849* -type f -links +1 | wc -l
  240395
#
total number of file directory entries..
# find 849* -type f | wc -l
  598018
#
total number of directories:
%find 849* -type d | wc -l
   51245

average filesize 23k
but major file types are: jpeg images.. 100kb
			jpeg thumbnails, 3kb
			transaction descriptors 1kb
########################################################
the bad news:

old (FreeBSD 2.x) tar with symlink hack
%time ~/tar cf /dev/null 849*
5.309u 73.584s 21:15.44 6.1%    329+5150k 786697+0io 0pf+0w

freebsd 4.8 tar
%time tar cf /dev/null 849
5.159u 17.724s 4:08.97 9.1%     364+5863k 66144+0io 1pf+0w

bsdtar
%time ~/bsdtar cf /dev/null 849*
16.423u 68.141s 31:40.82 4.4%   402+1639k 784651+0io 0pf+0w

It looks to me that the tar in 4.8 is doing something very
sneeky with it's IOs..
maybe using mmap?
maybe detecting /dev/null and instead discarding the data?
maybe reading the entire file in a single read?