From owner-freebsd-ports@FreeBSD.ORG  Fri Jul  6 16:45:07 2007
Return-Path: <owner-freebsd-ports@FreeBSD.ORG>
X-Original-To: ports@freebsd.org
Delivered-To: freebsd-ports@FreeBSD.ORG
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 6C96916A421;
	Fri,  6 Jul 2007 16:45:07 +0000 (UTC)
	(envelope-from kientzle@freebsd.org)
Received: from kientzle.com (h-66-166-149-50.snvacaid.covad.net
	[66.166.149.50])
	by mx1.freebsd.org (Postfix) with ESMTP id 35B8C13C480;
	Fri,  6 Jul 2007 16:45:07 +0000 (UTC)
	(envelope-from kientzle@freebsd.org)
Received: from [10.0.0.222] (p54.kientzle.com [66.166.149.54])
	by kientzle.com (8.12.9/8.12.9) with ESMTP id l66Gj6H7039341;
	Fri, 6 Jul 2007 09:45:06 -0700 (PDT)
	(envelope-from kientzle@freebsd.org)
Message-ID: <468E7192.8030105@freebsd.org>
Date: Fri, 06 Jul 2007 09:45:06 -0700
From: Tim Kientzle <kientzle@freebsd.org>
User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.12) Gecko/20060422
X-Accept-Language: en-us, en
MIME-Version: 1.0
To: Garrett Cooper <youshi10@u.washington.edu>
References: <468C96C0.1040603@u.washington.edu>	<468C9718.1050108@u.washington.edu>
	<468E60E9.80507@freebsd.org> <468E6C81.4060908@u.washington.edu>
In-Reply-To: <468E6C81.4060908@u.washington.edu>
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Cc: ports@freebsd.org, hackers@freebsd.org
Subject: Re: Finding slowdowns in pkg_install (continuations
	of	previous	threads)
X-BeenThere: freebsd-ports@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Porting software to FreeBSD <freebsd-ports.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-ports>,
	<mailto:freebsd-ports-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-ports>
List-Post: <mailto:freebsd-ports@freebsd.org>
List-Help: <mailto:freebsd-ports-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-ports>,
	<mailto:freebsd-ports-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 06 Jul 2007 16:45:07 -0000

>    -I tried ... buffering ...  the +CONTENTS file parsing function, and the
> majority of the time it yielded good results ....

One approach I prototyped sometime back was to use
libarchive in pkg_add as follows:
   * Open the archive
   * Read +CONTENTS directly into memory (it's
guaranteed to always be first in the archive)
   * Parse all of +CONTENTS at once
   * Continue scanning the archive, disposing
of each file as it appears in the archive.

Based on my experience with this, I would
suggest you just read all of +CONTENTS
directly into memory at once and parse
the whole thing in a single shot.
fopen(), then fstat() to get the size,
then allocate a buffer and read the whole
thing, then fclose().  You can then
parse it all at once.

As a bonus, your parser then becomes a nice
little bit of reusable code that reads
a block of memory and returns a structure describing
the package metadata.

Tim Kientzle