From owner-freebsd-hackers@FreeBSD.ORG Fri May 9 17:55:55 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C4B55106564A for ; Fri, 9 May 2008 17:55:55 +0000 (UTC) (envelope-from andenore@start.no) Received: from osl1smout1.broadpark.no (osl1smout1.broadpark.no [80.202.4.58]) by mx1.freebsd.org (Postfix) with ESMTP id 7C99A8FC31 for ; Fri, 9 May 2008 17:55:55 +0000 (UTC) (envelope-from andenore@start.no) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; format=flowed; delsp=yes; charset=us-ascii Received: from osl1sminn1.broadpark.no ([80.202.4.59]) by osl1smout1.broadpark.no (Sun Java(tm) System Messaging Server 6.3-3.01 (built Jul 12 2007; 32bit)) with ESMTP id <0K0M001EK4H6F040@osl1smout1.broadpark.no> for freebsd-hackers@freebsd.org; Fri, 09 May 2008 19:55:54 +0200 (CEST) Received: from duckjen.nextgentel.no ([84.48.194.205]) by osl1sminn1.broadpark.no (Sun Java(tm) System Messaging Server 6.3-3.01 (built Jul 12 2007; 32bit)) with ESMTP id <0K0M0050V4H4TB92@osl1sminn1.broadpark.no> for freebsd-hackers@freebsd.org; Fri, 09 May 2008 19:55:54 +0200 (CEST) Date: Fri, 09 May 2008 19:54:40 +0200 To: "freebsd-hackers@freebsd.org" From: Anders Nore References: <20080509124308.GA596@britannica.bec.de> <20080509170633.GB3571@britannica.bec.de> Message-id: In-reply-to: <20080509170633.GB3571@britannica.bec.de> User-Agent: Opera Mail/9.27 (FreeBSD) Cc: Joerg Sonnenberger Subject: Re: Adding .db support to pkg_tools X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 May 2008 17:55:55 -0000 On Fri, 09 May 2008 19:06:33 +0200, Joerg Sonnenberger wrote: > On Fri, May 09, 2008 at 06:50:10PM +0200, Anders Nore wrote: >> Yes that would probably be bad for the database, but I'm sure one can >> manage to get around this problem by copying it before changing the db >> and >> delete the copy if it doesn't fail. At the next time executed it will >> check >> for a copy, use that and assume that the last run was unsuccessful. > > /var/db/pkg contains 10MB for the various packages installed on my > laptop and 10MB for the cache of +CONTENTS. You don't want to copy that > around all the time. > >>> Secondly, I would also advisy against just storing all meta data in a >>> single key/value pair. For example, +CONTENTS can be extremely large. >>> Check texmf for a good example. >> >> When it comes to storing large values in a key/value pair, I think >> that's >> what bdb was designed for, handling large amounts of data (in the >> orders of >> gigabytes even in key's) fast. > > No, actually that is exactly what it was *not* designed for. Having > billions of keys is fine, but data that exceeds the size of a database > page is going to slow down. While it might not be a problem of you are > copying the data to a new file anyway, it also means that fragmentation > in the database will be more problematic. > > My main point is that for the interesting operations you want to > actually look up with fine grained keys and that's what is not possible > if you store the meta data as blob. In fact, storing the meta data as > blob is not faster than just using the filesystem. > You are probably right, but how would you store the key's? Is storing the key as e.g., 'portname-1.2_3+CONTENT' a good solution?