From owner-freebsd-ports@FreeBSD.ORG Fri Aug 20 10:00:39 2010 Return-Path: Delivered-To: freebsd-ports@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 110561065696; Fri, 20 Aug 2010 10:00:39 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 313908FC12; Fri, 20 Aug 2010 10:00:37 +0000 (UTC) Received: by qyk4 with SMTP id 4so3248855qyk.13 for ; Fri, 20 Aug 2010 03:00:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:sender:received :in-reply-to:references:date:x-google-sender-auth:message-id:subject :from:to:cc:content-type; bh=LO3fh6UnUk7p8KqI01J83smEOa6BPlyMNvy38u1bRDM=; b=ILogqeEA4UhcDdM8tqXV5CQH/g5KJ7Es3g605tmnJmwB73nx1U5IP2xiOEwZ+bYc1Q 4AMoMtO7UGXsOe27312Sxgvq+U3KD9rfELjC+EoiCGcGtArmlnJIvsFNGkUD6vOHNdTI 6EfMHCamwL6ko+AwHzh2RnCeVN0idCNrBcnhk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; b=Rl1OqylxlTTf18MqlbiBCCBOpiQ0X9X825T7ytjCsR+5BPKV4W32UHENl1dfRHVk3U R1/Y1/xtGQF0xMiCO+LsJdrXg6C7IqxP+ZAdJqDOECL35c8v3gsGtp5XE4SNsY/fducD NErAlp8PDJkGUByKN/KoSvQmS9DOgtKRg/omA= MIME-Version: 1.0 Received: by 10.224.11.131 with SMTP id t3mr815314qat.17.1282298437191; Fri, 20 Aug 2010 03:00:37 -0700 (PDT) Sender: ivoras@gmail.com Received: by 10.229.222.81 with HTTP; Fri, 20 Aug 2010 03:00:36 -0700 (PDT) In-Reply-To: References: <20100819143830.GJ35140@azathoth.lan> <4C6DA0FA.7080203@DataIX.net> Date: Fri, 20 Aug 2010 12:00:36 +0200 X-Google-Sender-Auth: gKdOgZ9mBOjelMOBCh2jBAxVXmY Message-ID: From: Ivan Voras To: Garrett Cooper Content-Type: text/plain; charset=UTF-8 Cc: bapt@freebsd.org, Florent Thoumie , Julien Laffaye , David Forsythe , Tim Kientzle , freebsd-ports@freebsd.org Subject: Re: what next for the pkg_install rewrite X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Aug 2010 10:00:39 -0000 On 20/08/2010, Garrett Cooper wrote: > 1. SQLite was killed before because of complexity and because it was > needs another package in base that isn't BSD licensed. That's why And... both ideas are completely wrong. SQLite can be imported as a single C file + header, which you must agree is practically the optimum, and its license is "public domain" which is, if anything, "freer" than BSDL and eminently compatible with it. > everyone in the know has been pushing for BDB 1.8x (in base). People BDB in base offer exactly one (1) single "feature", if you can call it that: storing and retrieving key-value binary blobs. It has no practical concurrency support, it's locking is laughable and upgrading data stored as binary blobs is even more nightmarish than maintaining the current plist format (and it will lead to similar uglyness soon - rather than upgrading the data piece called "x", I'm sure developers will introduce new keys called "x_extdata", "x_moredata" etc etc). SQLite on the other hand solves all these in the following way: 1) stores proper records, not blobs 2) has proper locking & concurrency support, ACID 3) the database schema can be modified on the fly for upgrading - fields can be added to existing table while preserving data. 4) is endian-agnostic, 32/64-bit agnostic and portable (I mean the database file) to an extremely large number of platforms. It is already used as a system database in OS X, iPhone and Android. Note that I'm promoting SQLite to replace the /var/db/pkg/* tree of directories and files with ad-hoc structure - all data in there should be in one SQLite database, which is stored in a single file. > suggested that to me back when I was trying to get off the ground in > GSoC 2006, they did it to you before Ivan, and I'm sure they've done > it before in the past. We need to implement things in terms of BDB > first, but make the APIs generic enough so that it _could_ be extended > to SQLite if people chose to do the work later on. Not planning an API for a transactional database in the first place will bring the whole thing down in the long term, and in any case will certainly push things 10 more years in the future. Note that SQLite can *not* be considered a drop-in replacement for BDB (any version of BDB) because in that case you will gain far less than is optimal. If you haven't used SQLite yet, please try it, from C, and then see if you can reevaluate your comment on its complexity. If you don't like SQL, this is also fine. You are not out of the game, there are many other parts to work on. As for backward compatibility: basically it can be done in two ways: 1) build a one-time converter that will read the present plist database and output a new database or 2) build a compatibility layer which would support both the old and the new format at the same time, so people upgrading will continue to use their old data. I consider the latter wasteful in terms of developer resources and because bit-rot will eventually make support for the old format decrepit. > 2. XML is a bad idea. Great in theory, wonderful in my browser, but a > bloated plaintext file with a lot of complexity. I would prefer a > database solution that could be dumped to a simple text file format if > needed. XML, at least in my proposal, would not be used for the system package database, but would replace (either in part or all with a single file) today's "+" files in the packages, together with other purposes where some metadata transfer is required. That said, I would also be happy with other self-described formats like JSON. XML is the front runner because it's more standard (industry standard, whether someone likes this fact or not!) and there is already a parser in base. > Again (and I can't overemphasize this): no matter what we do, say, > etc, unless we have buy-in from portmgr beforehand on anything here, > the work will never see a ports/src CVS/svn repo, etc. This includes > work done for past GSoCs, and current GSoCs. With all respect to portmgr, I expect it to keep a cool head and acknowledge the following: 1) Decisions must be made on objective terms which include consideration for clean / elegant implementation, long-term maintainability and standards. If there is to be a rewrite, it must be built to stand the test of next 10 years of usage, and not start compromising even before it is started. 2) Except in extreme cases, portmgr should decide based on functionality and not have that much say in the specifics of the implementation. Basically, the portmgr should in the first place approve the feature spec, and the src people should worry about the details of what can and cannot be done. This is not a src vs portmgr war, this is separation of duty. (of course there are overlaps in people's memberships)