From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 04:43:02 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D91CE16A400 for ; Sat, 12 May 2007 04:43:02 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id C50DB13C43E for ; Sat, 12 May 2007 04:43:02 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id AC3561A3C19 for ; Fri, 11 May 2007 21:43:48 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id CEA00513B7; Sat, 12 May 2007 00:43:01 -0400 (EDT) Date: Sat, 12 May 2007 00:43:01 -0400 From: Kris Kennaway To: freebsd-hackers@freebsd.org Message-ID: <20070512044301.GA43885@xor.obsecurity.org> References: <200705112302.35380.dragonsa@highveldmail.co.za> <20070511212847.GA30211@xor.obsecurity.org> <20070512022924.GA1017@dwpc.dwlabs.ca> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="9amGYk9869ThD9tj" Content-Disposition: inline In-Reply-To: <20070512022924.GA1017@dwpc.dwlabs.ca> User-Agent: Mutt/1.4.2.2i Cc: Kris Kennaway Subject: Re: DPS Initial Ideas X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 04:43:03 -0000 --9amGYk9869ThD9tj Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Fri, May 11, 2007 at 11:29:24PM -0300, Duane Whitty wrote: > Is it hoped / planned that storing the metadata in a berkeley DB > database will help with the parallelization of package building? That's somewhat orthogonal: the problem there is mutual exclusion and job ordering. > In your opinion what is the biggest problem(s) the ports system and > the package building system currently face? Is this a common problem, > i.e., is the issue facing building from ports the same as installing > from pre-built packages? I ask this in the context of infrastructure > as opposed to any tools currently being used. > If only one thing was going to be done to improve the ports system, > not including drafting more volunteers :) , what would you recommend > that one thing be? Those are both excellent questions, and I'm not sure I have all the answers. It's something I'll try to discuss with people at the BSDCan devsummit next week. I think the major area that needs further work is to do with improving the upgrade process. Historically, the ports collection and package tools did not support any notion of "do what needs to be done to upgrade this package". This was fine in the days when systems typically only had a handful of ports installed, and they each had only a few dependencies: you could update them by hand without much trouble using 'make deinstall' and 'make install'. Of course, it fails utterly to scale to the era of GNOME and modular X.org. This problem was more or less solved by tools like portupgrade [1], but these are bolted on to the side of the ports collection and underlying package tools rather than being properly integrated with them. One consequence of this is that the UPDATING file has a disturbing number of entries for manual steps required to update certain ports. Some of these are because of vendor screwups that we can't really fix (e.g. broken backawards compatibility with existing user files requiring data migration). But a lot of them come down to failures of the FreeBSD upgrade process: either a ports developer was lazy and didn't want to do the extra work to support automatic upgrades, or the upgrade tools are insufficiently flexible to handle the upgrade. My opinion is that every time a ports developer adds an entry to UPDATING that lists manual steps required to update a port (except for the vendor case above), it usually means we have failed as a project. Part of this problem comes down to metadata management and integration with the underlying tools. Some of the information that an upgrade tool needs to plan and manage the upgrades is not efficiently accessible using the pkg_tools (queries are O(N) or worse in the number of packages installed, etc). portupgrade currently solves this by maintaining its own parallel version of the metadata in a database that can be efficiently queried (it also solves part of the problem by ignoring it, which is a limitation that affected e.g. the xorg upgrade and required the UPDATING entry). The problem with maintaining a parallel database is that unless users always manipulate their ports and packages using portupgrade this database can become stale. Addressing this is the goal of Garrett Cooper's SoC project, by pushing down the database management into pkg_tools so hopefully portupgrade and pkg_tools can be made to share the database (or portupgrade can revert to querying efficiently via the pkg_tools). There are other problems affecting package and upgrade management that are due to scaling limitations and data management issues that have become relevant as the ports collection has grown in size and complexity. Some information (e.g. dependency lists) can currently only be obtained by recursively walking the tree, which is a very expensive operation for things like GNOME. There are probably other operations that are O(N) in some large number N that I can't think of right now. It would make sense to explore whether there are ways of avoiding or optimizing these expensive operations. Kris [1] There are other upgrade tools, but IMO none of them are as mature as portupgrade in terms of feature support or robustness, and they also do not attempt to solve the metadata management issues at all. --9amGYk9869ThD9tj Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGRUXVWry0BWjoQKURAo7XAJ4+zEVT9XvIVjU5Vj4CZmcPnnltHwCfddRu fIykdGkJQ1foR2Mgf88DFxg= =hAGV -----END PGP SIGNATURE----- --9amGYk9869ThD9tj--