Date: Thu, 20 Mar 2008 12:48:00 -0700 From: Doug Barton <dougb@FreeBSD.org> To: Michel Talon <talon@lpthe.jussieu.fr> Cc: freebsd-ports@FreeBSD.org Subject: Re: Utility for safe updating of ports in base system Message-ID: <47E2BF70.2050109@FreeBSD.org> In-Reply-To: <20080320094246.GA40807@lpthe.jussieu.fr> References: <20080320001048.GA39125@lpthe.jussieu.fr> <alpine.BSF.1.10.0803200047360.54264@ync.qbhto.arg> <20080320094246.GA40807@lpthe.jussieu.fr>
next in thread | previous in thread | raw e-mail | index | archive | help
Michel Talon wrote: > On Thu, Mar 20, 2008 at 01:05:27AM -0700, Doug Barton wrote: >> On Thu, 20 Mar 2008, Michel Talon wrote: >> >>> Doug Barton wrote: >>> i would venture to say that such an utility >>> should be able to upgrade things based of *binary* packages, and >>> consequently that portmaster is not a suitable candidate. >> That ability is not included in the current requirements document, and was >> not specifically mentioned the last time we had the discussion on the >> list. If the portmgr folks intend that to be a requirement, the current >> ideas list entry should be amended. > > Indeed you are right, but i think this should be a requirement for the > reasons discussed below, and is implicit in the fact that the projects > refers to a "portupgrade written in C", and portupgrade has such a > feature. Implicit requirements suck. :) However pav obviously agrees with you. >>> For example >>> pkg_add installs a binary package, if you want to compile and install >>> you run "make all install clean" in the ports tree. >> Um, you lost me there. > > This is simple, all pkg_* tools operate on binary packages at present, Ok, but that's not "compile and install," which is why I wanted to clarify. > it would be strange that a newcomer, pkg_upgrade has no way to operate > on binary packages. I think reasonable minds could differ on that, but now we know what pav at least thinks. >>> One of the >>> requirements of an upgrade system is predictability, this can only >>> be achieved by using binary packages. >> You gain a certain amount of flexibility with packages, at the expense of >> being able to customize things. As long as the user understands that, then >> it's fine. >> > > I agree that the possibility of compiling from source brings ability to > costumize things. However this very ability ruins all hope of having a > smooth upgrade system. Due to customization, as soon as you have many > ports installed, there is a combinatorial explosion of possibilities and > nobody can test that the upgrade process works. I agree with what you're saying to a point, however in my experience it's very rare that at least my particular combination of options leads to an explosion, so I'm not really sold on the idea of mandatory binary-only upgrades. > OpenBSD people have > been converted to this idea, and now push for an upgrade from binary > packages exclusively. I think we need to keep the "tools not policy" mantra in mind here. > So i think that the user > should have two possibilities: > - either he wants to customize, and he is on his own. He will need to > compile his software, and in this case portmaster is the perfect tool > for managing his upgrades. Thanks. :) > Since the compilation will take most of the > time, it is not relevant to consider performance questions on the > portmaster side. Having spent a substantial amount of time doing performance tuning on portmaster, I beg to differ. You're right of course that for the average port the majority of time is going to be spent in compiling, however if you're going to be doing something like 'portmaster -a' simply caching the results of the up-to-date tests of the lower level dependencies can cut the "portmaster time" for an upgrade of several ports by more than 1/3rd. And portmaster has a lot of other optimizations as well, such as downloading distfiles in the background, etc. > - or he wants to use a tried and true upgrade path, he doesn't want to > spend time compiling, he wants speed. Basically he wants the Debian > or Ubuntu experience. In this case, using prebuilt binary packages is > the only reasonable way of achieving the result. Well, given the overwhelming number of ports, and the lack of resources to do QA on them, I think we should substitute "more likely to succeed" for "tried and true" there. That said, I have used Ubuntu for a project I did for a client, and I really liked the package management system. However you're talking orders of magnitude more work (and several redesigns of core elements) before we could get close to "the Ubuntu experience." I'm concerned that in the interest of waiting for the perfect thing we do our users a disservice by not taking incremental steps toward the goal. >>> Another requirement, in my opinion, >>> is speed, and the lack of speed, which is completely hidden when you >>> compile your packages will be immediately apparent if you try to use >>> packages. Indeed portupgrade has options -P and -PP to work with >>> packages which could serve as a prototype for a "pkg_upgrade" written >>> in C, except that they work poorly, and in particular run slowly. >> Where do you think the slowness is? > > Several causes of slowness have already been identified. Parsing > /var/db/pkg is slow, s/is/can be/. There are some things I need to dig out of /var/db/pkg that are relatively slow, yes. By far the most expensive is determining the name of an installed port by grep'ing for ORIGIN in all the +CONTENTS files. However even that is not overwhelmingly slow. If I do a cold search for the very last installed port on my box (with 480+ ports installed) I get: real 0m1.095s user 0m0.083s sys 0m0.875s However, once the disk cache is primed I get results like this for the same search: real 0m0.261s user 0m0.020s sys 0m0.233s (note, this is for the very last port, so the average speed for either of these would be roughly 1/2 of the numbers above) Through experimentation and experience I've managed to fine tune the rest of the stuff to the point where the cost of slogging about in /var/db/pkg is almost zero. Now, would having some kind of database be better? Maybe, maybe not. I think it would be a worthy experiment (and as you pointed out, is already a SoC candidate) to find out exactly how much benefit, and how much cost to achieve it. > this is why the idea of using a database has been > advocated. Much worse, running make in a port directory, even only to > get the value of a variable is slow. Now on that I agree with you completely. But now you're talking about a much more fundamental redesign issue that is way outside the scope of the project we're discussing. > Programs like portupgrade do such > things repeatedly without caching the results in memory and incur > large time penalties. One more time for the record, portmaster caches results of everything, but _especially_ the output of make commands, wherever possible. > Similarly for each package he wants to download, > portupgrade opens an ftp connection and retreives it independently, etc. > Obviously no effort at all has been spent so that things are fast. Well, if that is being done sequentially I'd agree that's not optimal. Doing N number of downloads in parallel (where N should be user configurable) is preferable. Portmaster is half way there when it comes to distfile fetching. It will start a distfile fetch in the background for every port that needs it, so you get the parallelization, but the N isn't user-configurable yet. OTOH, I've never had a user complain that I'm swamping their machine with distfile downloads, so it's not been a high priority. >>> One of its features, that i consider very important for correct >>> operation, is that it computes the list of all packages to be deleted >>> and all packages to be installed and asks the user if he agrees before >>> doing anything. >> Why do you consider this an important feature? (I'm not disagreeing, just >> curious about your thought process here.) > > Because you can see that something is going to break in advance and > renounce to the upgrade. This occurred several times for me on Debian. > Usually this is because some package has a security problem, and this is > solved waiting one or two days. Ok, here is where I get confused. :) You've been talking about what is essentially a point-and-click update process that any user could do without needing special knowledge of the ports stuff; but now you're saying you want to be able to second guess it. I would say that this should probably be considered an advanced feature, and off by default. >>> Hence you can be sure that the upgrade process will >>> not end in a mess if something crashes in the middle, like it is the >>> case with all present standard FreeBSD upgraders. >> Not sure if this helps the situation you're referring to or not, but >> portmaster will by default make a backup package of each port that it >> updates, so if something dies in the middle you could back out of it by >> hand if you need to. >> > > Yes, so does portupgrade, but it deletes the backup as soon as the > installation of the new port succeeds. Portmaster waits till the entire process has completed successfully before wiping out all of the backup packages for that run (leaving any others undisturbed). >> Now all that said, I'd love to see us move to a much more robust package >> management system, or even just a better interface to the one we have. The >> problem is that I don't have the time to do that as a volunteer project, >> and I don't think anyone else does either. :) > > Probably the present package management system doesn't keep enough state > on each installed package so that an upgrade system works reliably. For > example, referring to the possibilities to tweak options, i think that > keeping a copy of the options used for compilation of the port in the > package could help, Here you and I are in agreement. I have had an occasionally-ongoing discussion with Kris about the relative virtue of putting data about the port into the package, and if I understand him correctly he comes out on the side of "If it has to do with compilation it belongs in /var/db/ports, not pkg." (See http://www.freebsd.org/cgi/query-pr.cgi?pr=106483 for example) My feeling is that all the information should be preserved, since even if the user installs a package this time, they might want to update by port next time. > or at least a flag indicating if the package is > basically standard or has been tweaked. Similarly keeping track of > wether a package has been installed by order of the end user or as a > dependency could help to unclutter after an upgrade. We sort of have that now with the +REQUIRED_BY file, and I think we have to be careful of how this is handled. My favorite example is the plugins for firefox and thunderbird. If you install one, what was the leaf port (firefox/thunderbird) now becomes a "branch" port (that is, it has dependencies, and is depended on), but if I uninstall the plugin I don't want my browser deleted as a stale dependency. > On the other hand, > basic libraries, which, when upgraded, require to upgrade a lot of ports > should have a better way to require that, rather that asking the end > user to read UPGRADING. The practice at this time is to bump PORTREVISION for the ports that have that dependency, although that is a one-size-doesn't-always-fit-all way to handle this issue. Good discussion, thanks! Doug -- This .signature sanitized for your protection
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?47E2BF70.2050109>