Date: Fri, 27 Jul 2007 21:48:47 -0400 From: Yoshihiro Ota <ota@j.email.ne.jp> To: Garrett Cooper <youshi10@u.washington.edu> Cc: freebsd-ports@freebsd.org Subject: Re: Call for testers for yet another ports upgrade program, ports+ Message-ID: <20070727214847.6708a918.ota@j.email.ne.jp> In-Reply-To: <46A9B112.7040408@u.washington.edu> References: <20070726011654.cec378be.ota@j.email.ne.jp> <46A866BE.1000407@u.washington.edu> <20070726233610.e536c2e2.ota@j.email.ne.jp> <46A9B112.7040408@u.washington.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, 27 Jul 2007 01:47:14 -0700 Garrett Cooper <youshi10@u.washington.edu> wrote: > Yoshihiro Ota wrote: > > On Thu, 26 Jul 2007 02:17:50 -0700 > > Garrett Cooper <youshi10@u.washington.edu> wrote: > > > > > >> Yoshihiro Ota wrote: > >> > > Portupgrade is not only slow reading INDEX file but also on dependency resolving and updating > > +CONTENTS files, too. I think portmaster is also one tries to read and do the same things but > > with shell script. I personally didn't have good luck with portmaster and haven't really used > > to evaluate. However, "portmaster -a -n" wasn't not fast, neither. By the way, it builds > > ports in background, doesn't it? > Yes, but a lot of time is spent reading the INDEX file, especially since > portupgrade / portinstall don't keep that information in memory > (probably for the better though). I, so far, only observe the behavior of portupgrade. I know it took so long doing something and guessed it was trying to resolve dependencies. It seems my guess wasn't quite right. > Like Doug said, portmaster doesn't built ports in the background, and > the way that the package system is currently setup I don't advise doing > that unless you plan everything out properly (I see potential for > duplicated installs, race conditions, and other unwanted possibilities). > I'm working on getting away from that at the moment. Even portupgrade doesn't work on some of these cases anyway. I had problems with: The default ghostscript port is changed from ghostscript-gnu to ghostscript-gpl. To upgrade portupgrade -o print/ghostscript-gpl ghostscript-gnu I had two ports required ghostscript-gnu and portupgrade couldn't handle this situation with this command. I guess I also need -r or something; now I come to think about it. I will try -r later. > I do see your variable naming scheme as a cause for concern though, > unfortunately. I steer clear of shell-like configurations like this when > you can potentially get malicious text inside a variable name / declaration. I didn't like it, either. ;) Normalizing got so ugly, I gave up and decided to use them as it is. > >> 2. Does your solution account for cases when you're trying to install > >> package a, which depends on package b, but because you built package b > >> while trying to build a, and are at an intermediate package c (between a > >> and b), the dependencies for a are only partially complete. Thus when > >> you try to install direct or indirect dependences, the install fails? > >> > > > > Some of answers are explain in the GNUmakefile. Anyway, ports+ categorizes all ports into > > three, NEW as ones not installed, UPGRADE as installed and newer version is available, NOOP as > > installed an no newer version is available. I only started with upgrading only and haven't > > really though thought installing new ones. I am not ready to talk about installing new ports > > via ports+. Short answer for upgrade is, "Yes, it does." Here are some examples. Let's say > > 'a' depends on 'b'. Then, it builds 'b' first and than 'a'. If new version of 'b' has new > > requirement of 'c'. Making 'a' will result installing 'c' and than upgrading 'b.' If 'e' > > requires 'f', 'g', and 'h' and these are independent from each other, upgrading 'e' with -j 3 > > will start upgrading 'f', 'g', and 'h' at the same time and once all three are upgraded, it > > will start on 'e'. > That's really good thinking sir for single installs. Props to you for > that :). > >> 3. How is this different from pkg_version combined with similar scripts? > >> > > > > I wasn't aware of pkg_version, but it looks like it only tells if port is up to date or not. > > It doesn't tell anything about dependencies among ports, does it? If so, it doesn't provide > > enough information to upgrade. > Every time you run make install, portmaster, or portupgrade, there's > a lot of stuff going on behind the scenes. One of the things involved is > installing package information in /var/db/pkg. That's where pkg_version > gets its information from, along with portmaster (AFAIK) and the > makefiles for ports (for sure). I could tell that 'make install' does a lots of things looking at top(1) how quickly 'last pid' increases during that time. I sometime picked up that pkg_create and pkg_info were running during that time in top(1). I was also surprised that as I upgraded KDE or GNOME, pkg_delete started taking longer and longer as I reached to the end of packages. > It does provide enough info to tell whether or not something needs > to be upgraded, based on the output (<, =, or >). It doesn't tell you > the official target you're upgrading to, but it doesn't really need to > do that though. It tells 'which one' to upgrade but not 'in which order' to upgrade, doesn't it? > > The difference from other two is the following. Portupgrade and portmaster "reads" what's in > > ports, INDEX, and what's installed, +CONTENTS, "solves" dependencies itself, and "manages" > > upgrading port itself. Instead, ports+ "converts" INDEX and +CONTENTS to (GNU) Makefile and > > let the dependency expert solve the rest. > > > > > I do like the ideas of simplified makefiles, but I see dependencies > changing frequency, unfortunately, and if dependencies are changed or > touch(1)'ed, then things have to be rebuilt to make gmake happy :). Your > exercise in doing this is interesting though, nonetheless. I described differences in implementation. But now I see the biggest difference and that leads these differences in our opinions. The difference is the assumption. Ports+ assumes that dependencies are STATIC and if not, make it STATIC. In fact, 'make config' can change dependency of a port. However, once it is done, it stays STATIC. That is why you do these for parallel build with ports+. # gmake UPGRADE='$(BSDMAKE) config' NEW='$(BSDMAKE) config' all # gmake -j 3 all BSDMAKE_OPTIONS='CONFIG_DONE=1' Once every port is config-ed, every dependency is STATIC. (Although ports+ doesn't yet support for adjusting dependency graph for such changes.) As a result, ports+ only evaluates inter-dependencies only once for all upgrade at beginning unlike portupgrade. (how is portmaster, by the way?) INDEX file contains all information such as port-origin, package name, and run-depends. +CONTENTS files contain the same information for installed ports and awk program can sweep them all at once. Q. Is it safe to assume all dependencies are STATIC? A. Yes, it is. Q. Why? A. RUN_DEPENDS are written in Makefiles in the ports system. Therefore, unless Makefiles are changed, they remain the same. If Makefiles are modified, INDEX file needs to be adjusted accordingly. Ports+ required up-to-date INDEX file. Therefore, all dependencies are STATIC and only needs to be evaluated only ONCE. That is base of the design in ports+. Hiro
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070727214847.6708a918.ota>