Date: Fri, 27 Jul 2007 01:47:14 -0700 From: Garrett Cooper <youshi10@u.washington.edu> To: Yoshihiro Ota <ota@j.email.ne.jp> Cc: freebsd-ports@freebsd.org Subject: Re: Call for testers for yet another ports upgrade program, ports+ Message-ID: <46A9B112.7040408@u.washington.edu> In-Reply-To: <20070726233610.e536c2e2.ota@j.email.ne.jp> References: <20070726011654.cec378be.ota@j.email.ne.jp> <46A866BE.1000407@u.washington.edu> <20070726233610.e536c2e2.ota@j.email.ne.jp>
next in thread | previous in thread | raw e-mail | index | archive | help
Yoshihiro Ota wrote: > On Thu, 26 Jul 2007 02:17:50 -0700 > Garrett Cooper <youshi10@u.washington.edu> wrote: > > >> Yoshihiro Ota wrote: >> >>> To Whom Slowness of Portupgrade Concerns a Lot: >>> >>> As I got tired of long waiting of portupgrade trying to resolve dependencies, I came up with >>> yet another tool for upgrading FreeBSD ports system. Unlink other tools, it tries to maximize >>> existing resource to maximize its performance. This program attempts to wrap around with >>> another 'make' and expand use of FreeBSD ports system. The heart of ports+ is parsing INDEX >>> and +CONTENTS files. The rest is handed to GNU make. I think it comes to a point where I seek >>> wider audiences to test with it. >>> >>> >> Hiro, >> You're correct when you say that reading INDEX* does take a long time >> for portupgrade, but the problem is partly with how portupgrade formats >> its version of the INDEX database in BDB format, as well as how it >> doesn't buffer up proper information (pkg database information for >> instance), and guesses at port dependencies (origins, deporigins, etc). >> > > Portupgrade is not only slow reading INDEX file but also on dependency resolving and updating +CONTENTS files, too. I think portmaster is also one tries to read and do the same things but with shell script. I personally didn't have good luck with portmaster and haven't really used to evaluate. However, "portmaster -a -n" wasn't not fast, neither. By the way, it builds ports in background, doesn't it? > Yes, but a lot of time is spent reading the INDEX file, especially since portupgrade / portinstall don't keep that information in memory (probably for the better though). Like Doug said, portmaster doesn't built ports in the background, and the way that the package system is currently setup I don't advise doing that unless you plan everything out properly (I see potential for duplicated installs, race conditions, and other unwanted possibilities). I'm working on getting away from that at the moment. > >> 1. Can your solution be made bsdmake and bsd awk compatible? >> > > I am not as familiar in newer BSD awk and make extensions. I can point out what kinds of GNU extensions are being used so that someone can tell it is also available in BSD make/awk. > > For awk, this feature is required, quoted from "man gawk". I am not sure if BSD awk implements this feature. > > The book indicates that command line variable assignment happens when > awk would otherwise open the argument as a file, which is after the > BEGIN block is executed. However, in earlier implementations, when > such an assignment appeared before any file names, the assignment would > happen before the BEGIN block was run. Applications came to depend on > this "feature." When awk was changed to match its documentation, the > -v option for assigning variables before program execution was added to > accommodate applications that depended upon the old behavior. (This > feature was agreed upon by both the Bell Laboratories and the GNU > developers.) > > For make, 1.1 and 1.2 below are in the same topic. I use GNU extension which keeps generating makefiles until dependency rule saticifies and use it part of the make rule. GNU extension reloads newer makefiles and reconstruct dependencies. I am not sure if it is possible in BSD make; however, this is not a heart of this program. We can work around with bootstrap script or something. > 1.3 This is recursive expansion of variables. If you take a look, every port needs upgrade uses $(UPGRADE). $(UPGRADE) is expanded multiple times using the target variable, $@, such that it backups up correct old package names and so on. This is very important and unless this works in BSD make, my tool really needed full redesign from scratch. > 1.4 is just how GNU make does shell invocations. It is available in most of Make tools. > 1.5 is something I am not sure which makes support these. I don't think GUN make recommends using special characters as variable names. However, when I tried, it was too difficult to normalize to alphabets and underscore only. If we cannot use variable like PORTS+IGNORE as a variable name, it will be very difficult and personally I don't even think about trying that myself. If you take a look at compat.gmk which ports+ generates as a part of its build rule, you can find something like "COMPAT_$(PKG_DIR)Hermes-1.3.3_2 = cp -f /usr/local/lib/libHermes.so.1 /usr/local/lib/compat/pkg&&" being variable. > > 1.1 How makefiles get remade: > http://uw713doc.sco.com/cgi-bin/info2html?(make.info)Remaking%2520Makefiles&lang=en > > 1.2 Including Other Makefiles: > http://uw713doc.sco.com/cgi-bin/info2html?(make.info)Include&lang=en > > 1.3 Computed Variable Names or recursive expandsion of variables: > http://uw713doc.sco.com/cgi-bin/info2html?(make.info)Computed%2520Names&lang=en > > 1.4 The `shell' Function > http://uw713doc.sco.com/cgi-bin/info2html?(make.info)Shell%2520Function&lang=en > > 1.5 Special characters in variable names such as dot, ., comma, ,, slash, /, and so on. > Ah, good ole SCO.. I can't believe they're posting docs like that. Lol. I do see your variable naming scheme as a cause for concern though, unfortunately. I steer clear of shell-like configurations like this when you can potentially get malicious text inside a variable name / declaration. >> 2. Does your solution account for cases when you're trying to install >> package a, which depends on package b, but because you built package b >> while trying to build a, and are at an intermediate package c (between a >> and b), the dependencies for a are only partially complete. Thus when >> you try to install direct or indirect dependences, the install fails? >> > > Some of answers are explain in the GNUmakefile. Anyway, ports+ categorizes all ports into three, NEW as ones not installed, UPGRADE as installed and newer version is available, NOOP as installed an no newer version is available. I only started with upgrading only and haven't really though thought installing new ones. I am not ready to talk about installing new ports via ports+. Short answer for upgrade is, "Yes, it does." Here are some examples. Let's say 'a' depends on 'b'. Then, it builds 'b' first and than 'a'. If new version of 'b' has new requirement of 'c'. Making 'a' will result installing 'c' and than upgrading 'b.' If 'e' requires 'f', 'g', and 'h' and these are independent from each other, upgrading 'e' with -j 3 will start upgrading 'f', 'g', and 'h' at the same time and once all three are upgraded, it will start on 'e'. > That's really good thinking sir for single installs. Props to you for that :). >> 3. How is this different from pkg_version combined with similar scripts? >> > > I wasn't aware of pkg_version, but it looks like it only tells if port is up to date or not. It doesn't tell anything about dependencies among ports, does it? If so, it doesn't provide enough information to upgrade. > Every time you run make install, portmaster, or portupgrade, there's a lot of stuff going on behind the scenes. One of the things involved is installing package information in /var/db/pkg. That's where pkg_version gets its information from, along with portmaster (AFAIK) and the makefiles for ports (for sure). It does provide enough info to tell whether or not something needs to be upgraded, based on the output (<, =, or >). It doesn't tell you the official target you're upgrading to, but it doesn't really need to do that though. > The difference from other two is the following. Portupgrade and portmaster "reads" what's in ports, INDEX, and what's installed, +CONTENTS, "solves" dependencies itself, and "manages" upgrading port itself. Instead, ports+ "converts" INDEX and +CONTENTS to (GNU) Makefile and let the dependency expert solve the rest. > > Hiro > I do like the ideas of simplified makefiles, but I see dependencies changing frequency, unfortunately, and if dependencies are changed or touch(1)'ed, then things have to be rebuilt to make gmake happy :). Your exercise in doing this is interesting though, nonetheless. -Garrett
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?46A9B112.7040408>