Date: Thu, 26 Jul 2007 23:36:10 -0400 From: Yoshihiro Ota <ota@j.email.ne.jp> To: Garrett Cooper <youshi10@u.washington.edu>, freebsd-ports@freebsd.org Subject: Re: Call for testers for yet another ports upgrade program, ports+ Message-ID: <20070726233610.e536c2e2.ota@j.email.ne.jp> In-Reply-To: <46A866BE.1000407@u.washington.edu> References: <20070726011654.cec378be.ota@j.email.ne.jp> <46A866BE.1000407@u.washington.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 26 Jul 2007 02:17:50 -0700
Garrett Cooper <youshi10@u.washington.edu> wrote:
> Yoshihiro Ota wrote:
> > To Whom Slowness of Portupgrade Concerns a Lot:
> >
> > As I got tired of long waiting of portupgrade trying to resolve dependencies, I came up with
> > yet another tool for upgrading FreeBSD ports system. Unlink other tools, it tries to maximize
> > existing resource to maximize its performance. This program attempts to wrap around with
> > another 'make' and expand use of FreeBSD ports system. The heart of ports+ is parsing INDEX
> > and +CONTENTS files. The rest is handed to GNU make. I think it comes to a point where I seek
> > wider audiences to test with it.
> >
>
> Hiro,
> You're correct when you say that reading INDEX* does take a long time
> for portupgrade, but the problem is partly with how portupgrade formats
> its version of the INDEX database in BDB format, as well as how it
> doesn't buffer up proper information (pkg database information for
> instance), and guesses at port dependencies (origins, deporigins, etc).
Portupgrade is not only slow reading INDEX file but also on dependency resolving and updating +CONTENTS files, too. I think portmaster is also one tries to read and do the same things but with shell script. I personally didn't have good luck with portmaster and haven't really used to evaluate. However, "portmaster -a -n" wasn't not fast, neither. By the way, it builds ports in background, doesn't it?
> 1. Can your solution be made bsdmake and bsd awk compatible?
I am not as familiar in newer BSD awk and make extensions. I can point out what kinds of GNU extensions are being used so that someone can tell it is also available in BSD make/awk.
For awk, this feature is required, quoted from "man gawk". I am not sure if BSD awk implements this feature.
The book indicates that command line variable assignment happens when
awk would otherwise open the argument as a file, which is after the
BEGIN block is executed. However, in earlier implementations, when
such an assignment appeared before any file names, the assignment would
happen before the BEGIN block was run. Applications came to depend on
this "feature." When awk was changed to match its documentation, the
-v option for assigning variables before program execution was added to
accommodate applications that depended upon the old behavior. (This
feature was agreed upon by both the Bell Laboratories and the GNU
developers.)
For make, 1.1 and 1.2 below are in the same topic. I use GNU extension which keeps generating makefiles until dependency rule saticifies and use it part of the make rule. GNU extension reloads newer makefiles and reconstruct dependencies. I am not sure if it is possible in BSD make; however, this is not a heart of this program. We can work around with bootstrap script or something.
1.3 This is recursive expansion of variables. If you take a look, every port needs upgrade uses $(UPGRADE). $(UPGRADE) is expanded multiple times using the target variable, $@, such that it backups up correct old package names and so on. This is very important and unless this works in BSD make, my tool really needed full redesign from scratch.
1.4 is just how GNU make does shell invocations. It is available in most of Make tools.
1.5 is something I am not sure which makes support these. I don't think GUN make recommends using special characters as variable names. However, when I tried, it was too difficult to normalize to alphabets and underscore only. If we cannot use variable like PORTS+IGNORE as a variable name, it will be very difficult and personally I don't even think about trying that myself. If you take a look at compat.gmk which ports+ generates as a part of its build rule, you can find something like "COMPAT_$(PKG_DIR)Hermes-1.3.3_2 = cp -f /usr/local/lib/libHermes.so.1 /usr/local/lib/compat/pkg&&" being variable.
1.1 How makefiles get remade:
http://uw713doc.sco.com/cgi-bin/info2html?(make.info)Remaking%2520Makefiles&lang=en
1.2 Including Other Makefiles:
http://uw713doc.sco.com/cgi-bin/info2html?(make.info)Include&lang=en
1.3 Computed Variable Names or recursive expandsion of variables:
http://uw713doc.sco.com/cgi-bin/info2html?(make.info)Computed%2520Names&lang=en
1.4 The `shell' Function
http://uw713doc.sco.com/cgi-bin/info2html?(make.info)Shell%2520Function&lang=en
1.5 Special characters in variable names such as dot, ., comma, ,, slash, /, and so on.
> 2. Does your solution account for cases when you're trying to install
> package a, which depends on package b, but because you built package b
> while trying to build a, and are at an intermediate package c (between a
> and b), the dependencies for a are only partially complete. Thus when
> you try to install direct or indirect dependences, the install fails?
Some of answers are explain in the GNUmakefile. Anyway, ports+ categorizes all ports into three, NEW as ones not installed, UPGRADE as installed and newer version is available, NOOP as installed an no newer version is available. I only started with upgrading only and haven't really though thought installing new ones. I am not ready to talk about installing new ports via ports+. Short answer for upgrade is, "Yes, it does." Here are some examples. Let's say 'a' depends on 'b'. Then, it builds 'b' first and than 'a'. If new version of 'b' has new requirement of 'c'. Making 'a' will result installing 'c' and than upgrading 'b.' If 'e' requires 'f', 'g', and 'h' and these are independent from each other, upgrading 'e' with -j 3 will start upgrading 'f', 'g', and 'h' at the same time and once all three are upgraded, it will start on 'e'.
> 3. How is this different from pkg_version combined with similar scripts?
I wasn't aware of pkg_version, but it looks like it only tells if port is up to date or not. It doesn't tell anything about dependencies among ports, does it? If so, it doesn't provide enough information to upgrade.
The difference from other two is the following. Portupgrade and portmaster "reads" what's in ports, INDEX, and what's installed, +CONTENTS, "solves" dependencies itself, and "manages" upgrading port itself. Instead, ports+ "converts" INDEX and +CONTENTS to (GNU) Makefile and let the dependency expert solve the rest.
Hiro
> -Garrett
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070726233610.e536c2e2.ota>