Date: Wed, 11 Nov 1998 08:45:28 -0600 (CST) From: Steve Price <sprice@hiwaay.net> To: Satoshi Asami <asami@FreeBSD.ORG> Cc: ports@FreeBSD.ORG, msmith@FreeBSD.ORG Subject: Re: auto ports build script Message-ID: <Pine.OSF.4.02.9811110801400.15738-100000@fly.HiWAAY.net> In-Reply-To: <199811110907.BAA23346@silvia.hip.berkeley.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, 11 Nov 1998, Satoshi Asami wrote:
# * 1) Does a port build?
# * 2) Does it play well with others?
#
# Actually what I had in mind is not to find out 2, but to build the
# ports without running into problems caused by 2.
Yes, 2) make things complicated but we shouldn't ignore it.
With Mike's promise to find the resources we need, we shouldn't
have a problem addressing 2) in a separate build tree.
# * The second is pretty easy to come by with a couple of simple
# * tweeks to the script I provided you. Answering this question
# * means installing every port we can prior to one the of interest
# * and only removing ones that would result in conflicting files
# * being installed. I have a 60% cut at this too. I got as far
# * as generating the PLIST for every port and building a conflict
# * map. The only problem was that on my P5-90 is took a shade
# * over an hour to just generate the PLISTs. :(
#
# Wow. That is "easy"? ;)
#
# What are you going to do with that information though? Say, I did
# what you mentioned above, recompiled port foo, and it failed with
# syntax error in line 694. What now?
You send out an error Email to ports-fixers@freebsd.org? :)
Seriously what comes next can be one of quite a number of
things, but the easiest answer to just move onto the next
port.
# I don't object to building the whole XFree86 for testing purposes.
# The reason I wanted to keep the package I mentioned small is because
# it is used (via pkg_add) for every port that requires X.
Ah, it is amazing what the dawn of a new day can do to the
fog in one's head. I'm finally beginning to see what you are
getting at. The really cool part is that the script you
already have does what you describe without all the pkg_add
business. Given a fresh chroot'd environment the algorithm goes
something like this:
for each line in INDEX file
save dependencies for this port
for each port
build an inverse dependency list, aka a required by list
for each port
if this port doesn't depend on any other port then
recurse(port)
And recurse() looks like this:
if this port has been visited before then
return
set visited to 1
for each port this one depends on
recurse(port)
install, package this port
for each port that requires this port
recurse(port)
deinstall this port
All this in 157 lines of perl code that should be pretty much
immune to changes to bsd.port*.mk. You also get a nifty
exclude feature to boot - like excluding XFree86-3.3.2 and
all the ports that depend on it. :)
# * # (iii) A place to copy fetched distfiles, initially empty
# * #
# * # (iv) A place to copy built packages, initially contains only (ii)
# *
# * These are easy enough and can be covered by the tarball I create
# * for (i).
#
# No, actually these are outside the chroot area. Maybe even on a
# different machine. The chroot area has empty /usr/ports/distfiles and
# /usr/ports/packages, and they will be copied to (iii) and (iv) in
# later steps (2f and 2g in my original mail).
Now I see said the blind man. All I have to do is copy it out
of the chroot'd environment. The only difference between what
you are describing is when this is done. If I really understand
your proposed method, you build up, copy out the good stuff, and
break down the chroot'd env for each port. What I gave you was
something that built up the tree once, simulated the one dependency
tree approach in between, and then copied everything at the very
end. This latter approach should be magnitudes faster, take
the same amount of total space (maybe a little more in a few
corner cases), and accomplishes the same thing. :)
# * # The steps:
# * #
# * # (1) Sort INDEX in reverse topological order, so dependencies will be
# * # listed first
# *
# * This is not as easy as it sounds. I suppose with a judicious tweek
# * here and there, the recurse routine could be made to spew out such a
# * list.
#
# Our dependencies are guaranteed to be acyclic, so all we need to do is
# create a dependency tree and print out nodes in post-order. This is
# easy (at least theoretically). I can even imagine writing an awk
# script to turn INDEX into a mini-Makefile that is fed into make and
# have it print out the list we want. After all, topological sort is
# make's speciality.
This is exactly what my little perl script does, except it doesn't
just print it out. It does the real work during the post-order
tranversal.
# In fact, I just did. :)
#
# ===
# awk -F '|' '{me=$1; here=$2; bdep=$8; rdep=$9; print "all: " me; print me ": " bdep " " rdep; printf("\t@echo %s\n", here)}' /usr/ports/INDEX
# ===
#
# I'm not sure if it is correct (it's a 2-minute hack job) but you get
# the idea.
I'll play with this a little more later on today. Pretty impressive
if this does in one long line of awk what took me 157 lines of perl. ;)
# Sorry. Yes, I mean we always start from scratch, we update /usr/ports
# once and build all packages without updating it again. (That's why
# (iii) and (iv) are initially empty.) I don't think it can be made to
# work if we have it constantly keep updating the stuff, especially in
# light of non-redistributable stuff. (See my other reply to Jordan on
# this.)
Yes, walking the whole tree once, updating, and starting over
was what I was proposing too.
# There are also problems with distfiles. Say, I have a port that's
# updated, and now it uses distfile B instead of A. Can I delete A?
# Not so fast -- it might be used in some other port. I can do and do a
# grep in the entire */*/files/md5, but that's not very fast. Also,
# once I do a "cvs update", it's not easy to find out what A was in the
# first place. (Same for package names.)
Mike Smith's pib can help out on this front.
# That's true. Maybe I should make bsd.port.mk ignore RUN_DEPENDS when
# PACKAGE_BUILDING is set.
This would be nice. Changing one line in the lps (little perl
script :) would make the whole thing much faster and cleaner
if this were implemented.
# I don't think the storage space for packages is that much of a
# problem, it's just a few gigs even if they are uncompressed. The
# temporary storage for work dirs is much more of a problem as it could
# go up to over a dozen gigs. (Depending on the parallelism of the 2
# loop in my original mail, it could be even higher.)
This was the reason that I put in the exclude function and why
I was considering putting in an include function. So you could
split the build in half simply by doing the following in two
different envs.
./build INDEX -x XFree86-3.3.2 perl-5.00502
./build INDEX -i XFree86-3.3.2 perl-5.00502
The first builds everything but the X and perl related stuff
and the latter does the rest.
# I think the problem is that we are talking two different things here.
# What I'm trying to come up here is a script that can rebuild the
# entire packages set in one or two days without any manual
# intervention. If a port can be built from an empty /usr/local, a
# package of it should be built by the automated script.
We're talking about addressing the same problem. I mentioned
2) above because that is something that I didn't address in the
script I gave you. We just have different approaches to solving
it. Imagine that. ;)
# Finding conflicts between ports is really another beast altogether.
# We know about most of them anyway (gtk, qt, ncurses, etc.). The issue
# on that front IMO is how to fix them. (And I've been asking
# maintainers to fix them forever, without much success.)
Yes conflict resolution is generally not something that can be
automated. I was working on a script that would find them and
ping the appropriate maintainer when the script found a problem -
kind of like Peter's automated-PR reminder stuff.
Steve
# Satoshi
#
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-ports" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.OSF.4.02.9811110801400.15738-100000>