Date: Wed, 11 Nov 1998 01:07:29 -0800 (PST) From: asami@FreeBSD.ORG (Satoshi Asami) To: sprice@hiwaay.net Cc: ports@FreeBSD.ORG, msmith@FreeBSD.ORG Subject: Re: auto ports build script Message-ID: <199811110907.BAA23346@silvia.hip.berkeley.edu> In-Reply-To: <Pine.OSF.4.02.9811102152030.15738-100000@fly.HiWAAY.net> (message from Steve Price on Tue, 10 Nov 1998 22:40:56 -0600 (CST))
next in thread | previous in thread | raw e-mail | index | archive | help
* # Really? I haven't had any trouble with mgetty and ifmail. I always
* # do my build with BATCH=yes. Do they check PACKAGE_BUILDING or something?
*
* Yes, both of these ports and a couple of more I've found since,
* need to add users that don't exist in the bindists already.
Ooh. That should be done by pw via pkg/INSTALL. (Otherwise the
packages won't work....)
* This is exactly what I tried to do - limit the number of ports
* installed at any given time. However, I tried to do it without
* all of the overhead that you describe below. There are really
* two major problems at hand:
*
* 1) Does a port build?
* 2) Does it play well with others?
Actually what I had in mind is not to find out 2, but to build the
ports without running into problems caused by 2.
* The second is pretty easy to come by with a couple of simple
* tweeks to the script I provided you. Answering this question
* means installing every port we can prior to one the of interest
* and only removing ones that would result in conflicting files
* being installed. I have a 60% cut at this too. I got as far
* as generating the PLIST for every port and building a conflict
* map. The only problem was that on my P5-90 is took a shade
* over an hour to just generate the PLISTs. :(
Wow. That is "easy"? ;)
What are you going to do with that information though? Say, I did
what you mentioned above, recompiled port foo, and it failed with
syntax error in line 694. What now?
* Granted the XFree86 port doesn't change much, but at over a million
* lines of code, building it from scratch each time sort of represents
* the extreme test of the rest of the system. If we just had a daily
* SNAP server this would make a lot of sense, since we wouldn't have
* to 'make world' from scratch ourselves. Of course this "daily
* SNAP" stuff doesn't make sense except for -current, since -stable
* is supposed to change much more slowly.
I don't object to building the whole XFree86 for testing purposes.
The reason I wanted to keep the package I mentioned small is because
it is used (via pkg_add) for every port that requires X.
* # (iii) A place to copy fetched distfiles, initially empty
* #
* # (iv) A place to copy built packages, initially contains only (ii)
*
* These are easy enough and can be covered by the tarball I create
* for (i).
No, actually these are outside the chroot area. Maybe even on a
different machine. The chroot area has empty /usr/ports/distfiles and
/usr/ports/packages, and they will be copied to (iii) and (iv) in
later steps (2f and 2g in my original mail).
* # The steps:
* #
* # (1) Sort INDEX in reverse topological order, so dependencies will be
* # listed first
*
* This is not as easy as it sounds. I suppose with a judicious tweek
* here and there, the recurse routine could be made to spew out such a
* list.
Our dependencies are guaranteed to be acyclic, so all we need to do is
create a dependency tree and print out nodes in post-order. This is
easy (at least theoretically). I can even imagine writing an awk
script to turn INDEX into a mini-Makefile that is fed into make and
have it print out the list we want. After all, topological sort is
make's speciality.
In fact, I just did. :)
===
awk -F '|' '{me=$1; here=$2; bdep=$8; rdep=$9; print "all: " me; print me ": " bdep " " rdep; printf("\t@echo %s\n", here)}' /usr/ports/INDEX
===
I'm not sure if it is correct (it's a 2-minute hack job) but you get
the idea.
* # (2) For each line in (1), do
* #
* # if package is not in (iv), then
*
* Here's where you lose me. I assumed the intent was to rebuild
* the package every time so that any changes since the previous
* build would be weighed in. If not, we still have to stick a
* stake in the ground and after N days we ignore the package we
* built previously and rebuild from scratch. I had set it up so
* that N was the time it took me to walk the entire tree. :)
Sorry. Yes, I mean we always start from scratch, we update /usr/ports
once and build all packages without updating it again. (That's why
(iii) and (iv) are initially empty.) I don't think it can be made to
work if we have it constantly keep updating the stuff, especially in
light of non-redistributable stuff. (See my other reply to Jordan on
this.)
There are also problems with distfiles. Say, I have a port that's
updated, and now it uses distfile B instead of A. Can I delete A?
Not so fast -- it might be used in some other port. I can do and do a
grep in the entire */*/files/md5, but that's not very fast. Also,
once I do a "cvs update", it's not easy to find out what A was in the
first place. (Same for package names.)
* This seems offly excessive and seems to miss the biggest
* problem Justin and I had while working on the 3.0 release.
* Most users don't start from a clean tree. They install a
* couple, delete a couple, install a couple, ... ad nausium.
* Justin had a bunch more ports installed most of the time
* (due to my limited disk space) and he caught quite a few
* "port X doesn't play nice with port Y" problems. Maybe I'm
* missing something but I don't see how the method you describe
* above helps resolve these types of conflicts.
As I should have said, it doesn't. :) That's not the problem I'm
trying to solve. I've been marking them MANUAL_PACKAGE_BUILD for a
long time and I've just about finished marking the entire tree. (Ok,
maybe not.) Finding them is not really the problem, fixing them is,
but that's not the problem I'm trying to solve here.
* Funny you should mention this, but I've been playing around with
* some changes with bsd.port.mk (nothing breath-taking mind you),
* but it seems that just ignoring RUN_DEPENDS saves quite a bit
* of time in the package building stage. The reason that this works
* is because the gist of the builds we are doing is just to check
* that the build works and maybe pointing out any conflicts with
* other ports that we might see. Writing a script that builds 'em
* and then runs 'em is definitely beyond my abilities for now. :)
That's true. Maybe I should make bsd.port.mk ignore RUN_DEPENDS when
PACKAGE_BUILDING is set.
* Yes, this could help. It does add quite a bit to the storage space
* requirement however. My box still hasn't used up the 3GB partition
* that I gave it and I still have a full install, with all the
* distfiles, and over 600 packages now. :)
I don't think the storage space for packages is that much of a
problem, it's just a few gigs even if they are uncompressed. The
temporary storage for work dirs is much more of a problem as it could
go up to over a dozen gigs. (Depending on the parallelism of the 2
loop in my original mail, it could be even higher.)
I think the problem is that we are talking two different things here.
What I'm trying to come up here is a script that can rebuild the
entire packages set in one or two days without any manual
intervention. If a port can be built from an empty /usr/local, a
package of it should be built by the automated script.
Finding conflicts between ports is really another beast altogether.
We know about most of them anyway (gtk, qt, ncurses, etc.). The issue
on that front IMO is how to fix them. (And I've been asking
maintainers to fix them forever, without much success.)
Satoshi
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-ports" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199811110907.BAA23346>