Date: Tue, 10 Nov 1998 18:46:52 -0800 (PST) From: asami@FreeBSD.ORG (Satoshi Asami) To: sprice@hiwaay.net Cc: ports@FreeBSD.ORG, msmith@FreeBSD.ORG Subject: Re: auto ports build script Message-ID: <199811110246.SAA13193@silvia.hip.berkeley.edu> In-Reply-To: <Pine.OSF.4.02.9811101821060.30795-100000@fly.HiWAAY.net> (message from Steve Price on Tue, 10 Nov 1998 18:47:57 -0600 (CST))
next in thread | previous in thread | raw e-mail | index | archive | help
* Howdy guys,
I'm moving this to ports as this is not really at a release
engineering state. I'm copying msmith since I'm not sure if he's on
the ports list.
* Here's a synopsis of what I did:
*
* - grabbed copy of the bindists for 3.0-RELEASE
* - installed them in /tmp/foo
* - `touch /tmp/foo/usr/share/info/dir`
* - checked out a fresh copy of the ports tree
* - created the packages and distfiles directory
* - fetched all the distfiles I didn't already have
* - copied the attached script to /tmp/foo/usr/ports/build
* - `chroot /tmp/foo`
* - `cd /usr/ports`
* - `./build INDEX`
*
* As an interesting exercise try running it with a commandline
* like so:
*
* BATCH=yes ./build INDEX XFree86-3.3.2 perl-5.00502
*
* Anything after the first argument is the name of a port to
* exclude from the build process - including all the ports that
* depend on the ones you specify.
I see.
* Some quick observations:
*
* - the tk-* ports clearly need X libs to build but they
* don't specify USE_XLIB
Fixed. :)
* - interactive ports like mgetty and ifmail will hang the
* build even when BATCH=yes is specified
Really? I haven't had any trouble with mgetty and ifmail. I always
do my build with BATCH=yes. Do they check PACKAGE_BUILDING or something?
* - The chroot'd env with all the distfiles and around 340
* packages already built the size is around 2GB. I tried
* to tweak the passes in the script to conserve space as
* much as possible by removing (cleaning) ports that aren't
* required any longer.
I think this is great start, but there are too many things that could
break in subtle ways if you have a whole bunch of ports installed. I
still think the only way to fully automate it is to start with an
empty /usr/local and /usr/X11R6 every time. The added bonus of this
is that it will find all of the missing *_DEPEND lines.
How about something like this?
Ingredients:
(i) A tarball that contains an extracted bindist, tweaked /etc and
whatever others (like share/info/dir) that are needed, plus portcheckout
(ii) A package of XFree86-3.3.2, but stripped down so it includes only the
minimum required for building other stuff (lib/lib*, bin/imake,
lib/X11/config, etc.)
(iii) A place to copy fetched distfiles, initially empty
(iv) A place to copy built packages, initially contains only (ii)
The steps:
(1) Sort INDEX in reverse topological order, so dependencies will be
listed first
(2) For each line in (1), do
if package is not in (iv), then
(2a) set up chroot directory
(2b) extract (i) into there
(2c) portcheckout port
(2d) for each dependency, do
if package is in (iv), pkg_add it
(2e) build package
(2f) cp -RP distfiles to (iii)
(2g) cp -RP packages to (iv)
(2h) rm -rf chroot directory
(3) Run clean-restricted or clean-cdrom on (iii) and (iv)
By sorting INDEX, most dependencies should be caught in the (2d)
pkg_add. It is important to not set FOR_CDROM or NO_RESTRICTED while
going through the loop, so ports and packages will be built for
everything that is required. Those will be deleted in (3).
The key is how to parallelize and otherwise optimize the (2) loop.
(2f) and (2g) may need some help from bsd.port.mk, as (for instance)
if we just copy the whole distfiles directory over, there always could
be a case where two ports try to copy the same distfile over at the
same time.
For speeding up (2), one thing we my want to do is to add an
"uncompressed" package type to pkg_*. Those packages should be much
faster to pkg_add. (They will also be faster to build, but that has
to be done sooner or later so it doesn't help the overall time.)
I wish I can use the space we have here at school for this, we have 20
FreeBSD machines with lots of disks (but they are nearly full). Maybe
I'll talk to my professor to make this our next research topic after
we find a way to move that data elsewhere. :)
Satoshi
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-ports" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199811110246.SAA13193>