From owner-freebsd-ports Tue Nov 10 18:47:18 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id SAA25952 for freebsd-ports-outgoing; Tue, 10 Nov 1998 18:47:18 -0800 (PST) (envelope-from owner-freebsd-ports@FreeBSD.ORG) Received: from vader.cs.berkeley.edu (vader.CS.Berkeley.EDU [128.32.38.234]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id SAA25946; Tue, 10 Nov 1998 18:47:16 -0800 (PST) (envelope-from asami@vader.cs.berkeley.edu) Received: from silvia.hip.berkeley.edu (sji-ca11-07.ix.netcom.com [209.109.237.7]) by vader.cs.berkeley.edu (8.8.7/8.7.3) with ESMTP id SAA25539; Tue, 10 Nov 1998 18:46:55 -0800 (PST) Received: (from asami@localhost) by silvia.hip.berkeley.edu (8.8.8/8.6.9) id SAA13193; Tue, 10 Nov 1998 18:46:52 -0800 (PST) Date: Tue, 10 Nov 1998 18:46:52 -0800 (PST) Message-Id: <199811110246.SAA13193@silvia.hip.berkeley.edu> To: sprice@hiwaay.net CC: ports@FreeBSD.ORG, msmith@FreeBSD.ORG In-reply-to: (message from Steve Price on Tue, 10 Nov 1998 18:47:57 -0600 (CST)) Subject: Re: auto ports build script From: asami@FreeBSD.ORG (Satoshi Asami) Sender: owner-freebsd-ports@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org * Howdy guys, I'm moving this to ports as this is not really at a release engineering state. I'm copying msmith since I'm not sure if he's on the ports list. * Here's a synopsis of what I did: * * - grabbed copy of the bindists for 3.0-RELEASE * - installed them in /tmp/foo * - `touch /tmp/foo/usr/share/info/dir` * - checked out a fresh copy of the ports tree * - created the packages and distfiles directory * - fetched all the distfiles I didn't already have * - copied the attached script to /tmp/foo/usr/ports/build * - `chroot /tmp/foo` * - `cd /usr/ports` * - `./build INDEX` * * As an interesting exercise try running it with a commandline * like so: * * BATCH=yes ./build INDEX XFree86-3.3.2 perl-5.00502 * * Anything after the first argument is the name of a port to * exclude from the build process - including all the ports that * depend on the ones you specify. I see. * Some quick observations: * * - the tk-* ports clearly need X libs to build but they * don't specify USE_XLIB Fixed. :) * - interactive ports like mgetty and ifmail will hang the * build even when BATCH=yes is specified Really? I haven't had any trouble with mgetty and ifmail. I always do my build with BATCH=yes. Do they check PACKAGE_BUILDING or something? * - The chroot'd env with all the distfiles and around 340 * packages already built the size is around 2GB. I tried * to tweak the passes in the script to conserve space as * much as possible by removing (cleaning) ports that aren't * required any longer. I think this is great start, but there are too many things that could break in subtle ways if you have a whole bunch of ports installed. I still think the only way to fully automate it is to start with an empty /usr/local and /usr/X11R6 every time. The added bonus of this is that it will find all of the missing *_DEPEND lines. How about something like this? Ingredients: (i) A tarball that contains an extracted bindist, tweaked /etc and whatever others (like share/info/dir) that are needed, plus portcheckout (ii) A package of XFree86-3.3.2, but stripped down so it includes only the minimum required for building other stuff (lib/lib*, bin/imake, lib/X11/config, etc.) (iii) A place to copy fetched distfiles, initially empty (iv) A place to copy built packages, initially contains only (ii) The steps: (1) Sort INDEX in reverse topological order, so dependencies will be listed first (2) For each line in (1), do if package is not in (iv), then (2a) set up chroot directory (2b) extract (i) into there (2c) portcheckout port (2d) for each dependency, do if package is in (iv), pkg_add it (2e) build package (2f) cp -RP distfiles to (iii) (2g) cp -RP packages to (iv) (2h) rm -rf chroot directory (3) Run clean-restricted or clean-cdrom on (iii) and (iv) By sorting INDEX, most dependencies should be caught in the (2d) pkg_add. It is important to not set FOR_CDROM or NO_RESTRICTED while going through the loop, so ports and packages will be built for everything that is required. Those will be deleted in (3). The key is how to parallelize and otherwise optimize the (2) loop. (2f) and (2g) may need some help from bsd.port.mk, as (for instance) if we just copy the whole distfiles directory over, there always could be a case where two ports try to copy the same distfile over at the same time. For speeding up (2), one thing we my want to do is to add an "uncompressed" package type to pkg_*. Those packages should be much faster to pkg_add. (They will also be faster to build, but that has to be done sooner or later so it doesn't help the overall time.) I wish I can use the space we have here at school for this, we have 20 FreeBSD machines with lots of disks (but they are nearly full). Maybe I'll talk to my professor to make this our next research topic after we find a way to move that data elsewhere. :) Satoshi To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-ports" in the body of the message