Date: Tue, 10 Nov 1998 22:40:56 -0600 (CST) From: Steve Price <sprice@hiwaay.net> To: Satoshi Asami <asami@FreeBSD.ORG> Cc: ports@FreeBSD.ORG, msmith@FreeBSD.ORG Subject: Re: auto ports build script Message-ID: <Pine.OSF.4.02.9811102152030.15738-100000@fly.HiWAAY.net> In-Reply-To: <199811110246.SAA13193@silvia.hip.berkeley.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 10 Nov 1998, Satoshi Asami wrote: # * - interactive ports like mgetty and ifmail will hang the # * build even when BATCH=yes is specified # # Really? I haven't had any trouble with mgetty and ifmail. I always # do my build with BATCH=yes. Do they check PACKAGE_BUILDING or something? Yes, both of these ports and a couple of more I've found since, need to add users that don't exist in the bindists already. # * - The chroot'd env with all the distfiles and around 340 # * packages already built the size is around 2GB. I tried # * to tweak the passes in the script to conserve space as # * much as possible by removing (cleaning) ports that aren't # * required any longer. # # I think this is great start, but there are too many things that could # break in subtle ways if you have a whole bunch of ports installed. I # still think the only way to fully automate it is to start with an # empty /usr/local and /usr/X11R6 every time. The added bonus of this # is that it will find all of the missing *_DEPEND lines. This is exactly what I tried to do - limit the number of ports installed at any given time. However, I tried to do it without all of the overhead that you describe below. There are really two major problems at hand: 1) Does a port build? 2) Does it play well with others? The first is handled by my script. I tried to make my first cut as fast as possible and with limited disk resources. I suppose it's intended use was to get near real-time feedback whether any changes committed to a port actually worked outside of the committer's environment. Even on my box (without the X ports and all of the interactive problems) I think I could make a sweep through the entire tree once every two days easy. The second is pretty easy to come by with a couple of simple tweeks to the script I provided you. Answering this question means installing every port we can prior to one the of interest and only removing ones that would result in conflicting files being installed. I have a 60% cut at this too. I got as far as generating the PLIST for every port and building a conflict map. The only problem was that on my P5-90 is took a shade over an hour to just generate the PLISTs. :( # How about something like this? # # Ingredients: # # (i) A tarball that contains an extracted bindist, tweaked /etc and # whatever others (like share/info/dir) that are needed, plus portcheckout Will do tomorrow. # (ii) A package of XFree86-3.3.2, but stripped down so it includes only the # minimum required for building other stuff (lib/lib*, bin/imake, # lib/X11/config, etc.) Granted the XFree86 port doesn't change much, but at over a million lines of code, building it from scratch each time sort of represents the extreme test of the rest of the system. If we just had a daily SNAP server this would make a lot of sense, since we wouldn't have to 'make world' from scratch ourselves. Of course this "daily SNAP" stuff doesn't make sense except for -current, since -stable is supposed to change much more slowly. # (iii) A place to copy fetched distfiles, initially empty # # (iv) A place to copy built packages, initially contains only (ii) These are easy enough and can be covered by the tarball I create for (i). # The steps: # # (1) Sort INDEX in reverse topological order, so dependencies will be # listed first This is not as easy as it sounds. I suppose with a judicious tweek here and there, the recurse routine could be made to spew out such a list. # (2) For each line in (1), do # # if package is not in (iv), then Here's where you lose me. I assumed the intent was to rebuild the package every time so that any changes since the previous build would be weighed in. If not, we still have to stick a stake in the ground and after N days we ignore the package we built previously and rebuild from scratch. I had set it up so that N was the time it took me to walk the entire tree. :) # (2a) set up chroot directory # # (2b) extract (i) into there # # (2c) portcheckout port # # (2d) for each dependency, do # # if package is in (iv), pkg_add it # # (2e) build package # # (2f) cp -RP distfiles to (iii) # # (2g) cp -RP packages to (iv) # # (2h) rm -rf chroot directory This seems offly excessive and seems to miss the biggest problem Justin and I had while working on the 3.0 release. Most users don't start from a clean tree. They install a couple, delete a couple, install a couple, ... ad nausium. Justin had a bunch more ports installed most of the time (due to my limited disk space) and he caught quite a few "port X doesn't play nice with port Y" problems. Maybe I'm missing something but I don't see how the method you describe above helps resolve these types of conflicts. # (3) Run clean-restricted or clean-cdrom on (iii) and (iv) This is not there yet, but I had intended on coding it when I had an entire first pass of the tree done. # By sorting INDEX, most dependencies should be caught in the (2d) # pkg_add. It is important to not set FOR_CDROM or NO_RESTRICTED while # going through the loop, so ports and packages will be built for # everything that is required. Those will be deleted in (3). Yes, I agree. This is done in the current script except for the removal of those ports which are restricted or not for CDROM, but this is easily fixed. # The key is how to parallelize and otherwise optimize the (2) loop. # (2f) and (2g) may need some help from bsd.port.mk, as (for instance) # if we just copy the whole distfiles directory over, there always could # be a case where two ports try to copy the same distfile over at the # same time. Funny you should mention this, but I've been playing around with some changes with bsd.port.mk (nothing breath-taking mind you), but it seems that just ignoring RUN_DEPENDS saves quite a bit of time in the package building stage. The reason that this works is because the gist of the builds we are doing is just to check that the build works and maybe pointing out any conflicts with other ports that we might see. Writing a script that builds 'em and then runs 'em is definitely beyond my abilities for now. :) # For speeding up (2), one thing we my want to do is to add an # "uncompressed" package type to pkg_*. Those packages should be much # faster to pkg_add. (They will also be faster to build, but that has # to be done sooner or later so it doesn't help the overall time.) Yes, this could help. It does add quite a bit to the storage space requirement however. My box still hasn't used up the 3GB partition that I gave it and I still have a full install, with all the distfiles, and over 600 packages now. :) - steve # I wish I can use the space we have here at school for this, we have 20 # FreeBSD machines with lots of disks (but they are nearly full). Maybe # I'll talk to my professor to make this our next research topic after # we find a way to move that data elsewhere. :) # # Satoshi # To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-ports" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.OSF.4.02.9811102152030.15738-100000>