Date: Mon, 23 Mar 2009 00:37:10 -0400 From: David Forsythe <dforsyth@FreeBSD.org> To: freebsd-ports@freebsd.org Subject: Parallel builds, PKG_DBDIR locks Message-ID: <dfaab3f70903222137t7fe2e0ddl989197d29fad7d77@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hey all, Last year for my Summer of Code project, I worked on adding parallel build support to ports. While it worked, some of it didn't work right, and I wasn't really happy with the final result. I finally got some free time lately and decided to clean it up a little bit, and since it's at a point where I can use it without fearing cataclysmic failure, I figured I'd share with anyone else who might be interested. What bsd.locked.mk allows you to do first and foremost is build more than one port at a time without fearing for your PKG_DBDIR or overwrites when ports with common dependencies build as long as you have the USE_LOCKS flag set. It creates a lock in a building ports directory that, if kept consistent, will prevent the port from being built more than once at a time. When it comes time for a port to register itself (fake-pkg), PKG_DBDIR itself is used as a lock to prevent inadvertent clobbering of the database. I also added a locking for fetches, so that if ports share distfiles (qt4 stuff, for instance), your builds won't overwrite the files or outright fail. With P_DEPENDS and P_FETCH, you have the option of parallel dependency handling and distfile fetching. P_DEPENDS=4, for instance, will allow a port to build up to 4 dependencies at a time. P_FETCH=3 will allow up to 3 distfiles of a port to be fetched at a time. I've generated a patch for anyone who wants to test this out. If the patch fails to apply cleanly, just grab the makefile and add the conditional for USE_LOCKS right before the master-sites-ALL target. If you don't have USE_LOCKS set, everything will work as if bsd.locked.mk isn't even there. I had a patch from last summer that made pkg_install and pkg_deinstall honor the locks as well, but I'm not sure where I put it, and it's probably no good at this point anyway. It was pretty simple, regardless, and I could redo it if anyone actually takes an interest in this. Run make config-recursive or use BATCH if you use this or else you might be some nasty suprises. patch: http://bsdtips.utcorp.net/~dforsyth/bsd.locked.mk.diff (apply it in your ports makefile directory) makefile: http://bsdtips.utcorp.net/~dforsyth/bsd.locked.mk Thanks, Dave -- David Forsythe
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?dfaab3f70903222137t7fe2e0ddl989197d29fad7d77>