From owner-freebsd-ports@FreeBSD.ORG Mon Mar 23 05:00:19 2009 Return-Path: Delivered-To: freebsd-ports@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BD9FD10656CF for ; Mon, 23 Mar 2009 05:00:19 +0000 (UTC) (envelope-from dforsythe@gmail.com) Received: from mail-gx0-f176.google.com (mail-gx0-f176.google.com [209.85.217.176]) by mx1.freebsd.org (Postfix) with ESMTP id 79C2E8FC37 for ; Mon, 23 Mar 2009 05:00:19 +0000 (UTC) (envelope-from dforsythe@gmail.com) Received: by gxk24 with SMTP id 24so6993277gxk.19 for ; Sun, 22 Mar 2009 22:00:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:date :x-google-sender-auth:message-id:subject:from:to:content-type; bh=NjuSpCGui9lUy5lovUAxwgYAiFOsZce9X6MeG8cVOvw=; b=lR9duEgt7nVgZuAyvrDyBSQcvuJReAVyXxZ4ELtUrVub2waMASpA652lbcEpXZaxuW iWzZTUuosXQxSg8vdQFtoCbmWeGqRvSVErDkPETNBDGHOnspFvKx9zDT1OJulJi3fl1l +y1uEjJjD9H8aYEIxelQXXkv9wPXUmVvncQ8c= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:content-type; b=asnqNZaF6PZLdjoCji+TjSd5CdKdsfIY88KlvDlHf8ql5a7193c0gb0WNftX/ZSJVd PD7+hn9mATvurKOb8sU23XWcA6V9YwxqYrmx9Ha/BcV0Etr2WZxVtOsX3QPMDA2OOZ8q sVCFIPcmFBIVe5jwQBewxF8pa6k4P3MICBVr4= MIME-Version: 1.0 Sender: dforsythe@gmail.com Received: by 10.220.98.202 with SMTP id r10mr2359398vcn.67.1237783030780; Sun, 22 Mar 2009 21:37:10 -0700 (PDT) Date: Mon, 23 Mar 2009 00:37:10 -0400 X-Google-Sender-Auth: 0c5e775f647f208a Message-ID: From: David Forsythe To: freebsd-ports@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Parallel builds, PKG_DBDIR locks X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2009 05:00:20 -0000 Hey all, Last year for my Summer of Code project, I worked on adding parallel build support to ports. While it worked, some of it didn't work right, and I wasn't really happy with the final result. I finally got some free time lately and decided to clean it up a little bit, and since it's at a point where I can use it without fearing cataclysmic failure, I figured I'd share with anyone else who might be interested. What bsd.locked.mk allows you to do first and foremost is build more than one port at a time without fearing for your PKG_DBDIR or overwrites when ports with common dependencies build as long as you have the USE_LOCKS flag set. It creates a lock in a building ports directory that, if kept consistent, will prevent the port from being built more than once at a time. When it comes time for a port to register itself (fake-pkg), PKG_DBDIR itself is used as a lock to prevent inadvertent clobbering of the database. I also added a locking for fetches, so that if ports share distfiles (qt4 stuff, for instance), your builds won't overwrite the files or outright fail. With P_DEPENDS and P_FETCH, you have the option of parallel dependency handling and distfile fetching. P_DEPENDS=4, for instance, will allow a port to build up to 4 dependencies at a time. P_FETCH=3 will allow up to 3 distfiles of a port to be fetched at a time. I've generated a patch for anyone who wants to test this out. If the patch fails to apply cleanly, just grab the makefile and add the conditional for USE_LOCKS right before the master-sites-ALL target. If you don't have USE_LOCKS set, everything will work as if bsd.locked.mk isn't even there. I had a patch from last summer that made pkg_install and pkg_deinstall honor the locks as well, but I'm not sure where I put it, and it's probably no good at this point anyway. It was pretty simple, regardless, and I could redo it if anyone actually takes an interest in this. Run make config-recursive or use BATCH if you use this or else you might be some nasty suprises. patch: http://bsdtips.utcorp.net/~dforsyth/bsd.locked.mk.diff (apply it in your ports makefile directory) makefile: http://bsdtips.utcorp.net/~dforsyth/bsd.locked.mk Thanks, Dave -- David Forsythe