From owner-svn-src-all@FreeBSD.ORG Thu Apr 24 18:14:16 2014 Return-Path: Delivered-To: svn-src-all@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 637E9ABC; Thu, 24 Apr 2014 18:14:16 +0000 (UTC) Received: from mho-02-ewr.mailhop.org (mho-02-ewr.mailhop.org [204.13.248.72]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1C6321BD8; Thu, 24 Apr 2014 18:14:15 +0000 (UTC) Received: from c-24-8-230-52.hsd1.co.comcast.net ([24.8.230.52] helo=damnhippie.dyndns.org) by mho-02-ewr.mailhop.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1WdO9x-0001lB-0c; Thu, 24 Apr 2014 18:14:09 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id s3OIE6J1010838; Thu, 24 Apr 2014 12:14:06 -0600 (MDT) (envelope-from ian@FreeBSD.org) X-Mail-Handler: Dyn Standard SMTP by Dyn X-Originating-IP: 24.8.230.52 X-Report-Abuse-To: abuse@dyndns.com (see http://www.dyndns.com/services/sendlabs/outbound_abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX19rhB2pAZ3NuhtJy9ilOvlF Subject: Re: svn commit: r263778 - in head: bin lib lib/clang sbin share/mk usr.bin usr.sbin From: Ian Lepore To: "Justin T. Gibbs" In-Reply-To: References: <201403262230.s2QMUdH6021943@svn.freebsd.org> <20140327181245.GA69977@stack.nl> <7A86F5E9-DBE9-4D3F-B166-C02F8386B722@FreeBSD.org> <1398086131.1124.371.camel@revolution.hippie.lan> Content-Type: text/plain; charset="windows-1251" Date: Thu, 24 Apr 2014 12:14:06 -0600 Message-ID: <1398363246.61646.73.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by damnhippie.dyndns.org id s3OIE6J1010838 Cc: src-committers@FreeBSD.org, Jilles Tjoelker , svn-src-all@FreeBSD.org, Dimitry Andric , svn-src-head@FreeBSD.org, Warner Losh X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Apr 2014 18:14:16 -0000 I couldn't think of a way to do that, because the directory itself isn't the dependency. That is, whether one directory has a newer/older timestamp than the other isn't a function of the build process. The fact that someone else mentioned inserting .WAIT into the list is something I took as a confirmation that the regular dependency mechanism wouldn't work (so maybe I didn't think about it hard enough). -- Ian On Thu, 2014-04-24 at 11:17 -0600, Justin T. Gibbs wrote: > Why wouldn=92t we fix these with explicit subdir dependencies (e.g: sub= dira: subdirb)? I=92m pretty sure I had this working with a hacked up bs= d.subdir.mk at a previous job, but my memory is hazy. >=20 > =97 > Justin > =20 > On Apr 21, 2014, at 7:35 AM, Warner Losh wrote: >=20 > > (sorry for the top post) > >=20 > > This looks good to my eye. I=92d be tempted to toss in a comment abou= t the > > __wait=3D.WAIT construct is due to the primitive nature of bmake=92s = parser > > so it runs afoul of the .for/.if construction rules and this is neede= d to expand > > the for variable. It tripped me up when I looked at it, until I recal= led a comment > > from similar code in NetBSD. > >=20 > > Warner > >=20 > > On Apr 21, 2014, at 7:15 AM, Ian Lepore wrote: > >=20 > >> On Thu, 2014-03-27 at 20:44 +0100, Dimitry Andric wrote: > >>> On 27 Mar 2014, at 19:12, Jilles Tjoelker wrote: > >>>> On Thu, Mar 27, 2014 at 11:05:00AM -0600, Warner Losh wrote: > >>>>> On Mar 26, 2014, at 4:30 PM, Dimitry Andric wro= te: > >>>>>> Author: dim > >>>>>> Date: Wed Mar 26 22:30:38 2014 > >>>>>> New Revision: 263778 > >>>>>> URL: http://svnweb.freebsd.org/changeset/base/263778 > >>>>=20 > >>>>>> Log: > >>>>>> Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to = process > >>>>>> all the SUBDIR entries in parallel, instead of serially. Apply = this > >>>>>> option to a selected number of Makefiles, which can greatly spee= d up the > >>>>>> build on multi-core machines, when using make -j. > >>>>=20 > >>>>>> This can be extended to more Makefiles later on, whenever they a= re > >>>>>> verified to work correctly with parallel building. > >>>>=20 > >>>>> Why not have this =91opt out=92 rather than =91opt in=92 like it = is now? Are > >>>>> there any known bad dependencies this introduces? > >>>>=20 > >>>> I'm paranoid about build systems ;) It is easy to add dependencies > >>>> across directories and as long as directories are built in sequenc= e, > >>>> nothing goes wrong. > >>>>=20 > >>>> In fact, I had enabled SUBDIR_PARALLEL in sys/modules/Makefile as = well, > >>>> but this caused mysterious failures with some kernels such as mips > >>>> ADM5120. > >>>=20 > >>> There are a bunch of other parts that don't really like parallel bu= ilds > >>> at the moment. For example, gnu/usr.bin/binutils needs its librari= es > >>> (libbfd.a, etc) built first, before it can link the programs. Simi= lar > >>> for gnu/usr.bin/cc, which needs libiberty, libcpp, etc before being= able > >>> to build the rest of gcc. > >>>=20 > >>> Most of these cases can hopefully be solved by adding .WAIT targets= at > >>> strategic points in the SUBDIR lists, but this also needs a bit of = extra > >>> logic in bsd.subdir.mk. > >>>=20 > >>> -Dimitry > >>>=20 > >>=20 > >> It turns out I needed the .WAIT functionality to use SUBDIR_PARALLEL= for > >> $work, so I came up with the attached, does this look okay to commit= ? > >>=20 > >> -- Ian > >>=20 > >> diff -r 67802e319fc6 share/mk/bsd.subdir.mk > >> --- a/share/mk/bsd.subdir.mk Sun Apr 20 21:01:07 2014 -0600 > >> +++ b/share/mk/bsd.subdir.mk Mon Apr 21 06:59:37 2014 -0600 > >> @@ -4,10 +4,10 @@ > >> # The include file contains the default targets > >> # for building subdirectories. > >> # > >> -# For all of the directories listed in the variable SUBDIRS, the > >> +# For all of the directories listed in the variable SUBDIR, the > >> # specified directory will be visited and the target made. There is > >> # also a default target which allows the command "make subdir" where > >> -# subdir is any directory listed in the variable SUBDIRS. > >> +# subdir is any directory listed in the variable SUBDIR. > >> # > >> # > >> # +++ variables +++ > >> @@ -42,7 +42,7 @@ distribute: > >>=20 > >> _SUBDIR: .USE > >> .if defined(SUBDIR) && !empty(SUBDIR) && !defined(NO_SUBDIR) > >> - @${_+_}for entry in ${SUBDIR}; do \ > >> + @${_+_}for entry in ${SUBDIR:N.WAIT}; do \ > >> if test -d ${.CURDIR}/$${entry}.${MACHINE_ARCH}; then \ > >> ${ECHODIR} "=3D=3D=3D> ${DIRPRFX}$${entry}.${MACHINE_ARCH} (${.TA= RGET:realinstall=3Dinstall})"; \ > >> edir=3D$${entry}.${MACHINE_ARCH}; \ > >> @@ -57,7 +57,7 @@ distribute: > >> done > >> .endif > >>=20 > >> -${SUBDIR}: .PHONY > >> +${SUBDIR:N.WAIT}: .PHONY > >> ${_+_}@if test -d ${.TARGET}.${MACHINE_ARCH}; then \ > >> cd ${.CURDIR}/${.TARGET}.${MACHINE_ARCH}; \ > >> else \ > >> @@ -65,13 +65,18 @@ distribute: > >> fi; \ > >> ${MAKE} all > >>=20 > >> +__wait=3D.WAIT > >> .for __target in all all-man checkdpadd clean cleandepend cleandir \ > >> depend distribute lint maninstall manlint \ > >> obj objlink realinstall regress tags \ > >> ${SUBDIR_TARGETS} > >> .ifdef SUBDIR_PARALLEL > >> +__subdir_targets=3D > >> .for __dir in ${SUBDIR} > >> -${__target}: ${__target}_subdir_${__dir} > >> +.if ${__wait} =3D=3D ${__dir} > >> +__subdir_targets+=3D .WAIT > >> +.else > >> +__subdir_targets+=3D ${__target}_subdir_${__dir} > >> ${__target}_subdir_${__dir}: .MAKE > >> @${_+_}set -e; \ > >> if test -d ${.CURDIR}/${__dir}.${MACHINE_ARCH}; then \ > >> @@ -85,7 +90,9 @@ distribute: > >> fi; \ > >> ${MAKE} ${__target:realinstall=3Dinstall} \ > >> DIRPRFX=3D${DIRPRFX}$$edir/ > >> +.endif > >> .endfor > >> +${__target}: ${__subdir_targets} > >> .else > >> ${__target}: _SUBDIR > >> .endif > >=20 > >=20 > >=20 >=20 >=20