From owner-svn-src-all@FreeBSD.ORG Thu Apr 24 18:20:06 2014 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0949CF3C for ; Thu, 24 Apr 2014 18:20:06 +0000 (UTC) Received: from mail-ie0-f170.google.com (mail-ie0-f170.google.com [209.85.223.170]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C173F1C39 for ; Thu, 24 Apr 2014 18:20:05 +0000 (UTC) Received: by mail-ie0-f170.google.com with SMTP id rd18so2784926iec.15 for ; Thu, 24 Apr 2014 11:19:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=akdzDwaxau/4KOVEMlt2SQ3z3rK+EkmISy8+TCxNwIw=; b=bx9ncAQt563lOtWZtYniMQemZG7XxyOTvNXy+3wIw/1tbOGW1gS7vYaAIJr0IAKBVV KuNgpi/DUGECGUfL/5kui75HaJkLW8S0rVw4lDbRw5AUsOWcasl4OXu5iNHIF07LVZ8H AKRO272frIjzkEuvUcn+tzarnQ9XNFgJfMo7gzaoFPZok1KPkDSQ5pkE2Brn7AcaLixU EM8+qkUGl2Y8XR871c0YISw8DC8T8Atd5C71XhrsODD/NmVssZm837LwLjmL5VZClKDQ wdlEzYiIx7A4kFG7XWGIgXFU74LfYclE9QNO4/RnRSGJOPj1U9zXFhPRGW7MZCHzjIbU XxKw== X-Gm-Message-State: ALoCoQkCI1nzamW5GHTWjrlUfbBIsJUBlPftOBcEOHvIziZUzGP+VpiWSqw5pO6COD5KDyM9qlW3 X-Received: by 10.42.86.196 with SMTP id v4mr3215705icl.62.1398363599669; Thu, 24 Apr 2014 11:19:59 -0700 (PDT) Received: from [10.0.0.119] (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPSA id fx1sm877818igd.1.2014.04.24.11.19.58 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 24 Apr 2014 11:19:59 -0700 (PDT) Sender: Warner Losh Content-Type: text/plain; charset=windows-1251 Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\)) Subject: Re: svn commit: r263778 - in head: bin lib lib/clang sbin share/mk usr.bin usr.sbin From: Warner Losh In-Reply-To: <1398363246.61646.73.camel@revolution.hippie.lan> Date: Thu, 24 Apr 2014 12:19:57 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: <201403262230.s2QMUdH6021943@svn.freebsd.org> <20140327181245.GA69977@stack.nl> <7A86F5E9-DBE9-4D3F-B166-C02F8386B722@FreeBSD.org> <1398086131.1124.371.camel@revolution.hippie.lan> <1398363246.61646.73.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1874) Cc: src-committers@FreeBSD.org, Jilles Tjoelker , svn-src-all@FreeBSD.org, Dimitry Andric , svn-src-head@FreeBSD.org, "Justin T. Gibbs" X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Apr 2014 18:20:06 -0000 The dependency would have to be on the artificial targets that are created, and could easily be done. .WAIT is good for things like libc where everything is dependent on it, but poor for other dependencies which would be expressed as individual dependencies. Warner On Apr 24, 2014, at 12:14 PM, Ian Lepore wrote: > I couldn't think of a way to do that, because the directory itself = isn't > the dependency. That is, whether one directory has a newer/older > timestamp than the other isn't a function of the build process. The > fact that someone else mentioned inserting .WAIT into the list is > something I took as a confirmation that the regular dependency = mechanism > wouldn't work (so maybe I didn't think about it hard enough). >=20 > -- Ian >=20 > On Thu, 2014-04-24 at 11:17 -0600, Justin T. Gibbs wrote: >> Why wouldn=92t we fix these with explicit subdir dependencies (e.g: = subdira: subdirb)? I=92m pretty sure I had this working with a hacked = up bsd.subdir.mk at a previous job, but my memory is hazy. >>=20 >> =97 >> Justin >>=20 >> On Apr 21, 2014, at 7:35 AM, Warner Losh wrote: >>=20 >>> (sorry for the top post) >>>=20 >>> This looks good to my eye. I=92d be tempted to toss in a comment = about the >>> __wait=3D.WAIT construct is due to the primitive nature of bmake=92s = parser >>> so it runs afoul of the .for/.if construction rules and this is = needed to expand >>> the for variable. It tripped me up when I looked at it, until I = recalled a comment >>> from similar code in NetBSD. >>>=20 >>> Warner >>>=20 >>> On Apr 21, 2014, at 7:15 AM, Ian Lepore wrote: >>>=20 >>>> On Thu, 2014-03-27 at 20:44 +0100, Dimitry Andric wrote: >>>>> On 27 Mar 2014, at 19:12, Jilles Tjoelker wrote: >>>>>> On Thu, Mar 27, 2014 at 11:05:00AM -0600, Warner Losh wrote: >>>>>>> On Mar 26, 2014, at 4:30 PM, Dimitry Andric = wrote: >>>>>>>> Author: dim >>>>>>>> Date: Wed Mar 26 22:30:38 2014 >>>>>>>> New Revision: 263778 >>>>>>>> URL: http://svnweb.freebsd.org/changeset/base/263778 >>>>>>=20 >>>>>>>> Log: >>>>>>>> Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to = process >>>>>>>> all the SUBDIR entries in parallel, instead of serially. Apply = this >>>>>>>> option to a selected number of Makefiles, which can greatly = speed up the >>>>>>>> build on multi-core machines, when using make -j. >>>>>>=20 >>>>>>>> This can be extended to more Makefiles later on, whenever they = are >>>>>>>> verified to work correctly with parallel building. >>>>>>=20 >>>>>>> Why not have this =91opt out=92 rather than =91opt in=92 like it = is now? Are >>>>>>> there any known bad dependencies this introduces? >>>>>>=20 >>>>>> I'm paranoid about build systems ;) It is easy to add = dependencies >>>>>> across directories and as long as directories are built in = sequence, >>>>>> nothing goes wrong. >>>>>>=20 >>>>>> In fact, I had enabled SUBDIR_PARALLEL in sys/modules/Makefile as = well, >>>>>> but this caused mysterious failures with some kernels such as = mips >>>>>> ADM5120. >>>>>=20 >>>>> There are a bunch of other parts that don't really like parallel = builds >>>>> at the moment. For example, gnu/usr.bin/binutils needs its = libraries >>>>> (libbfd.a, etc) built first, before it can link the programs. = Similar >>>>> for gnu/usr.bin/cc, which needs libiberty, libcpp, etc before = being able >>>>> to build the rest of gcc. >>>>>=20 >>>>> Most of these cases can hopefully be solved by adding .WAIT = targets at >>>>> strategic points in the SUBDIR lists, but this also needs a bit of = extra >>>>> logic in bsd.subdir.mk. >>>>>=20 >>>>> -Dimitry >>>>>=20 >>>>=20 >>>> It turns out I needed the .WAIT functionality to use = SUBDIR_PARALLEL for >>>> $work, so I came up with the attached, does this look okay to = commit? >>>>=20 >>>> -- Ian >>>>=20 >>>> diff -r 67802e319fc6 share/mk/bsd.subdir.mk >>>> --- a/share/mk/bsd.subdir.mk Sun Apr 20 21:01:07 2014 -0600 >>>> +++ b/share/mk/bsd.subdir.mk Mon Apr 21 06:59:37 2014 -0600 >>>> @@ -4,10 +4,10 @@ >>>> # The include file contains the default targets >>>> # for building subdirectories. >>>> # >>>> -# For all of the directories listed in the variable SUBDIRS, the >>>> +# For all of the directories listed in the variable SUBDIR, the >>>> # specified directory will be visited and the target made. There is >>>> # also a default target which allows the command "make subdir" = where >>>> -# subdir is any directory listed in the variable SUBDIRS. >>>> +# subdir is any directory listed in the variable SUBDIR. >>>> # >>>> # >>>> # +++ variables +++ >>>> @@ -42,7 +42,7 @@ distribute: >>>>=20 >>>> _SUBDIR: .USE >>>> .if defined(SUBDIR) && !empty(SUBDIR) && !defined(NO_SUBDIR) >>>> - @${_+_}for entry in ${SUBDIR}; do \ >>>> + @${_+_}for entry in ${SUBDIR:N.WAIT}; do \ >>>> if test -d ${.CURDIR}/$${entry}.${MACHINE_ARCH}; then \ >>>> ${ECHODIR} "=3D=3D=3D> = ${DIRPRFX}$${entry}.${MACHINE_ARCH} (${.TARGET:realinstall=3Dinstall})"; = \ >>>> edir=3D$${entry}.${MACHINE_ARCH}; \ >>>> @@ -57,7 +57,7 @@ distribute: >>>> done >>>> .endif >>>>=20 >>>> -${SUBDIR}: .PHONY >>>> +${SUBDIR:N.WAIT}: .PHONY >>>> ${_+_}@if test -d ${.TARGET}.${MACHINE_ARCH}; then \ >>>> cd ${.CURDIR}/${.TARGET}.${MACHINE_ARCH}; \ >>>> else \ >>>> @@ -65,13 +65,18 @@ distribute: >>>> fi; \ >>>> ${MAKE} all >>>>=20 >>>> +__wait=3D.WAIT >>>> .for __target in all all-man checkdpadd clean cleandepend cleandir = \ >>>> depend distribute lint maninstall manlint \ >>>> obj objlink realinstall regress tags \ >>>> ${SUBDIR_TARGETS} >>>> .ifdef SUBDIR_PARALLEL >>>> +__subdir_targets=3D >>>> .for __dir in ${SUBDIR} >>>> -${__target}: ${__target}_subdir_${__dir} >>>> +.if ${__wait} =3D=3D ${__dir} >>>> +__subdir_targets+=3D .WAIT >>>> +.else >>>> +__subdir_targets+=3D ${__target}_subdir_${__dir} >>>> ${__target}_subdir_${__dir}: .MAKE >>>> @${_+_}set -e; \ >>>> if test -d ${.CURDIR}/${__dir}.${MACHINE_ARCH}; then \ >>>> @@ -85,7 +90,9 @@ distribute: >>>> fi; \ >>>> ${MAKE} ${__target:realinstall=3Dinstall} \ >>>> DIRPRFX=3D${DIRPRFX}$$edir/ >>>> +.endif >>>> .endfor >>>> +${__target}: ${__subdir_targets} >>>> .else >>>> ${__target}: _SUBDIR >>>> .endif >>>=20 >>>=20 >>>=20 >>=20 >>=20 >=20 >=20