From owner-svn-src-all@FreeBSD.ORG Mon Apr 21 13:15:44 2014 Return-Path: Delivered-To: svn-src-all@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 26321490; Mon, 21 Apr 2014 13:15:44 +0000 (UTC) Received: from mho-02-ewr.mailhop.org (mho-02-ewr.mailhop.org [204.13.248.72]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D262F167E; Mon, 21 Apr 2014 13:15:43 +0000 (UTC) Received: from c-24-8-230-52.hsd1.co.comcast.net ([24.8.230.52] helo=damnhippie.dyndns.org) by mho-02-ewr.mailhop.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1WcE4N-0007Kp-Ji; Mon, 21 Apr 2014 13:15:35 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id s3LDFV2O007367; Mon, 21 Apr 2014 07:15:31 -0600 (MDT) (envelope-from ian@FreeBSD.org) X-Mail-Handler: Dyn Standard SMTP by Dyn X-Originating-IP: 24.8.230.52 X-Report-Abuse-To: abuse@dyndns.com (see http://www.dyndns.com/services/sendlabs/outbound_abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX1/gZ4yqjHMLFlxGBGR1fAN2 Subject: Re: svn commit: r263778 - in head: bin lib lib/clang sbin share/mk usr.bin usr.sbin From: Ian Lepore To: Dimitry Andric In-Reply-To: <7A86F5E9-DBE9-4D3F-B166-C02F8386B722@FreeBSD.org> References: <201403262230.s2QMUdH6021943@svn.freebsd.org> <20140327181245.GA69977@stack.nl> <7A86F5E9-DBE9-4D3F-B166-C02F8386B722@FreeBSD.org> Content-Type: multipart/mixed; boundary="=-5mNCT3uufMi0VnEJyCeB" Date: Mon, 21 Apr 2014 07:15:31 -0600 Message-ID: <1398086131.1124.371.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Cc: svn-src-head@FreeBSD.org, svn-src-all@FreeBSD.org, src-committers@FreeBSD.org, Warner Losh , Jilles Tjoelker X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Apr 2014 13:15:44 -0000 --=-5mNCT3uufMi0VnEJyCeB Content-Type: text/plain; charset="iso-8859-7" Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by damnhippie.dyndns.org id s3LDFV2O007367 On Thu, 2014-03-27 at 20:44 +0100, Dimitry Andric wrote: > On 27 Mar 2014, at 19:12, Jilles Tjoelker wrote: > > On Thu, Mar 27, 2014 at 11:05:00AM -0600, Warner Losh wrote: > >> On Mar 26, 2014, at 4:30 PM, Dimitry Andric wrote: > >>> Author: dim > >>> Date: Wed Mar 26 22:30:38 2014 > >>> New Revision: 263778 > >>> URL: http://svnweb.freebsd.org/changeset/base/263778 > >=20 > >>> Log: > >>> Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to pro= cess > >>> all the SUBDIR entries in parallel, instead of serially. Apply thi= s > >>> option to a selected number of Makefiles, which can greatly speed u= p the > >>> build on multi-core machines, when using make -j. > >=20 > >>> This can be extended to more Makefiles later on, whenever they are > >>> verified to work correctly with parallel building. > >=20 > >> Why not have this =A1opt out=A2 rather than =A1opt in=A2 like it is = now? Are > >> there any known bad dependencies this introduces? > >=20 > > I'm paranoid about build systems ;) It is easy to add dependencies > > across directories and as long as directories are built in sequence, > > nothing goes wrong. > >=20 > > In fact, I had enabled SUBDIR_PARALLEL in sys/modules/Makefile as wel= l, > > but this caused mysterious failures with some kernels such as mips > > ADM5120. >=20 > There are a bunch of other parts that don't really like parallel builds > at the moment. For example, gnu/usr.bin/binutils needs its libraries > (libbfd.a, etc) built first, before it can link the programs. Similar > for gnu/usr.bin/cc, which needs libiberty, libcpp, etc before being abl= e > to build the rest of gcc. >=20 > Most of these cases can hopefully be solved by adding .WAIT targets at > strategic points in the SUBDIR lists, but this also needs a bit of extr= a > logic in bsd.subdir.mk. >=20 > -Dimitry > =20 It turns out I needed the .WAIT functionality to use SUBDIR_PARALLEL for $work, so I came up with the attached, does this look okay to commit? -- Ian --=-5mNCT3uufMi0VnEJyCeB Content-Disposition: inline; filename="parallel_make_wait.diff" Content-Type: text/x-patch; name="parallel_make_wait.diff"; charset="us-ascii" Content-Transfer-Encoding: 7bit diff -r 67802e319fc6 share/mk/bsd.subdir.mk --- a/share/mk/bsd.subdir.mk Sun Apr 20 21:01:07 2014 -0600 +++ b/share/mk/bsd.subdir.mk Mon Apr 21 06:59:37 2014 -0600 @@ -4,10 +4,10 @@ # The include file contains the default targets # for building subdirectories. # -# For all of the directories listed in the variable SUBDIRS, the +# For all of the directories listed in the variable SUBDIR, the # specified directory will be visited and the target made. There is # also a default target which allows the command "make subdir" where -# subdir is any directory listed in the variable SUBDIRS. +# subdir is any directory listed in the variable SUBDIR. # # # +++ variables +++ @@ -42,7 +42,7 @@ distribute: _SUBDIR: .USE .if defined(SUBDIR) && !empty(SUBDIR) && !defined(NO_SUBDIR) - @${_+_}for entry in ${SUBDIR}; do \ + @${_+_}for entry in ${SUBDIR:N.WAIT}; do \ if test -d ${.CURDIR}/$${entry}.${MACHINE_ARCH}; then \ ${ECHODIR} "===> ${DIRPRFX}$${entry}.${MACHINE_ARCH} (${.TARGET:realinstall=install})"; \ edir=$${entry}.${MACHINE_ARCH}; \ @@ -57,7 +57,7 @@ distribute: done .endif -${SUBDIR}: .PHONY +${SUBDIR:N.WAIT}: .PHONY ${_+_}@if test -d ${.TARGET}.${MACHINE_ARCH}; then \ cd ${.CURDIR}/${.TARGET}.${MACHINE_ARCH}; \ else \ @@ -65,13 +65,18 @@ distribute: fi; \ ${MAKE} all +__wait=.WAIT .for __target in all all-man checkdpadd clean cleandepend cleandir \ depend distribute lint maninstall manlint \ obj objlink realinstall regress tags \ ${SUBDIR_TARGETS} .ifdef SUBDIR_PARALLEL +__subdir_targets= .for __dir in ${SUBDIR} -${__target}: ${__target}_subdir_${__dir} +.if ${__wait} == ${__dir} +__subdir_targets+= .WAIT +.else +__subdir_targets+= ${__target}_subdir_${__dir} ${__target}_subdir_${__dir}: .MAKE @${_+_}set -e; \ if test -d ${.CURDIR}/${__dir}.${MACHINE_ARCH}; then \ @@ -85,7 +90,9 @@ distribute: fi; \ ${MAKE} ${__target:realinstall=install} \ DIRPRFX=${DIRPRFX}$$edir/ +.endif .endfor +${__target}: ${__subdir_targets} .else ${__target}: _SUBDIR .endif --=-5mNCT3uufMi0VnEJyCeB--