From owner-svn-src-all@FreeBSD.ORG Wed Apr 9 18:17:00 2014 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7F2B48E5; Wed, 9 Apr 2014 18:17:00 +0000 (UTC) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 68FC71B9A; Wed, 9 Apr 2014 18:17:00 +0000 (UTC) Received: from svn.freebsd.org ([127.0.1.70]) by svn.freebsd.org (8.14.8/8.14.8) with ESMTP id s39IH0Y4067599; Wed, 9 Apr 2014 18:17:00 GMT (envelope-from dim@svn.freebsd.org) Received: (from dim@localhost) by svn.freebsd.org (8.14.8/8.14.8/Submit) id s39IGwcR067582; Wed, 9 Apr 2014 18:16:58 GMT (envelope-from dim@svn.freebsd.org) Message-Id: <201404091816.s39IGwcR067582@svn.freebsd.org> From: Dimitry Andric Date: Wed, 9 Apr 2014 18:16:58 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r264303 - in stable: 10/bin 10/gnu/usr.bin 10/lib 10/lib/clang 10/sbin 10/share/mk 10/usr.bin 10/usr.bin/clang 10/usr.sbin 8/bin 8/gnu/usr.bin 8/lib 8/sbin 8/share/mk 8/usr.bin 8/usr.sb... X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Apr 2014 18:17:00 -0000 Author: dim Date: Wed Apr 9 18:16:58 2014 New Revision: 264303 URL: http://svnweb.freebsd.org/changeset/base/264303 Log: MFC r263778: Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to process all the SUBDIR entries in parallel, instead of serially. Apply this option to a selected number of Makefiles, which can greatly speed up the build on multi-core machines, when using make -j. This can be extended to more Makefiles later on, whenever they are verified to work correctly with parallel building. I tested this on a 24-core machine, with make -j48 buildworld (N = 6): before stddev after stddev ======= ====== ======= ====== real time 1741.1 16.5 959.8 2.7 user time 12468.7 16.4 14393.0 16.8 sys time 1825.0 54.8 2110.6 22.8 (user+sys)/real 8.2 17.1 E.g. the build was approximately 45% faster in real time. On machines with less cores, or with lower -j settings, the speedup will not be as impressive. But at least you can now almost max out a machine with buildworld! Submitted by: jilles MFC r263833: Enable parallel building for gnu/usr.bin and usr.bin/clang too. Modified: stable/9/bin/Makefile stable/9/gnu/usr.bin/Makefile stable/9/lib/Makefile (contents, props changed) stable/9/lib/clang/Makefile stable/9/sbin/Makefile stable/9/share/mk/bsd.subdir.mk stable/9/usr.bin/Makefile stable/9/usr.bin/clang/Makefile stable/9/usr.sbin/Makefile (contents, props changed) Directory Properties: stable/9/bin/ (props changed) stable/9/lib/ (props changed) stable/9/lib/clang/ (props changed) stable/9/sbin/ (props changed) stable/9/share/ (props changed) stable/9/share/mk/ (props changed) stable/9/usr.bin/ (props changed) stable/9/usr.bin/clang/ (props changed) stable/9/usr.sbin/ (props changed) Changes in other areas also in this revision: Modified: stable/10/bin/Makefile stable/10/gnu/usr.bin/Makefile stable/10/lib/Makefile stable/10/lib/clang/Makefile stable/10/sbin/Makefile stable/10/share/mk/bsd.subdir.mk stable/10/usr.bin/Makefile stable/10/usr.bin/clang/Makefile stable/10/usr.sbin/Makefile stable/8/bin/Makefile stable/8/gnu/usr.bin/Makefile (contents, props changed) stable/8/lib/Makefile (contents, props changed) stable/8/sbin/Makefile stable/8/share/mk/bsd.subdir.mk stable/8/usr.bin/Makefile stable/8/usr.sbin/Makefile (contents, props changed) Directory Properties: stable/10/ (props changed) stable/8/bin/ (props changed) stable/8/gnu/ (props changed) stable/8/gnu/usr.bin/ (props changed) stable/8/lib/ (props changed) stable/8/sbin/ (props changed) stable/8/share/ (props changed) stable/8/share/mk/ (props changed) stable/8/usr.bin/ (props changed) stable/8/usr.sbin/ (props changed) Modified: stable/9/bin/Makefile ============================================================================== --- stable/9/bin/Makefile Wed Apr 9 14:50:55 2014 (r264302) +++ stable/9/bin/Makefile Wed Apr 9 18:16:58 2014 (r264303) @@ -55,4 +55,6 @@ SUBDIR+= csh SUBDIR:= ${SUBDIR:O} +SUBDIR_PARALLEL= + .include Modified: stable/9/gnu/usr.bin/Makefile ============================================================================== --- stable/9/gnu/usr.bin/Makefile Wed Apr 9 14:50:55 2014 (r264302) +++ stable/9/gnu/usr.bin/Makefile Wed Apr 9 18:16:58 2014 (r264303) @@ -53,4 +53,6 @@ _cc= cc _gdb= gdb .endif +SUBDIR_PARALLEL= + .include Modified: stable/9/lib/Makefile ============================================================================== --- stable/9/lib/Makefile Wed Apr 9 14:50:55 2014 (r264302) +++ stable/9/lib/Makefile Wed Apr 9 18:16:58 2014 (r264303) @@ -240,4 +240,8 @@ _libusbhid= libusbhid _libusb= libusb .endif +.if !make(install) +SUBDIR_PARALLEL= +.endif + .include Modified: stable/9/lib/clang/Makefile ============================================================================== --- stable/9/lib/clang/Makefile Wed Apr 9 14:50:55 2014 (r264302) +++ stable/9/lib/clang/Makefile Wed Apr 9 18:16:58 2014 (r264303) @@ -101,4 +101,6 @@ SUBDIR+=libllvmdebuginfo \ SUBDIR+= include +SUBDIR_PARALLEL= + .include Modified: stable/9/sbin/Makefile ============================================================================== --- stable/9/sbin/Makefile Wed Apr 9 14:50:55 2014 (r264302) +++ stable/9/sbin/Makefile Wed Apr 9 18:16:58 2014 (r264303) @@ -113,4 +113,6 @@ SUBDIR+= routed SUBDIR:= ${SUBDIR:O} +SUBDIR_PARALLEL= + .include Modified: stable/9/share/mk/bsd.subdir.mk ============================================================================== --- stable/9/share/mk/bsd.subdir.mk Wed Apr 9 14:50:55 2014 (r264302) +++ stable/9/share/mk/bsd.subdir.mk Wed Apr 9 18:16:58 2014 (r264303) @@ -68,7 +68,26 @@ ${SUBDIR}: .PHONY .for __target in all all-man checkdpadd clean cleandepend cleandir \ cleanilinks depend distribute lint maninstall manlint obj objlink \ realinstall regress tags ${SUBDIR_TARGETS} +.ifdef SUBDIR_PARALLEL +.for __dir in ${SUBDIR} +${__target}: ${__target}_subdir_${__dir} +${__target}_subdir_${__dir}: .MAKE + @${_+_}set -e; \ + if test -d ${.CURDIR}/${__dir}.${MACHINE_ARCH}; then \ + ${ECHODIR} "===> ${DIRPRFX}${__dir}.${MACHINE_ARCH} (${__target:realinstall=install})"; \ + edir=${__dir}.${MACHINE_ARCH}; \ + cd ${.CURDIR}/$${edir}; \ + else \ + ${ECHODIR} "===> ${DIRPRFX}${__dir} (${__target:realinstall=install})"; \ + edir=${__dir}; \ + cd ${.CURDIR}/$${edir}; \ + fi; \ + ${MAKE} ${__target:realinstall=install} \ + DIRPRFX=${DIRPRFX}$$edir/ +.endfor +.else ${__target}: _SUBDIR +.endif .endfor .for __target in files includes Modified: stable/9/usr.bin/Makefile ============================================================================== --- stable/9/usr.bin/Makefile Wed Apr 9 14:50:55 2014 (r264302) +++ stable/9/usr.bin/Makefile Wed Apr 9 18:16:58 2014 (r264303) @@ -351,4 +351,6 @@ SUBDIR+= wtmpcvt SUBDIR:= ${SUBDIR:O} +SUBDIR_PARALLEL= + .include Modified: stable/9/usr.bin/clang/Makefile ============================================================================== --- stable/9/usr.bin/clang/Makefile Wed Apr 9 14:50:55 2014 (r264302) +++ stable/9/usr.bin/clang/Makefile Wed Apr 9 18:16:58 2014 (r264303) @@ -23,4 +23,6 @@ SUBDIR+=bugpoint \ opt .endif +SUBDIR_PARALLEL= + .include Modified: stable/9/usr.sbin/Makefile ============================================================================== --- stable/9/usr.sbin/Makefile Wed Apr 9 14:50:55 2014 (r264302) +++ stable/9/usr.sbin/Makefile Wed Apr 9 18:16:58 2014 (r264303) @@ -335,4 +335,6 @@ SUBDIR+= wpa SUBDIR:= ${SUBDIR:O} +SUBDIR_PARALLEL= + .include