From owner-svn-src-all@FreeBSD.ORG Wed Apr 9 18:17:02 2014 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 20C8D8F1; Wed, 9 Apr 2014 18:17:02 +0000 (UTC) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0A4D51B9B; Wed, 9 Apr 2014 18:17:02 +0000 (UTC) Received: from svn.freebsd.org ([127.0.1.70]) by svn.freebsd.org (8.14.8/8.14.8) with ESMTP id s39IH1to067625; Wed, 9 Apr 2014 18:17:01 GMT (envelope-from dim@svn.freebsd.org) Received: (from dim@localhost) by svn.freebsd.org (8.14.8/8.14.8/Submit) id s39IH0a3067606; Wed, 9 Apr 2014 18:17:00 GMT (envelope-from dim@svn.freebsd.org) Message-Id: <201404091817.s39IH0a3067606@svn.freebsd.org> From: Dimitry Andric Date: Wed, 9 Apr 2014 18:17:00 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-8@freebsd.org Subject: svn commit: r264303 - in stable: 10/bin 10/gnu/usr.bin 10/lib 10/lib/clang 10/sbin 10/share/mk 10/usr.bin 10/usr.bin/clang 10/usr.sbin 8/bin 8/gnu/usr.bin 8/lib 8/sbin 8/share/mk 8/usr.bin 8/usr.sb... X-SVN-Group: stable-8 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Apr 2014 18:17:02 -0000 Author: dim Date: Wed Apr 9 18:16:58 2014 New Revision: 264303 URL: http://svnweb.freebsd.org/changeset/base/264303 Log: MFC r263778: Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to process all the SUBDIR entries in parallel, instead of serially. Apply this option to a selected number of Makefiles, which can greatly speed up the build on multi-core machines, when using make -j. This can be extended to more Makefiles later on, whenever they are verified to work correctly with parallel building. I tested this on a 24-core machine, with make -j48 buildworld (N = 6): before stddev after stddev ======= ====== ======= ====== real time 1741.1 16.5 959.8 2.7 user time 12468.7 16.4 14393.0 16.8 sys time 1825.0 54.8 2110.6 22.8 (user+sys)/real 8.2 17.1 E.g. the build was approximately 45% faster in real time. On machines with less cores, or with lower -j settings, the speedup will not be as impressive. But at least you can now almost max out a machine with buildworld! Submitted by: jilles MFC r263833: Enable parallel building for gnu/usr.bin and usr.bin/clang too. Modified: stable/8/bin/Makefile stable/8/gnu/usr.bin/Makefile (contents, props changed) stable/8/lib/Makefile (contents, props changed) stable/8/sbin/Makefile stable/8/share/mk/bsd.subdir.mk stable/8/usr.bin/Makefile stable/8/usr.sbin/Makefile (contents, props changed) Directory Properties: stable/8/bin/ (props changed) stable/8/gnu/ (props changed) stable/8/gnu/usr.bin/ (props changed) stable/8/lib/ (props changed) stable/8/sbin/ (props changed) stable/8/share/ (props changed) stable/8/share/mk/ (props changed) stable/8/usr.bin/ (props changed) stable/8/usr.sbin/ (props changed) Changes in other areas also in this revision: Modified: stable/10/bin/Makefile stable/10/gnu/usr.bin/Makefile stable/10/lib/Makefile stable/10/lib/clang/Makefile stable/10/sbin/Makefile stable/10/share/mk/bsd.subdir.mk stable/10/usr.bin/Makefile stable/10/usr.bin/clang/Makefile stable/10/usr.sbin/Makefile stable/9/bin/Makefile stable/9/gnu/usr.bin/Makefile stable/9/lib/Makefile (contents, props changed) stable/9/lib/clang/Makefile stable/9/sbin/Makefile stable/9/share/mk/bsd.subdir.mk stable/9/usr.bin/Makefile stable/9/usr.bin/clang/Makefile stable/9/usr.sbin/Makefile (contents, props changed) Directory Properties: stable/10/ (props changed) stable/9/bin/ (props changed) stable/9/lib/ (props changed) stable/9/lib/clang/ (props changed) stable/9/sbin/ (props changed) stable/9/share/ (props changed) stable/9/share/mk/ (props changed) stable/9/usr.bin/ (props changed) stable/9/usr.bin/clang/ (props changed) stable/9/usr.sbin/ (props changed) Modified: stable/8/bin/Makefile ============================================================================== --- stable/8/bin/Makefile Wed Apr 9 14:50:55 2014 (r264302) +++ stable/8/bin/Makefile Wed Apr 9 18:16:58 2014 (r264303) @@ -54,4 +54,6 @@ _rmail= rmail _csh= csh .endif +SUBDIR_PARALLEL= + .include Modified: stable/8/gnu/usr.bin/Makefile ============================================================================== --- stable/8/gnu/usr.bin/Makefile Wed Apr 9 14:50:55 2014 (r264302) +++ stable/8/gnu/usr.bin/Makefile Wed Apr 9 18:16:58 2014 (r264303) @@ -63,4 +63,6 @@ _cc= cc _gdb= gdb .endif +SUBDIR_PARALLEL= + .include Modified: stable/8/lib/Makefile ============================================================================== --- stable/8/lib/Makefile Wed Apr 9 14:50:55 2014 (r264302) +++ stable/8/lib/Makefile Wed Apr 9 18:16:58 2014 (r264303) @@ -203,4 +203,8 @@ _libusbhid= libusbhid _libusb= libusb .endif +.if !make(install) +SUBDIR_PARALLEL= +.endif + .include Modified: stable/8/sbin/Makefile ============================================================================== --- stable/8/sbin/Makefile Wed Apr 9 14:50:55 2014 (r264302) +++ stable/8/sbin/Makefile Wed Apr 9 18:16:58 2014 (r264303) @@ -154,4 +154,6 @@ _mca= mca _sunlabel= sunlabel .endif +SUBDIR_PARALLEL= + .include Modified: stable/8/share/mk/bsd.subdir.mk ============================================================================== --- stable/8/share/mk/bsd.subdir.mk Wed Apr 9 14:50:55 2014 (r264302) +++ stable/8/share/mk/bsd.subdir.mk Wed Apr 9 18:16:58 2014 (r264303) @@ -69,7 +69,26 @@ ${SUBDIR}: .PHONY depend distribute lint maninstall manlint \ obj objlink realinstall regress tags \ ${SUBDIR_TARGETS} +.ifdef SUBDIR_PARALLEL +.for __dir in ${SUBDIR} +${__target}: ${__target}_subdir_${__dir} +${__target}_subdir_${__dir}: .MAKE + @${_+_}set -e; \ + if test -d ${.CURDIR}/${__dir}.${MACHINE_ARCH}; then \ + ${ECHODIR} "===> ${DIRPRFX}${__dir}.${MACHINE_ARCH} (${__target:realinstall=install})"; \ + edir=${__dir}.${MACHINE_ARCH}; \ + cd ${.CURDIR}/$${edir}; \ + else \ + ${ECHODIR} "===> ${DIRPRFX}${__dir} (${__target:realinstall=install})"; \ + edir=${__dir}; \ + cd ${.CURDIR}/$${edir}; \ + fi; \ + ${MAKE} ${__target:realinstall=install} \ + DIRPRFX=${DIRPRFX}$$edir/ +.endfor +.else ${__target}: _SUBDIR +.endif .endfor .for __target in files includes Modified: stable/8/usr.bin/Makefile ============================================================================== --- stable/8/usr.bin/Makefile Wed Apr 9 14:50:55 2014 (r264302) +++ stable/8/usr.bin/Makefile Wed Apr 9 18:16:58 2014 (r264303) @@ -393,4 +393,6 @@ _smbutil= smbutil _smbutil= smbutil .endif +SUBDIR_PARALLEL= + .include Modified: stable/8/usr.sbin/Makefile ============================================================================== --- stable/8/usr.sbin/Makefile Wed Apr 9 14:50:55 2014 (r264302) +++ stable/8/usr.sbin/Makefile Wed Apr 9 18:16:58 2014 (r264303) @@ -520,4 +520,6 @@ _eeprom= eeprom _ofwdump= ofwdump .endif +SUBDIR_PARALLEL= + .include