From owner-freebsd-current@freebsd.org Mon Apr 3 18:38:59 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 39ACDD2CAFD for ; Mon, 3 Apr 2017 18:38:59 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-210-79.reflexion.net [208.70.210.79]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EF0486EB for ; Mon, 3 Apr 2017 18:38:58 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 16683 invoked from network); 3 Apr 2017 18:41:40 -0000 Received: from unknown (HELO mail-cs-02.app.dca.reflexion.local) (10.81.19.2) by 0 (rfx-qmail) with SMTP; 3 Apr 2017 18:41:40 -0000 Received: by mail-cs-02.app.dca.reflexion.local (Reflexion email security v8.40.0) with SMTP; Mon, 03 Apr 2017 14:38:52 -0400 (EDT) Received: (qmail 22382 invoked from network); 3 Apr 2017 18:38:52 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 3 Apr 2017 18:38:52 -0000 Received: from [192.168.1.119] (c-67-170-167-181.hsd1.or.comcast.net [67.170.167.181]) by iron2.pdx.net (Postfix) with ESMTPSA id 85632EC91EB; Mon, 3 Apr 2017 11:38:51 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: FYI: what it takes for RAM+swap to build devel/llvm40 with 4 processors or cores and WITH__DEBUG= (powerpc64 example) From: Mark Millard In-Reply-To: Date: Mon, 3 Apr 2017 11:38:50 -0700 Cc: FreeBSD Toolchain , FreeBSD Current , FreeBSD PowerPC ML Content-Transfer-Encoding: quoted-printable Message-Id: <543D47AE-7C0D-4F0E-83B4-BFEE5B802346@dsl-only.net> References: <3EDEF0B7-59C5-4648-9737-6682E18645BC@dsl-only.net> <39C60316-F905-490D-B0AB-BC24D7F351A2@dsl-only.net> <7F94CE59-D2CC-4D6F-B1CD-FF3D1F8EDCE7@FreeBSD.org> <45E32F4F-A238-47AA-8E1E-7AD4D9E857D9@dsl-only.net> <20170329155316.GK59667@spindle.one-eyed-alien.net> To: Brooks Davis , Dimitry Andric , FreeBSD Ports X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Apr 2017 18:38:59 -0000 On 2017-Apr-1, at 3:51 AM, Mark Millard wrote: > On 2017-Mar-31, at 4:51 PM, Mark Millard = wrote: >=20 >> On 2017-Mar-30, at 7:51 PM, Mark Millard = wrote: >>=20 >>> On 2017-Mar-30, at 1:22 PM, Mark Millard = wrote: >>>=20 >>>> Sounds like the ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG technique >>>> would not change the "WITNESS and INVARIANTS"-like part of the >>>> issue. In fact if WITH_DEBUG=3D causes the cmake debug-style >>>> llvm40 build ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG might not >>>> make any difference: separate enforcing of lack of optimization. >>>>=20 >>>> But just to see what results I've done "pkg delete llvm40" >>>> and am doing another build with ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG=3D= >>>> and its supporting code in place in addition to using WITH_DEBUG=3D >>>> as the type of build fro FreeBSD's viewpoint. >>>>=20 >>>> If you know that the test is a waste of machine cycles, you can >>>> let me know if you want. >>>=20 >>> The experiment showed that ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG >>> use made no difference for devel/llvm40 so devel/llvm40 itself >>> has to change such as what Dimitry Andric reported separately >>> as a working change to the Makefile . >>>=20 >>> (ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG would still have its uses >>> for various other ports.) >>=20 >> I've now tried with both ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG and: >=20 > I may have had a textual error that prevented > ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG from even potentially > contributing. So I'll re-run this test. >=20 > For now I presume that what I reported was okay and so > I continue to refer to these figures later below. The retry got the same 42 GiByte and 102 GiByte sizes. (And again I was not monitoring the swap space usage.) So functionality like ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG (keeping optimization flags in CFLAGS) does not contribute to devel/llvm40's handling. Apparently the CMAKE_BUILD_TYPE has full control over such and RelWithDebInfo still makes for a massive build, though not as big as for DEBUG. For my context I've chosen to go with: # # =46rom a local /usr/ports/Mk/bsd.port.mk extension: ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG=3D # .if ${.CURDIR:M*/devel/llvm*} #WITH_DEBUG=3D .else WITH_DEBUG=3D .endif WITH_DEBUG_FILES=3D (where ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG is from a local change to /usr/ports/Mk/bsd.port.mk but is not justified via devel/llvm* ports but via behavior for most other ports.) >> # svnlite diff /usr/ports/devel/llvm40/ >> Index: /usr/ports/devel/llvm40/Makefile >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> --- /usr/ports/devel/llvm40/Makefile (revision 436747) >> +++ /usr/ports/devel/llvm40/Makefile (working copy) >> @@ -236,6 +236,11 @@ >>=20 >> .include >>=20 >> +.if defined(WITH_DEBUG) >> +CMAKE_BUILD_TYPE=3D RelWithDebInfo >> +STRIP=3D >> +.endif >> + >> _CRTLIBDIR=3D = ${LLVM_PREFIX:S|${PREFIX}/||}/lib/clang/${LLVM_RELEASE}/lib/freebsd >> .if ${ARCH} =3D=3D "amd64" >> _COMPILER_RT_LIBS=3D \ >>=20 >>=20 >>=20 >> pkg delete after the build reports: >>=20 >> Installed packages to be REMOVED: >> llvm40-4.0.0 >>=20 >> Number of packages to be removed: 1 >>=20 >> The operation will free 42 GiB. >>=20 >> So down by 7 GiBytes from 49 GiBytes. >>=20 >> (I did not actually delete it.) >>=20 >> Also: >>=20 >> # du -sg /usr/obj/portswork/usr/ports/devel/llvm40 >> 102 /usr/obj/portswork/usr/ports/devel/llvm40 >>=20 >> which is down by 16 GiBytes from 118 GiBytes. >>=20 >> Reminder: These are from portmaster -DK so no >> cleanup after the build, which is what leaves >> the source code and such around in case of >> needing to look at a problem. >>=20 >> (102+42) GiBytes =3D=3D 146 GiBytes. >> vs. >> (118+49) GiBytes =3D=3D 167 GiBytes. >>=20 >> So a difference of 21 GiBytes (or so). >>=20 >> But that is for everything in each case (and >> WITH_DEBUG=3D in use): >>=20 >> # more /var/db/ports/devel_llvm40/options >> # This file is auto-generated by 'make config'. >> # Options for llvm40-4.0.0.r4 >> _OPTIONS_READ=3Dllvm40-4.0.0.r4 >> _FILE_COMPLETE_OPTIONS_LIST=3DCLANG DOCS EXTRAS LIT LLD LLDB >> OPTIONS_FILE_SET+=3DCLANG >> OPTIONS_FILE_SET+=3DDOCS >> OPTIONS_FILE_SET+=3DEXTRAS >> OPTIONS_FILE_SET+=3DLIT >> OPTIONS_FILE_SET+=3DLLD >> OPTIONS_FILE_SET+=3DLLDB >>=20 >> So avoiding WITH_DEBUG=3D and/or various build options >> is still the major way of avoiding use of lots of space >> if it is an issue. >>=20 >>=20 >>=20 >> Why no RAM+SWAP total report this time: >>=20 >> As far as I know FreeBSD does not track or report peak >> swap-space usage since the last boot. And, unfortunately >> I was not around to just sit and watch a top display this >> time and I did not set up any periodic recording into a >> file. >>=20 >> That is why I've not reported on the RAM+SWAP total >> this time. It will have to be another experiment >> some other time. >>=20 >> [I do wish FreeBSD had a way of reporting peak swap-space >> usage.] >=20 > I've also tried without WITH_DEBUG=3D and now. . . >=20 >=20 > # pkg delete llvm40 > Checking integrity... done (0 conflicting) > Deinstallation has been requested for the following 1 packages (of 0 = packages in the universe): >=20 > Installed packages to be REMOVED: > llvm40-4.0.0 >=20 > Number of packages to be removed: 1 >=20 > The operation will free 1 GiB. >=20 > Proceed with deinstalling packages? [y/N]: n >=20 >=20 > # du -sg /usr/obj/portswork/usr/ports/devel/llvm40/ > 5 /usr/obj/portswork/usr/ports/devel/llvm40/ >=20 >=20 > So the alternatives (with everything built each time): >=20 > (5+1) GiBytes =3D=3D 6 GiBytes. (without WITH_DEBUG=3D) > vs. > (102+42) GiBytes =3D=3D 146 GiBytes. (WITH_DEBUG=3D but the adjusted = llvm40/Makefiele) > vs. > (118+49) GiBytes =3D=3D 167 GiBytes. (WITH_DEBUG=3D with Makefile = adjustment) >=20 >=20 > I'll likely end up having /etc/make.conf contain > something like the following for most or all of > my FreeBSD environments: >=20 > # > # =46rom a local /usr/ports/Mk/bsd.port.mk extension: > ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG=3D > # > .if ${.CURDIR:M*/devel/llvm*} > #WITH_DEBUG=3D > .else > WITH_DEBUG=3D > .endif > WITH_DEBUG_FILES=3D >=20 >=20 > Along with using: >=20 > # svnlite diff /usr/ports/Mk/ > Index: /usr/ports/Mk/bsd.port.mk > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- /usr/ports/Mk/bsd.port.mk (revision 436747) > +++ /usr/ports/Mk/bsd.port.mk (working copy) > @@ -1646,7 +1646,11 @@ > STRIP_CMD=3D ${TRUE} > .endif > DEBUG_FLAGS?=3D -g > +.if defined(ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG) > +CFLAGS:=3D ${CFLAGS} ${DEBUG_FLAGS} > +.else > CFLAGS:=3D ${CFLAGS:N-O*:N-fno-strict*} ${DEBUG_FLAGS} > +.endif > .if defined(INSTALL_TARGET) > INSTALL_TARGET:=3D ${INSTALL_TARGET:S/^install-strip$/install/g} > .endif >=20 >=20 > [Although apparently the ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG > use has no effect for devel/llvm40 .] =3D=3D=3D Mark Millard markmi at dsl-only.net