Date: Mon, 31 Mar 2003 07:26:28 +0300 From: Ruslan Ermilov <ru@freebsd.org> To: "M. Warner Losh" <imp@bsdimp.com> Cc: arch@freebsd.org Subject: Re: depend + all vs dependall Message-ID: <20030331042628.GA65700@sunbay.com> In-Reply-To: <20030329.163343.53040416.imp@bsdimp.com> References: <20030329.163343.53040416.imp@bsdimp.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--WhfpMioaduB5tiZL
Content-Type: multipart/mixed; boundary="gBBFr7Ir9EOA20Yy"
Content-Disposition: inline
--gBBFr7Ir9EOA20Yy
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Sat, Mar 29, 2003 at 04:33:43PM -0700, M. Warner Losh wrote:
> NetBSD created a dependall target some time ago. This target does a
> make depend and then a make all so they only have to traverse the tree
> once for these two stages rather than twice. The time of a buildworld
> came up in a discussion recently and I thought I'd see how hard it
> would be to do something similar in FreeBSD. Here are my preliminary
> results.
>=20
> Machine: Dell Inspiron 8000, 256M RAM, P3-700
> time make buildworld
> (2:04:34 wall time, didn't save the actual output :-(.
>=20
> Machine: Dual Athlon XP2000+ 1.5G RAM aac controller.
>=20
> time make buildworld -j 8 -s
>=20
> run 0: did the above to 'flush the caches/load the sources in ram'
>=20
> Pre-change:
>=20
> Run 1:
> 1941.458u 723.640s 32:23.67 137.1% 2747+2215k 1447+145802io 465pf+0w
> Run 2:
> 1942.160u 729.972s 31:45.84 140.2% 2748+2212k 1423+145755io 465pf+0w
>=20
> After Changes:
>=20
> Run 1:
> 1922.767u 723.847s 30:48.64 143.1% 2785+2201k 1312+148256io 465pf+0w
> Run 2:
> 1922.661u 725.477s 30:49.99 143.1% 2788+2201k 1378+148489io 465pf+0w
>=20
> So it looks like it saves a little over a minute out of 32 (1925s
> average vs 1849s average, or almost a 4% reduction) on my big build
> box.
>=20
Where are these 1925 and 1849 averages?
> My only concern with the patches is that they might interact badly
> with a bug I remember from the FreeBSD 1.1R days, but can't reproduce,
> in make. Once upon a time, 'make depend all' was different than 'make
> depend && make all' because the .depend files weren't re-read after
> the depend phase, but before the all phase, whereas two makes this
> would be the case. Since this change combines the two, I'm a little
> worried about that. Is that still a bug in FreeBSD's make? It won't
> matter for a pure, virgin tree, but might for incremental builds...
>=20
=2Edepend files are read at the time makefiles are read, once at
startup, so calling "make depend all" is a bug.
> =3D=3D=3D=3D //depot/user/imp/freebsd-imp/Makefile.inc1#18 (text+ko) =3D=
=3D=3D=3D
>=20
> @@ -319,18 +319,12 @@
> @echo ">>> stage 4: building libraries"
> @echo "--------------------------------------------------------------"
> cd ${.CURDIR}; ${WMAKE} -DNOHTML -DNOINFO -DNOMAN -DNOFSCHG libraries
> -_depend:
> - @echo
> - @echo "--------------------------------------------------------------"
> - @echo ">>> stage 4: make dependencies"
> - @echo "--------------------------------------------------------------"
> - cd ${.CURDIR}; ${WMAKE} par-depend
> everything:
> @echo
> @echo "--------------------------------------------------------------"
> @echo ">>> stage 4: building everything.."
> @echo "--------------------------------------------------------------"
> - cd ${.CURDIR}; ${WMAKE} all
> + cd ${.CURDIR}; ${WMAKE} dependall
> =20
This pessimizes "everything" that I often use standalone.
> =3D=3D=3D=3D //depot/user/imp/freebsd-imp/share/mk/bsd.subdir.mk#2 (text+=
ko) =3D=3D=3D=3D
>=20
> @@ -25,8 +25,8 @@
> # put the stuff into the right "distribution".
> #
> # afterinstall, all, all-man, beforeinstall, checkdpadd,
> -# clean, cleandepend, cleandir, depend, install, lint, maninstall,
> -# obj, objlink, realinstall, regress, tags
> +# clean, cleandepend, cleandir, depend, dependall, install, lint,
> +# maninstall, obj, objlink, realinstall, regress, tags
> #
> =20
> .include <bsd.init.mk>
> @@ -67,7 +67,7 @@
> =20
> =20
> .for __target in all all-man checkdpadd clean cleandepend cleandir \
> - depend distribute lint maninstall \
> + depend dependall distribute lint maninstall \
> obj objlink realinstall regress tags
> ${__target}: _SUBDIR
> .endfor
If you want to really try dependall, you should implement it
in bsd.subdir.mk like the distribute target, similar to this:
=2Eif !target(dependall)
dependall:
cd ${.CURDIR}; \
${MAKE} depend -DNO_SUBDIR; \
${MAKE} all -DNO_SUBDIR
=2Eendif
Also, your test is not honest because original Makefile.inc1
did not parallelize the "all" stage of "buildworld", by not
implementing par-all. Last time I tried par-all, it saved
me 16% of time from the -j8 buildworld:
26m8.74s real 26m13.08s user 11m9.70s sys (old)
21m48.52s real 26m20.60s user 11m4.95s sys (new)
Attached is the message with the patch. It has some Russian,
but also includes a patch. Note that par-all only parallelizes
top-level bsd.subdir.mk makefiles, as we depend on the ordering
of traversing SUBDIRs in a few places. The plan is to drop
this assumption in places that don't need this ordering.
Cheers,
--=20
Ruslan Ermilov Sysadmin and DBA,
ru@sunbay.com Sunbay Software AG,
ru@FreeBSD.org FreeBSD committer,
+380.652.512.251 Simferopol, Ukraine
http://www.FreeBSD.org The Power To Serve
http://www.oracle.com Enabling The Information Age
--gBBFr7Ir9EOA20Yy
Content-Type: message/rfc822
Content-Disposition: inline
Date: Fri, 4 Oct 2002 17:44:10 +0300
From: Ruslan Ermilov <ru@FreeBSD.org>
To: unisol@Me-262.ua.net
Cc: freebsd@FreeBSDDiary.org.ua
Subject: Re: [freebsd] FreeBSD upgrade
Message-ID: <20021004144410.GB82605@sunbay.com>
References: <20021002171801.D45096@free.ukr.net>
<20021002142042.GA54439@freeman.ukrtelecom.net>
<20021002143602.GA96604@burka.carrier.kiev.ua>
<20021002144251.GB54730@freeman.ukrtelecom.net>
<20021002211915.A91783@phantom.cris.net>
<20021003073011.GK40754@netch.kiev.ua> <20021003111001.GA8724@sunbay.com>
<20021004171349.B30990@Me-262.ua.net>
Mime-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
protocol="application/pgp-signature"; boundary="ZoaI/ZTpAVc4A5k6"
Content-Disposition: inline
In-Reply-To: <20021004171349.B30990@Me-262.ua.net>
User-Agent: Mutt/1.3.99i
--ZoaI/ZTpAVc4A5k6
Content-Type: text/plain; charset=koi8-r
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Fri, Oct 04, 2002 at 05:13:49PM +0300, unisol@Me-262.ua.net wrote:
> On Thu, Oct 03, 2002 at 02:10:01PM +0300, Ruslan Ermilov wrote:
> > On Thu, Oct 03, 2002 at 10:30:11AM +0300, Valentin Nechayev wrote:
> > > Wed, Oct 02, 2002 at 21:19:15, phantom wrote about "Re: [freebsd] Fr=
eeBSD upgrade":=20
> > >=20
> > > > > > > =EE=C1 PIII 1G + ATA100 make buildworld =D0=D2=CF=C8=CF=C4=C9=
=D4 =DA=C1 ~50 =CD=C9=CE=D5=D4
> > > > > > =E8=CD. Celeron 1G + ata100 make world, =C9 =D0=D2=C1=D7=CC=C5=
=CE=CE=D9=CA /etc/make.conf ~ 37 min.
> > > > > =E8=CD. Dual PIII 1.4G + SCSI RAID10 + =C4=C5=C6=CF=CC=D4=CE=D9=
=CA /etc/make.conf=20
> > > > > make -j100 buildworld - 23 =CD=C9=CE=D5=D4=D9
> > > > PIV 1.4G + ATA66 + default /etc/make.conf
> > > > make -j4 buildworld -- 24 minutes
> > > > PS: Dolgo pupkami budem meryatsya ? :-)
> > >=20
> > > =EE=C1=D7=C5=D2=CE=CF, =C4=CF=CC=C7=CF. =E5=D3=CC=C9 =CE=C5 =D0=CF=CE=
=C9=CD=C1=D4=D8, =DE=D4=CF =DB=D4=C1=D4=CE=D9=CA make buildworld (=C2=C5=DA=
-pipe,
> > > =C2=C5=DA -j) =CD=C5=D2=D1=C5=D4 =D3=CB=CF=D2=CF=D3=D4=D8 =C4=C9=D3=
=CB=CF=D7=CF=CA =D0=CF=C4=D3=C9=D3=D4=C5=CD=D9, =C1 =CE=C5 =D0=D2=CF=C3=C5=
=D3=D3=CF=D2=C1 ;)
> =EE=C5=D4 - =CE=C5=D0=D2=C1=D7=C4=C1. =E4=C9=D3=CB =D7=CC=C9=D1=C5=D4, =
=CE=CF =CF=CF=CF=DE=C5=CE=D8 =CD=C1=CC=CF, =CD=CF=D6=C5=D4 =D4=CF=CC=D8=CB=
=CF =D5 dual athlon
> =DC=D4=CF =C2=D5=C4=C5=D4 =D3=D5=DD=C5=D3=D4=D7=C5=CE=CE=CF. =F0=D2=CF=D7=
=C5=D2=C5=CE=CF =CE=C1 =D3=C2=CF=D2=CB=C5 world =CE=C1 2xPIII - -j4/-j8 =C4=
=C1=CC=CF
> =D0=D2=CF=D3=D4=CF =C4=D7=D5=CB=D2=C1=D4=CE=D9=CA =D0=D2=C9=D2=CF=D3=D4, =
-j16 - =D4=CF=D2=CD=CF=DA=CE=D5=CC=CF =D5=D6=C5 =CE=C5=CD=CE=CF=C7=CF.
>=20
> > =F0=CF=DE=C5=CD=D5 "=C2=C5=DA"? =FA=C4=C5=D3=D8-=D4=CF =CB=C1=CB =D2=
=C1=DA =D2=C5=DE=D8 =C9=C4=A3=D4 =CF "=D3". :-)
> >=20
> > =F1 =C4=D5=CD=C1=C0, =DE=D4=CF -j100 -- =DC=D4=CF =D3=CC=C9=DB=CB=CF=CD=
. =F1 =C2=D9 =D0=CF=D3=CF=D7=C5=D4=CF=D7=C1=CC =CE=C1=DE=C1=D4=D8 =D3
> -j100 - =DC=D4=CF =D5=D6=C5 =D0=CF=DE=D4=C9 "=D0=D2=CF=C9=DA=D7=CF=C4=C9=
=D4=C5=CC=D8=CE=CF=D3=D4=D8 =C4=C9=D3=CB=CF=D7=CF=CA =D3=C9=D3=D4=C5=CD=D9".
> =F7=C9=C4=C5=CC =CC=C9 =CB=D4=CF, =DE=D4=CF =D0=D2=CF=C9=D3=C8=CF=C4=C9=
=D4 =D3 P4-1.7GHz/1G DDR/100G IDE, =CB=CF=C7=C4=C1
> =CE=C1 =CE=A3=CD =D7=C9=D3=C9=D4 3000-4000 =D0=D2=CF=C3=C5=D3=D3=CF=D7, =
=C7=C4=C5 1/4 =C9=DA =CE=C9=C8 =CF=D4=CB=D2=D9=D7=C1=C5=D4 ~5 =C6=C1=CA=CC=
=CF=D7
> =C9 =C4=C5=CC=C1=C5=D4 fflush =CB=C1=D6=C4=D5=C0 =D3=C5=CB=D5=CE=C4=D5 =
=D0=CF=D3=CC=C5 =D7=D9=D7=CF=C4=C1 ~6-10 =C2=C1=CA=D4?
> =F7=CF=D4 =D4=CF =F4=CF=D2=CD=CF=DA=C1. =F0=D2=C9=DE=A3=CD, =D7=CE=C1=DE=
=C1=CC=C5 =D7=D3=A3 =DC=D4=CF =D7=D0=CF=CC=CE=C5 =CE=CF=D2=CD=C1=CC=D8=CE=
=CF =D2=C1=C2=CF=D4=C1=C5=D4,
> =D0=CF=D4=CF=CD LA =D7=DA=CC=C5=D4=C1=C5=D4, =D7=D3=A3 =D4=CF=D2=CD=CF=DA=
=C9=D4... =E1 =D7=D9=D7=CF=C4=C1 - =CB=CF=D0=C5=CA=CB=C9, ~1MB/s,
> 160-200 transactions/s.
> > -j4 =C9 =D5=D7=C5=CC=C9=DE=C9=D7=C1=D4=D8 =D7 =C4=D7=C1 =D2=C1=DA=C1 =
=C4=CF =D4=C5=C8 =D0=CF=D2, =D0=CF=CB=C1 =C2=D5=C4=C5=CD =D3=CD=D9=D3=CC --
> > =DA=C1=CB=CF=CE =E1=CD=C4=C1=CC=D1. =E4=CC=D1 =C2=CF=CC=D8=DB=C9=CE=D3=
=D4=D7=C1 =CD=C1=DB=C9=CE =CE=C1 =CB=CF=D4=CF=D2=D9=C8 =CD=CE=C5 =D0=D2=C9=
=C8=CF=C4=C9=CC=CF=D3=D8
> > =C4=C5=CC=C1=D4=D8 -j world'=D9 =C9 release'=D9, =CF=D0=D4=C9=CD=C1=CC=
=D8=CE=D9=CD =CF=CB=C1=DA=D9=D7=C1=CC=CF=D3=D8 -j4,
> > =C4=C1=D6=C5 =CE=C1 2xCPU SMP =C9 1xCPU =CD=C1=DB=C9=CE=C1=C8.
> =E5=D3=D4=C5=D3=D4=D7=C5=CE=CE=CF - parallel =D4=CF =CF=CE=CF =C4=C1, =CE=
=CF =D7=D3=A3 =D2=C1=D7=CE=CF =CE=C5 =D7=D3=A3 =D2=C1=D3=D0=C1=D2=C1=CC=C5=
=CC=C9=D7=C1=C5=D4=D3=D1 -
> =D0=CF=CC=CF=D7=C9=CE=D5 =D7=D2=C5=CD=C5=CE=C9 =D7=D3=A3 =D2=C1=D7=CE=CF =
"=CE=C9=DE=C5=C7=CF =CE=C5 =D0=D2=CF=C9=D3=C8=CF=C4=C9=D4", -j=D3=CB=CF=CC=
=D8=CB=CF=C8=CF=DE=C5=DB=D8
> =D0=C9=DB=C9 - =D0=D2=CF=D4=C9=D7 dependencies&so on =CF=CE=CF =CE=C5 =D0=
=D9=D4=C1=C5=D4=D3=D1 "=D0=C5=D2=C5=D4=D8"...
>=20
=EB=D3=D4=C1=D4=C9 =CF =D0=C1=D2=C1=CC=CC=C5=CC=D8=CE=CF=D3=D4=C9. =F1 =D7=
=D3=A3 =DA=C1=C2=D9=D7=C1=C0 =D0=CF=D0=D2=CF=C2=CF=D7=C1=D4=D8 =DC=D4=CF=D4=
=D0=C1=D4=DE. =E9=CE=D4=C5=D2=C5=D3=CE=CF,
=CB=C1=CB =CB =CE=C5=CD=D5 =CF=D4=CE=C5=D3=A3=D4=D3=D1 buildworld (RELENG_4=
). =EB=D4=CF-=CE=C9=C2=D5=C4=D8 =C8=CF=DE=C5=D4 "=DE=C5=D3=D4=CE=CF"
=D0=CF=CD=C5=D2=D1=D4=D8 =DC=D4=CF =CE=C1 idle =CD=C1=DB=C9=CE=CB=C5 "=D3" =
=C9 "=C2=C5=DA"? =F0=CF=C4 "=DE=C5=D3=D4=CE=CF=D3=D4=D8=C0" =D1 =D0=CF=CE=
=C9=CD=C1=C0
=D7 =D0=C5=D2=D7=D5=C0 =CF=DE=C5=D2=C5=C4=D8 =CF=C4=C9=CE=C1=CB=CF=D7=D9=C5=
=D5=D3=CC=CF=D7=C9=D1 =DA=C1=D0=D5=D3=CB=C1 (=CF=D4=D3=D5=D4=D3=D4=D7=C9=
=C5 =D0=CF=D3=D4=CF=D2=CF=CE=CE=C5=CA
=DA=C1=C7=D2=D5=DA=CB=C9, =CF=C4=C9=CE=C1=CB=CF=D7=CF=C5 =DE=C9=D3=CC=CF ma=
ke(1) =D0=D2=CF=C3=C5=D3=D3=CF=D7 (=DA=CE=C1=DE=C5=CE=C9=C5 -j), =C9 =C9=DA=
=CE=C1=DE=C1=CC=D8=CE=CF=C5
=CF=D4=D3=D5=D4=D3=D4=D7=C9=C5 /usr/obj (=CD=CF=D6=CE=CF =C4=CC=D1 =D5=D3=
=CB=CF=D2=C5=CE=C9=D1 =D4=CF=C7=C4=C1 =C9 =D3 -DNOCLEAN).
=F0=CF =C9=C4=C5=C5 =C4=CF=CC=D6=CE=CF =D2=C1=C2=CF=D4=C1=D4=D8 =C2=C5=DA =
=D0=D2=CF=C2=CC=C5=CD (=C5=D3=CC=C9 =CE=C5 =D2=C1=C2=CF=D4=C1=C5=D4, =DA=CE=
=C1=DE=C9=D4 =C7=C4=C5-=D4=CF
=C2=C1=C7=C1), =C9 =D1 =D4=C1=CB =D0=CF=CC=C1=C7=C1=C0 =DE=D4=CF =D7=D9=C9=
=C7=D2=D9=DB =C4=CF=CC=D6=C5=CE =C2=D9=D4=D8 =CE=C1=CC=C9=C3=CF. =E9=CE=D4=
=C5=D2=C5=D3=CE=CF =C2=D9=CC=CF
=C2=D9 =D5=D3=CC=D9=DB=C1=D4=D8 =CF =D2=C5=DA=D5=CC=D8=D4=C1=D4=C1=C8. =E5=
=D3=CC=C9 =D2=C5=DA=D5=CC=D8=D4=C1=D4=D9 =C2=D5=C4=D5=D4 =D7=D0=C5=DE=C1=D4=
=CC=D1=C0=DD=C9=CD=C9,
=CF=C2=C5=DD=C1=C0 =DC=D4=CF =DA=C1=CB=CF=CD=CD=C9=D4=C9=D4=D8 (=D7 =CB=C1=
=CB=CF=CA-=CE=C9=C2=D5=C4=D8 =C6=CF=D2=CD=C5). :-)
%%%
Index: Makefile.inc1
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
RCS file: /home/ncvs/src/Makefile.inc1,v
retrieving revision 1.141.2.56
diff -u -r1.141.2.56 Makefile.inc1
--- Makefile.inc1 30 Aug 2002 18:26:49 -0000 1.141.2.56
+++ Makefile.inc1 4 Oct 2002 14:38:02 -0000
@@ -317,7 +317,7 @@
@echo "--------------------------------------------------------------"
@echo ">>> stage 4: building everything.."
@echo "--------------------------------------------------------------"
- cd ${.CURDIR}; ${WMAKE} all
+ cd ${.CURDIR}; ${WMAKE} par-all
=20
=20
WMAKE_TGTS=3D
@@ -748,7 +748,7 @@
_prebuild_libs: ${_prebuild_libs:S/$/__L/}
_generic_libs: ${_generic_libs:S/$/__L/}
=20
-.for __target in clean cleandepend cleandir depend includes obj
+.for __target in all clean cleandepend cleandir depend includes obj
.for entry in ${SUBDIR}
${entry}.${__target}__D: .PHONY
@if test -d ${.CURDIR}/${entry}.${MACHINE_ARCH}; then \
%%%
Cheers,
--=20
Ruslan Ermilov Sysadmin and DBA,
ru@sunbay.com Sunbay Software AG,
ru@FreeBSD.org FreeBSD committer,
+380.652.512.251 Simferopol, Ukraine
http://www.FreeBSD.org The Power To Serve
http://www.oracle.com Enabling The Information Age
--ZoaI/ZTpAVc4A5k6
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (FreeBSD)
iD8DBQE9nak6Ukv4P6juNwoRAtPOAJ0ebBsXZyzXcwi4E6TpgieQ7bGGjwCfV9nj
hec1ZLIllEJXSSpsf9Sb9Mc=
=5tGi
-----END PGP SIGNATURE-----
--ZoaI/ZTpAVc4A5k6--
--gBBFr7Ir9EOA20Yy--
--WhfpMioaduB5tiZL
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (FreeBSD)
iD8DBQE+h8N0Ukv4P6juNwoRAqNXAJ9JLM/XJn34G8TyFDs3wyzhLNId3ACeJYnP
FcqW/YDrV4UV0wkgHcfNEYY=
=RAnI
-----END PGP SIGNATURE-----
--WhfpMioaduB5tiZL--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20030331042628.GA65700>