Date: Tue, 03 Feb 2009 07:40:10 +0000 From: "Poul-Henning Kamp" <phk@phk.freebsd.dk> To: obrien@FreeBSD.org Cc: arch@FreeBSD.org Subject: Re: svn commit: r187132 - head/usr.bin/make Message-ID: <9061.1233646810@critter.freebsd.dk> In-Reply-To: Your message of "Mon, 02 Feb 2009 14:06:29 PST." <20090202220628.GA76833@dragon.NUXI.org>
next in thread | previous in thread | raw e-mail | index | archive | help
In message <20090202220628.GA76833@dragon.NUXI.org>, "David O'Brien" writes:
David,
I am disappointed.
You of all people here should know better than making such a mess
out of benchmarks.
First of all, you don't bother to even calculate a standard deviation
even though we have a neat tool that does that in the base system,
it's called "ministat", try it.
Second you totally bungle your data collection, by not eliminating
cache-effects. It should be evident to anybody that when your
numbers always follow the pattern "high, low, low", that another
run is necessary (you need 3 for stddev) and the first one should
be discarded.
Poul-Henning
>The abbreviated results for local disk are:
>
>MAKE=make@r187921 /usr/bin/time -h $MAKE -j16 -Q buildworld >outfile 2>&1
> 25m38.07s real 1h30m6.59s user 45m37.27s sys
> 25m15.23s real 1h30m1.67s user 45m28.50s sys
> 25m22.65s real 1h30m8.85s user 45m32.10s sys
> (1538.07 + 1515.23 + 1522.65)/3 = 1525.32 sec ave
> [10% improvement would reduce build by 2m33s]
Standard deviation: 11.65 seconds
>MAKE=make@r187921 /usr/bin/time -h $MAKE -j16 -Q -s buildworld >outfile 2>&1
> 24m36.12s real 1h27m27.18s user 44m26.88s sys
> 24m34.77s real 1h27m27.82s user 44m24.56s sys
> 24m29.01s real 1h27m22.16s user 44m28.23s sys
> (1476.12 + 1474.77 + 1469.01)/3 = 1473.30 sec ave
> => 1473.30 / 1525.32 * 100 - 100 = -3.41% change in build time
standard deviation: 3.77 second
Difference at 95.0% confidence
-52.0167 +/- 19.6298
-3.41022% +/- 1.28694%
(Student's t, pooled s = 8.6605)
> [compared with below]
> => 1473.30 / 1511.70 * 100 - 100 = -2.54% change in build time
Difference at 95.0% confidence
38.4033 +/- 31.7193
2.60662% +/- 2.15294%
(Student's t, pooled s = 13.9942)
>MAKE=make@r187921 /usr/bin/time -h $MAKE -j16 buildworld >outfile 2>&1
> 25m3.95s real 1h27m40.02s user 45m19.12s sys
> 24m57.35s real 1h27m33.31s user 45m6.81s sys
> 25m33.81s real 1h27m39.68s user 44m50.02s sys
> (1503.95 + 1497.35 + 1533.81)/3 = 1511.70 sec ave
> => 1511.70 / 1525.32 * 100 - 100 = -.89% change in build time
standard deviation: 19.42 second
>MAKE=make@r187921 /usr/bin/time -h $MAKE buildworld >outfile 2>&1
> 1h48m7.30s real 1h28m22.07s user 22m28.01s sys
> 1h48m0.94s real 1h28m35.60s user 22m7.59s sys
> 1h48m4.20s real 1h28m39.28s user 21m58.27s sys
> (6487.30 + 6480.94 + 6484.20)/3 = 6484.15 sec ave
> [10% improvement would reduce build by 10m48s]
standard deviation 3.18 seconds
>MAKE=make@r187921 /usr/bin/time -h $MAKE -s buildworld >outfile 2>&1
> 1h47m56.48s real 1h28m35.91s user 22m0.57s sys
> 1h47m55.65s real 1h28m40.66s user 21m56.70s sys
> 1h51m5.05s real 1h28m48.22s user 22m21.96s sys
> (6476.48 + 6475.65 + 6665.05)/3 = 6539.06 sec ave
N Min Max Median Avg Stddev
x 3 6475.65 6665.05 6476.48 6539.06 109.11133
+ 3 6480.94 6487.3 6484.2 6484.1467 3.1803354
No difference proven at 95.0% confidence
> If we toss out the high value and use 6476.48 twice, ave = 6476.20
> => 6476.20 / 6539.06 * 100 - 100 = -.96% change in build time
We don't. Statistics is not a television show.
>The NFS results are:
>
>MAKE=make@r187921 /usr/bin/time -h $MAKE -j16 buildworld >outfile 2>&1
> 35m33.59s real 1h33m20.08s user 52m1.04s sys
> 31m9.58s real 1h33m43.16s user 52m41.46s sys
> 31m18.94s real 1h33m40.45s user 52m41.58s sys
> (2133.59 + 1869.58 + 1878.94)/3 = 1960.70 sec ave
standard deviation 149.79 seconds
>MAKE=make@r187921 /usr/bin/time -h $MAKE -j16 -Q -s buildworld >outfile 2>&1
> 31m42.62s real 1h33m28.64s user 52m4.32s sys
> 31m14.54s real 1h33m21.25s user 52m8.88s sys
> 31m17.26s real 1h33m22.48s user 52m10.93s sys
> (1902.62 + 1874.54 + 1877.26)/3 = 1884.81 sec ave
> => 1884.81 / 1960.70 * 100 - 100 = -3.87% change in build time
standard deviation 15.48 seconds
N Min Max Median Avg Stddev
x 3 1869.58 2133.59 1878.94 1960.7033 149.79737
+ 3 1874.54 1902.62 1877.26 1884.8067 15.486631
No difference proven at 95.0% confidence
> To be fair as in the local disk case, redoing the above:
> (1878.94 + 1869.58 + 1878.94)/3 = 1875.82 sec ave (adjusted)
> => 1884.81 / 1875.82 * 100 - 100 = .48% change in build time
That has nothing to do with "fair", that is fudging the numbers.
What you should have done, was realize that you need to run four
tests and throw the first one which primes the cache away.
>MAKE=make@r187921 /usr/bin/time -h $MAKE buildworld >outfile 2>&1
> 2h21m30.74s real 1h30m28.41s user 29m8.64s sys
> 2h21m2.15s real 1h30m25.54s user 29m2.99s sys
> 2h21m0.24s real 1h30m25.93s user 29m3.85s sys
> (8490.74 + 8462.15 + 8460.24)/3 = 8471.04 sec ave
standard deviation 17.08 seconds
>MAKE=make@r187921 /usr/bin/time -h $MAKE -s buildworld >outfile 2>&1
> 2h20m48.71s real 1h30m26.02s user 29m7.66s sys
> 2h21m5.95s real 1h30m21.87s user 29m10.28s sys
> 2h21m7.70s real 1h30m29.85s user 29m4.29s sys
> (8448.71 + 8465.95 + 8467.7)/3 = 8460.79 sec ave
standard deviation 10.49 seconds
No difference proven at 95.0% confidence
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9061.1233646810>