From owner-freebsd-arch@FreeBSD.ORG Tue Feb 3 07:40:13 2009 Return-Path: Delivered-To: arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0050E106566B for ; Tue, 3 Feb 2009 07:40:12 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id 957A08FC21 for ; Tue, 3 Feb 2009 07:40:12 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (critter.freebsd.dk [192.168.61.3]) by phk.freebsd.dk (Postfix) with ESMTP id 170C83F129; Tue, 3 Feb 2009 07:40:11 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.14.3/8.14.3) with ESMTP id n137eA77009062; Tue, 3 Feb 2009 07:40:10 GMT (envelope-from phk@critter.freebsd.dk) To: obrien@FreeBSD.org From: "Poul-Henning Kamp" In-Reply-To: Your message of "Mon, 02 Feb 2009 14:06:29 PST." <20090202220628.GA76833@dragon.NUXI.org> Date: Tue, 03 Feb 2009 07:40:10 +0000 Message-ID: <9061.1233646810@critter.freebsd.dk> Sender: phk@critter.freebsd.dk Cc: arch@FreeBSD.org Subject: Re: svn commit: r187132 - head/usr.bin/make X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Feb 2009 07:40:13 -0000 In message <20090202220628.GA76833@dragon.NUXI.org>, "David O'Brien" writes: David, I am disappointed. You of all people here should know better than making such a mess out of benchmarks. First of all, you don't bother to even calculate a standard deviation even though we have a neat tool that does that in the base system, it's called "ministat", try it. Second you totally bungle your data collection, by not eliminating cache-effects. It should be evident to anybody that when your numbers always follow the pattern "high, low, low", that another run is necessary (you need 3 for stddev) and the first one should be discarded. Poul-Henning >The abbreviated results for local disk are: > >MAKE=make@r187921 /usr/bin/time -h $MAKE -j16 -Q buildworld >outfile 2>&1 > 25m38.07s real 1h30m6.59s user 45m37.27s sys > 25m15.23s real 1h30m1.67s user 45m28.50s sys > 25m22.65s real 1h30m8.85s user 45m32.10s sys > (1538.07 + 1515.23 + 1522.65)/3 = 1525.32 sec ave > [10% improvement would reduce build by 2m33s] Standard deviation: 11.65 seconds >MAKE=make@r187921 /usr/bin/time -h $MAKE -j16 -Q -s buildworld >outfile 2>&1 > 24m36.12s real 1h27m27.18s user 44m26.88s sys > 24m34.77s real 1h27m27.82s user 44m24.56s sys > 24m29.01s real 1h27m22.16s user 44m28.23s sys > (1476.12 + 1474.77 + 1469.01)/3 = 1473.30 sec ave > => 1473.30 / 1525.32 * 100 - 100 = -3.41% change in build time standard deviation: 3.77 second Difference at 95.0% confidence -52.0167 +/- 19.6298 -3.41022% +/- 1.28694% (Student's t, pooled s = 8.6605) > [compared with below] > => 1473.30 / 1511.70 * 100 - 100 = -2.54% change in build time Difference at 95.0% confidence 38.4033 +/- 31.7193 2.60662% +/- 2.15294% (Student's t, pooled s = 13.9942) >MAKE=make@r187921 /usr/bin/time -h $MAKE -j16 buildworld >outfile 2>&1 > 25m3.95s real 1h27m40.02s user 45m19.12s sys > 24m57.35s real 1h27m33.31s user 45m6.81s sys > 25m33.81s real 1h27m39.68s user 44m50.02s sys > (1503.95 + 1497.35 + 1533.81)/3 = 1511.70 sec ave > => 1511.70 / 1525.32 * 100 - 100 = -.89% change in build time standard deviation: 19.42 second >MAKE=make@r187921 /usr/bin/time -h $MAKE buildworld >outfile 2>&1 > 1h48m7.30s real 1h28m22.07s user 22m28.01s sys > 1h48m0.94s real 1h28m35.60s user 22m7.59s sys > 1h48m4.20s real 1h28m39.28s user 21m58.27s sys > (6487.30 + 6480.94 + 6484.20)/3 = 6484.15 sec ave > [10% improvement would reduce build by 10m48s] standard deviation 3.18 seconds >MAKE=make@r187921 /usr/bin/time -h $MAKE -s buildworld >outfile 2>&1 > 1h47m56.48s real 1h28m35.91s user 22m0.57s sys > 1h47m55.65s real 1h28m40.66s user 21m56.70s sys > 1h51m5.05s real 1h28m48.22s user 22m21.96s sys > (6476.48 + 6475.65 + 6665.05)/3 = 6539.06 sec ave N Min Max Median Avg Stddev x 3 6475.65 6665.05 6476.48 6539.06 109.11133 + 3 6480.94 6487.3 6484.2 6484.1467 3.1803354 No difference proven at 95.0% confidence > If we toss out the high value and use 6476.48 twice, ave = 6476.20 > => 6476.20 / 6539.06 * 100 - 100 = -.96% change in build time We don't. Statistics is not a television show. >The NFS results are: > >MAKE=make@r187921 /usr/bin/time -h $MAKE -j16 buildworld >outfile 2>&1 > 35m33.59s real 1h33m20.08s user 52m1.04s sys > 31m9.58s real 1h33m43.16s user 52m41.46s sys > 31m18.94s real 1h33m40.45s user 52m41.58s sys > (2133.59 + 1869.58 + 1878.94)/3 = 1960.70 sec ave standard deviation 149.79 seconds >MAKE=make@r187921 /usr/bin/time -h $MAKE -j16 -Q -s buildworld >outfile 2>&1 > 31m42.62s real 1h33m28.64s user 52m4.32s sys > 31m14.54s real 1h33m21.25s user 52m8.88s sys > 31m17.26s real 1h33m22.48s user 52m10.93s sys > (1902.62 + 1874.54 + 1877.26)/3 = 1884.81 sec ave > => 1884.81 / 1960.70 * 100 - 100 = -3.87% change in build time standard deviation 15.48 seconds N Min Max Median Avg Stddev x 3 1869.58 2133.59 1878.94 1960.7033 149.79737 + 3 1874.54 1902.62 1877.26 1884.8067 15.486631 No difference proven at 95.0% confidence > To be fair as in the local disk case, redoing the above: > (1878.94 + 1869.58 + 1878.94)/3 = 1875.82 sec ave (adjusted) > => 1884.81 / 1875.82 * 100 - 100 = .48% change in build time That has nothing to do with "fair", that is fudging the numbers. What you should have done, was realize that you need to run four tests and throw the first one which primes the cache away. >MAKE=make@r187921 /usr/bin/time -h $MAKE buildworld >outfile 2>&1 > 2h21m30.74s real 1h30m28.41s user 29m8.64s sys > 2h21m2.15s real 1h30m25.54s user 29m2.99s sys > 2h21m0.24s real 1h30m25.93s user 29m3.85s sys > (8490.74 + 8462.15 + 8460.24)/3 = 8471.04 sec ave standard deviation 17.08 seconds >MAKE=make@r187921 /usr/bin/time -h $MAKE -s buildworld >outfile 2>&1 > 2h20m48.71s real 1h30m26.02s user 29m7.66s sys > 2h21m5.95s real 1h30m21.87s user 29m10.28s sys > 2h21m7.70s real 1h30m29.85s user 29m4.29s sys > (8448.71 + 8465.95 + 8467.7)/3 = 8460.79 sec ave standard deviation 10.49 seconds No difference proven at 95.0% confidence -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence.