From owner-freebsd-performance@FreeBSD.ORG Wed Oct 20 18:09:46 2010 Return-Path: Delivered-To: performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0B1331065679 for ; Wed, 20 Oct 2010 18:09:46 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (m209-73.dsl.rawbw.com [198.144.209.73]) by mx1.freebsd.org (Postfix) with ESMTP id B87698FC22 for ; Wed, 20 Oct 2010 18:09:45 +0000 (UTC) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.14.4/8.14.4) with ESMTP id o9KHmsk7046960 for ; Wed, 20 Oct 2010 10:48:54 -0700 (PDT) (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.14.4/8.14.4/Submit) id o9KHmsdh046959 for performance@freebsd.org; Wed, 20 Oct 2010 10:48:54 -0700 (PDT) (envelope-from david) Date: Wed, 20 Oct 2010 10:48:54 -0700 From: David Wolfskill To: performance@freebsd.org Message-ID: <20101020174854.GZ21226@albert.catwhisker.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="OJWLbGElk4npXSe3" Content-Disposition: inline User-Agent: Mutt/1.4.2.3i Cc: Subject: Possible evidence of performance regression for 8.1-S (vs. 7.1) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Oct 2010 18:09:46 -0000 --OJWLbGElk4npXSe3 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable At work, my focus is on facilitating & improving the productivity of the developers. Much of the time they spend is in waiting for a build to complete -- thus, something that reduces the time thus spent is likely to be beneficial (all other things being approximately equal), while something that increases that time is (similarly) likely to be detrimental. The software builds in question are done in a FreeBSD/i386 environment; recent hardware (and my test platform) is HP DL180 G6s. Almost 2 years ago, we migrated from a lightly-patched 6.2-R to 7.1-R with 5 commits that were made to 7.1-S backported to it. On the same hardware (not the HP mentioned above), I measured a 35% reduction in elapsed time for one particular form of the build in question. This was encouraging. A couple of days ago, I updated the active slice on my 8.x reference machine to 8.1-STABLE #5 r214029 and proceeded to start some timed builds; here are some fairly raw timing data: Start Stop real user sys OS 1287436357 1287461948 25590.99 81502.22 18115.07 8.1-S 1287462797 1287488766 25969.26 81452.14 17920.14 8.1-S 1287489641 1287515287 25645.84 81548.40 18256.52 8.1-S 1287516151 1287541481 25329.64 81546.23 18294.10 8.1-S 1287542355 1287568599 26244.59 81431.47 17902.39 8.1-S 1287525363 1287546846 21483.13 82628.20 21703.09 7.1-R+ 1287548005 1287569100 21094.63 82853.19 22185.02 7.1-R+ 1287570300 1287591371 21071.33 82756.81 21943.22 7.1-R+ After the 3rd build under 8.1-S had completed, and I looked at the results so far, I became a bit concerned (as I wasn't expecting each build to take over 7 hours), so I started a similar set of test builds on my 7.x reference machine; 3 of those have completed so far, and that's what I'm reporting (above) at this time. Each iteration involves: * Clearing a "sandbox" (subdirectory) on a local file system on the system under test. * Un-tarring a reference sandbox from NFS storage to the local sandbox. * Entering the sandbox, then performing the build command under /usr/bin/time (to get the above timing information). Note that the reported times are strictly for the "build" part of each itewration. Most of the "build tools" are "captive" -- maintained by a group at work and accessed via NFS. They are normally built under FreeBSD 6.2; we use the compatNx ports to be able to run such programs. The 8.x reference machine was created by cloning the 7.x reference machine (the OS "drive" is a RAID 1; I broke the mirror and physically booted the (soon-to-be) 8.x machine from a single drive from the 7.x mirror, changed the hostname & IP address, then allowed the RAID firmware on the controller to "re-silver" that mirror). Once that finished, I performed a fairly standard source upgrade to 8.0-R on one slice, cloned that slice, booted from the cloned slice, and did a source upgrade to more recent points along the stable/8 branch, culminating with the above-cited 8.1-STABLE #5 r214029. At this point, I've left the installed ports alone, except that the 8.x slices have the compat7x port installed. Thus, the only difference between the 7.1-R test slice and the 8.1-S test slice should be the OS itself, the physical hardware (which is the same as I can make it), and the actual switch ports that each uses. So.... taking the "real" columns from the above, placing them in a couple of files, and running "awk '{print $1/60}'" against each (to convert from seconds to minutes, just for human scaling), then running ministat I see: dwolf-bsd(8.1-S)[19] ministat -s !$ ministat -s elapsed_{7,8} x elapsed_7 + elapsed_8 +--------------------------------------------------------------------------= ----+ | xx x + ++ += +| ||_MA___| = | | |__M_A___= _| | +--------------------------------------------------------------------------= ----+ N Min Max Median Avg Stddev x 3 351.189 358.052 351.577 353.606 3.8552332 + 5 422.161 437.41 427.431 429.268 5.9238499 Difference at 95.0% confidence 75.662 +/- 9.51485 21.3973% +/- 2.6908% (Student's t, pooled s =3D 5.32437) dwolf-bsd(8.1-S)[20]=20 Now, I expect to get another cople of results from the 7.1-R test, but I doubt that they will be significantly different from what we see above: the 7.1 results are more in line with what is seen "in real life." While it's very likely the case that some things might be done to make things better, my concern at this point is that -- doing the same work in the same way -- 8.1-S (@r214029) appears to be performing significantly worse than 7.1-R+patches. Since I'd like to be able to justify migrating beyond 7.x soon, I'd appreciate suggestions for identifying (and fixing) the apparent regression. FWIW, the workload is fairly CPU intensive during most of the run; the I/O done during (most of) the test appears to be very light, and the memory used is fairly modest. In each of the test machines, I have turned off HTT (HyperThreading Technology); hw.ncpu reports 8 for each. Thanks! Peace, david --=20 David H. Wolfskill david@catwhisker.org Depriving a girl or boy of an opportunity for education is evil. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --OJWLbGElk4npXSe3 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (FreeBSD) iEYEARECAAYFAky/K4UACgkQmprOCmdXAD2nmgCdHHUcVDkMLL9wvGL6p6TQEvZ/ GocAnjKYuOSwt5295c0gWmRaSGgaZuLg =rjut -----END PGP SIGNATURE----- --OJWLbGElk4npXSe3-- From owner-freebsd-performance@FreeBSD.ORG Thu Oct 21 00:00:03 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 81039106566B for ; Thu, 21 Oct 2010 00:00:03 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from out-0.mx.aerioconnect.net (outk.internet-mail-service.net [216.240.47.234]) by mx1.freebsd.org (Postfix) with ESMTP id 57D968FC17 for ; Thu, 21 Oct 2010 00:00:03 +0000 (UTC) Received: from idiom.com (postfix@mx0.idiom.com [216.240.32.160]) by out-0.mx.aerioconnect.net (8.13.8/8.13.8) with ESMTP id o9KNnI8K005434 for ; Wed, 20 Oct 2010 16:49:18 -0700 X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (h-67-100-89-137.snfccasy.static.covad.net [67.100.89.137]) by idiom.com (Postfix) with ESMTP id 555D92D6019 for ; Wed, 20 Oct 2010 16:49:18 -0700 (PDT) Message-ID: <4CBF8032.8000609@freebsd.org> Date: Wed, 20 Oct 2010 16:50:10 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.11) Gecko/20101013 Thunderbird/3.1.5 MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <20101020174854.GZ21226@albert.catwhisker.org> In-Reply-To: <20101020174854.GZ21226@albert.catwhisker.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.67 on 216.240.47.51 Subject: Re: Possible evidence of performance regression for 8.1-S (vs. 7.1) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Oct 2010 00:00:03 -0000 On 10/20/10 10:48 AM, David Wolfskill wrote: > [...] > The 8.x reference machine was created by cloning the 7.x reference > machine (the OS "drive" is a RAID 1; I broke the mirror and physically > booted the (soon-to-be) 8.x machine from a single drive from the > 7.x mirror, changed the hostname& IP address, then allowed the > RAID firmware on the controller to "re-silver" that mirror). Once > that finished, I performed a fairly standard source upgrade to 8.0-R > on one slice, cloned that slice, booted from the cloned slice, and > did a source upgrade to more recent points along the stable/8 branch, > culminating with the above-cited 8.1-STABLE #5 r214029. At this > point, I've left the installed ports alone, except that the 8.x > slices have the compat7x port installed. > [...] try the 7.x machine but running the 8.x kernel.. i.e. change nothing, but boot the new kernel. > FWIW, the workload is fairly CPU intensive during most of the run; the > I/O done during (most of) the test appears to be very light, and the > memory used is fairly modest. In each of the test machines, I have > turned off HTT (HyperThreading Technology); hw.ncpu reports 8 for each. try with HTT on for modern hardware.. > Thanks! > > Peace, > david From owner-freebsd-performance@FreeBSD.ORG Thu Oct 21 22:03:54 2010 Return-Path: Delivered-To: performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 87149106564A for ; Thu, 21 Oct 2010 22:03:54 +0000 (UTC) (envelope-from dan@dan.emsphone.com) Received: from email2.allantgroup.com (email2.emsphone.com [199.67.51.116]) by mx1.freebsd.org (Postfix) with ESMTP id 3187D8FC14 for ; Thu, 21 Oct 2010 22:03:53 +0000 (UTC) Received: from dan.emsphone.com (dan.emsphone.com [199.67.51.101]) by email2.allantgroup.com (8.14.4/8.14.4) with ESMTP id o9LLrV9d067722 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 21 Oct 2010 16:53:31 -0500 (CDT) (envelope-from dan@dan.emsphone.com) Received: from dan.emsphone.com (smmsp@localhost [127.0.0.1]) by dan.emsphone.com (8.14.4/8.14.4) with ESMTP id o9LLrUE0039881 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 21 Oct 2010 16:53:31 -0500 (CDT) (envelope-from dan@dan.emsphone.com) Received: (from dan@localhost) by dan.emsphone.com (8.14.4/8.14.4/Submit) id o9LLrUB0039880; Thu, 21 Oct 2010 16:53:30 -0500 (CDT) (envelope-from dan) Date: Thu, 21 Oct 2010 16:53:30 -0500 From: Dan Nelson To: David Wolfskill Message-ID: <20101021215330.GA86224@dan.emsphone.com> References: <20101020174854.GZ21226@albert.catwhisker.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20101020174854.GZ21226@albert.catwhisker.org> X-OS: FreeBSD 8.1-STABLE User-Agent: Mutt/1.5.21 (2010-09-15) X-Virus-Scanned: clamav-milter 0.96.3 at email2.allantgroup.com X-Virus-Status: Clean X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.6 (email2.allantgroup.com [199.67.51.78]); Thu, 21 Oct 2010 16:53:31 -0500 (CDT) X-Scanned-By: MIMEDefang 2.68 on 199.67.51.78 Cc: performance@freebsd.org Subject: Re: Possible evidence of performance regression for 8.1-S (vs. 7.1) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Oct 2010 22:03:54 -0000 In the last episode (Oct 20), David Wolfskill said: > Almost 2 years ago, we migrated from a lightly-patched 6.2-R to 7.1-R with > 5 commits that were made to 7.1-S backported to it. On the same hardware > (not the HP mentioned above), I measured a 35% reduction in elapsed time > for one particular form of the build in question. This was encouraging. > > A couple of days ago, I updated the active slice on my 8.x reference > machine to 8.1-STABLE #5 r214029 and proceeded to start some timed builds; > here are some fairly raw timing data: > > Start Stop real user sys OS > 1287436357 1287461948 25590.99 81502.22 18115.07 8.1-S > 1287462797 1287488766 25969.26 81452.14 17920.14 8.1-S > 1287489641 1287515287 25645.84 81548.40 18256.52 8.1-S > 1287516151 1287541481 25329.64 81546.23 18294.10 8.1-S > 1287542355 1287568599 26244.59 81431.47 17902.39 8.1-S > > 1287525363 1287546846 21483.13 82628.20 21703.09 7.1-R+ > 1287548005 1287569100 21094.63 82853.19 22185.02 7.1-R+ > 1287570300 1287591371 21071.33 82756.81 21943.22 7.1-R+ An observation: on 8.1, both user and sys times are less, but real time is higher. So 8.1 finished the build using less CPU, but spent more time waiting for something else. Disk? Network? I don't suppose the machines are low enough on RAM that you end up swapping at any point? Maybe there was a change to /usr/bin/make that is causing it to launch jobs slower? Julian's suggestion of booting the 8.1 kernel on the 7.1 OS will definitely narrow down the list of suspects. -- Dan Nelson dnelson@allantgroup.com From owner-freebsd-performance@FreeBSD.ORG Thu Oct 21 22:42:38 2010 Return-Path: Delivered-To: performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8332E106566C for ; Thu, 21 Oct 2010 22:42:38 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (m209-73.dsl.rawbw.com [198.144.209.73]) by mx1.freebsd.org (Postfix) with ESMTP id 4AD4A8FC0C for ; Thu, 21 Oct 2010 22:42:37 +0000 (UTC) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.14.4/8.14.4) with ESMTP id o9LMgbPX056173 for ; Thu, 21 Oct 2010 15:42:37 -0700 (PDT) (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.14.4/8.14.4/Submit) id o9LMgb51056172 for performance@freebsd.org; Thu, 21 Oct 2010 15:42:37 -0700 (PDT) (envelope-from david) Date: Thu, 21 Oct 2010 15:42:37 -0700 From: David Wolfskill To: performance@freebsd.org Message-ID: <20101021224237.GG52404@albert.catwhisker.org> References: <20101020174854.GZ21226@albert.catwhisker.org> <20101021215330.GA86224@dan.emsphone.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="SdvjNjn6lL3tIsv0" Content-Disposition: inline In-Reply-To: <20101021215330.GA86224@dan.emsphone.com> User-Agent: Mutt/1.4.2.3i Cc: Subject: Re: Possible evidence of performance regression for 8.1-S (vs. 7.1) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Oct 2010 22:42:38 -0000 --SdvjNjn6lL3tIsv0 Content-Type: multipart/mixed; boundary="w/VI3ydZO+RcZ3Ux" Content-Disposition: inline --w/VI3ydZO+RcZ3Ux Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Oct 21, 2010 at 04:53:30PM -0500, Dan Nelson wrote: > ... > An observation: on 8.1, both user and sys times are less, but real time = is > higher. So 8.1 finished the build using less CPU, but spent more time > waiting for something else. Interesting; thanks for pointing that out. > Disk? Network? I don't suppose the machines > are low enough on RAM that you end up swapping at any point? They have 6GB RAM each, but we're running FreeBSD/i386 without PAE. Still, the workloads under test were the only things running (other than fairly constant administrivia) on each of them, and the build is not especially memory-constrained, as far I as I know. As far as disk, during the (vast) majority of the execution time, disk I/O occurs at such a low rate that it's barely noticeable. And these were local builds -- that is, done on local disk (vs. NFS). The captive build tools were accessed via NFS, but that part is the same on each machine. I've attached a file that shows some of the additional rusage fields from /usr/bin/time -- it's fairly wide (240 chars) -- that may provide some additional insight. Some of the differences I note: * maxrss is almost identical (but slightly higher for 8.1 vs. 7.1). * avgsms for 8.1 is small in each case, but 8.1 is about 20 - 30% of 7.1. * pgrec is a bit lower in 8.1. * pf varies a bit, but looks about the same for each. * sw is 0 for all runs (which is as expected). * bio is noticeably lower in 8.1. * boo is also noticeably lower in 8.1. * msgs is also noticably lower in 8.1. * msgr is just about identical in all cases. * sigr is lower in 8.1. * 8.1's vcsw is about 60% of 7.1's. * 8.1's icsw is about 50% of 7.1's. [Each of the above percentages is an "eyeball estimate," so use an appropriate grain of salt.] > Maybe there > was a change to /usr/bin/make that is causing it to launch jobs slower?= =20 We use "captive" versions of make, so the version on the system is a moot point. > Julian's suggestion of booting the 8.1 kernel on the 7.1 OS will definite= ly > narrow down the list of suspects. I'll see about doing something along those lines, but I doubt it will be all that helpful, actually. --=20 David H. Wolfskill david@catwhisker.org Depriving a girl or boy of an opportunity for education is evil. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --w/VI3ydZO+RcZ3Ux Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=time_stats start stop real user sys maxrss avgsms pgrec pf sw bio boo msgs msgr sigr vcsw icsw os 1287436357 1287461948 25590.99 81502.22 18115.07 403408 40 1632501960 70091 0 1526450 3948957 52333862 16776 414554 116146505 69359713 FreeBSD 8.1-STABLE i386 1287462797 1287488766 25969.26 81452.14 17920.14 403416 33 1632275106 70921 0 1569112 3992055 52289209 16780 415153 112395968 63795293 FreeBSD 8.1-STABLE i386 1287489641 1287515287 25645.84 81548.40 18256.52 403408 44 1633037428 68757 0 1623321 3939857 52405680 16782 415519 120381724 72994804 FreeBSD 8.1-STABLE i386 1287516151 1287541481 25329.64 81546.23 18294.10 403416 44 1632579746 67606 0 1510313 3932011 52341946 16776 415086 120137651 74621868 FreeBSD 8.1-STABLE i386 1287542355 1287568599 26244.59 81431.47 17902.39 403416 30 1632203678 68433 0 1522405 3959469 52304716 16776 415022 111403586 62745487 FreeBSD 8.1-STABLE i386 1287525363 1287546846 21483.13 82628.20 21703.09 403344 170 1646120708 69576 0 1714249 4096652 66054010 16776 426045 181765163 138243214 FreeBSD 7.1-RELEASE-JNPR-4 i386 1287548005 1287569100 21094.63 82853.19 22185.02 403344 194 1646134922 69189 0 1731574 4130719 66054759 16776 425545 200096792 162326694 FreeBSD 7.1-RELEASE-JNPR-4 i386 1287570300 1287591371 21071.33 82756.81 21943.22 403348 184 1646103669 70720 0 1841404 4097966 66064504 16786 426741 187956827 152273232 FreeBSD 7.1-RELEASE-JNPR-4 i386 1287592592 1287614103 21511.23 82637.30 21849.90 403344 177 1646110915 69878 0 1704692 4088578 66057042 16776 424758 189088692 145093952 FreeBSD 7.1-RELEASE-JNPR-4 i386 1287615323 1287636770 21446.42 82715.81 21708.97 403344 169 1646267475 68996 0 1702403 4096640 66087016 16776 426374 175517603 138999711 FreeBSD 7.1-RELEASE-JNPR-4 i386 --w/VI3ydZO+RcZ3Ux-- --SdvjNjn6lL3tIsv0 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (FreeBSD) iEYEARECAAYFAkzAwdwACgkQmprOCmdXAD0rGwCfaB9t5X3cbjbs8QJyDMsxt20V BvwAn2m6HZ/HlPkHyureCvvNEItwCWGZ =i9MB -----END PGP SIGNATURE----- --SdvjNjn6lL3tIsv0-- From owner-freebsd-performance@FreeBSD.ORG Fri Oct 22 01:05:36 2010 Return-Path: Delivered-To: performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0E88E1065673 for ; Fri, 22 Oct 2010 01:05:36 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 95E8B8FC0C for ; Fri, 22 Oct 2010 01:05:35 +0000 (UTC) Received: by wwb24 with SMTP id 24so239077wwb.31 for ; Thu, 21 Oct 2010 18:05:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:sender:received :in-reply-to:references:date:x-google-sender-auth:message-id:subject :from:to:cc:content-type; bh=KKYC/350p8m5/5B6U7rzUi3mcS779LdwgYw2WXXZzHQ=; b=gvWOg8nYHAndGWXltDhY6tVsuam9dX8YAjHiOVwR7Lpxl2jBaicYP/KIw0c4mBaqCp TpzGraYneLhm63qAL4NFb+UgEuYIcxHX3WErsZ8RWTffzayM6R6urulNPtQ5+zTLzbzk 1ToEBU2IpakqMMb+uAItu/Ads7JQQBn5bEEOc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; b=CgO2k2Gfq/S3YRZqcSks5wc7GW8k2mhdk3vNytqVf2yE0GTdZjfh6QmbnEfl4dcKbJ In7Q5cbzst/S9O5wFDujrCn5Trv7bv3jAln7X8rsViCSH4yaH3Z0zzSP2OZQi1ehaDe+ TNqTPoMYShLDKiWWcJ+ohPN89AQAm40QkseS4= MIME-Version: 1.0 Received: by 10.216.28.204 with SMTP id g54mr10473766wea.73.1287707961763; Thu, 21 Oct 2010 17:39:21 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.216.229.5 with HTTP; Thu, 21 Oct 2010 17:39:21 -0700 (PDT) In-Reply-To: <20101021224237.GG52404@albert.catwhisker.org> References: <20101020174854.GZ21226@albert.catwhisker.org> <20101021215330.GA86224@dan.emsphone.com> <20101021224237.GG52404@albert.catwhisker.org> Date: Fri, 22 Oct 2010 08:39:21 +0800 X-Google-Sender-Auth: KUmt9jUgHAdI2vxfCtFkVO69X2M Message-ID: From: Adrian Chadd To: David Wolfskill Content-Type: text/plain; charset=ISO-8859-1 Cc: performance@freebsd.org Subject: Re: Possible evidence of performance regression for 8.1-S (vs. 7.1) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Oct 2010 01:05:36 -0000 On 22 October 2010 06:42, David Wolfskill wrote: >> Julian's suggestion of booting the 8.1 kernel on the 7.1 OS will definitely >> narrow down the list of suspects. > > I'll see about doing something along those lines, but I doubt it will be > all that helpful, actually. It narrows down the suspect to the kernel, rather than the userland. The third option is that it's both kernel+userland and some way they play badly together; let's hope it isn't that. adrian From owner-freebsd-performance@FreeBSD.ORG Fri Oct 22 13:13:29 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0F5D3106566B for ; Fri, 22 Oct 2010 13:13:29 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id BB6C78FC19 for ; Fri, 22 Oct 2010 13:13:28 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1P9HQz-0001Yq-GD for freebsd-performance@freebsd.org; Fri, 22 Oct 2010 15:13:25 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 22 Oct 2010 15:13:25 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 22 Oct 2010 15:13:25 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Fri, 22 Oct 2010 15:13:21 +0200 Lines: 29 Message-ID: References: <20101020174854.GZ21226@albert.catwhisker.org> <20101021215330.GA86224@dan.emsphone.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.12) Gecko/20101018 Thunderbird/3.0.8 In-Reply-To: <20101021215330.GA86224@dan.emsphone.com> X-Enigmail-Version: 1.0.1 Subject: Re: Possible evidence of performance regression for 8.1-S (vs. 7.1) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Oct 2010 13:13:29 -0000 On 10/21/10 23:53, Dan Nelson wrote: > In the last episode (Oct 20), David Wolfskill said: >> Almost 2 years ago, we migrated from a lightly-patched 6.2-R to 7.1-R with >> 5 commits that were made to 7.1-S backported to it. On the same hardware >> (not the HP mentioned above), I measured a 35% reduction in elapsed time >> for one particular form of the build in question. This was encouraging. >> >> A couple of days ago, I updated the active slice on my 8.x reference >> machine to 8.1-STABLE #5 r214029 and proceeded to start some timed builds; >> here are some fairly raw timing data: >> >> Start Stop real user sys OS >> 1287436357 1287461948 25590.99 81502.22 18115.07 8.1-S >> 1287462797 1287488766 25969.26 81452.14 17920.14 8.1-S >> 1287489641 1287515287 25645.84 81548.40 18256.52 8.1-S >> 1287516151 1287541481 25329.64 81546.23 18294.10 8.1-S >> 1287542355 1287568599 26244.59 81431.47 17902.39 8.1-S >> >> 1287525363 1287546846 21483.13 82628.20 21703.09 7.1-R+ >> 1287548005 1287569100 21094.63 82853.19 22185.02 7.1-R+ >> 1287570300 1287591371 21071.33 82756.81 21943.22 7.1-R+ > > An observation: on 8.1, both user and sys times are less, but real time is > higher. So 8.1 finished the build using less CPU, but spent more time > waiting for something else. Disk? Network? I don't suppose the machines NFS? AFAIK NFS was changed in 8.x? From owner-freebsd-performance@FreeBSD.ORG Fri Oct 22 23:29:08 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B0E051065673 for ; Fri, 22 Oct 2010 23:29:08 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (m209-73.dsl.rawbw.com [198.144.209.73]) by mx1.freebsd.org (Postfix) with ESMTP id 67CAB8FC16 for ; Fri, 22 Oct 2010 23:29:08 +0000 (UTC) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.14.4/8.14.4) with ESMTP id o9MNHULK063775 for ; Fri, 22 Oct 2010 16:17:30 -0700 (PDT) (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.14.4/8.14.4/Submit) id o9MNHUrS063774 for freebsd-performance@freebsd.org; Fri, 22 Oct 2010 16:17:30 -0700 (PDT) (envelope-from david) Date: Fri, 22 Oct 2010 16:17:30 -0700 From: David Wolfskill To: freebsd-performance@freebsd.org Message-ID: <20101022231730.GP52404@albert.catwhisker.org> References: <20101020174854.GZ21226@albert.catwhisker.org> <4CBF8032.8000609@freebsd.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="AwNVUpjOmSj7UnwZ" Content-Disposition: inline In-Reply-To: <4CBF8032.8000609@freebsd.org> User-Agent: Mutt/1.4.2.3i Subject: Re: Possible evidence of performance regression for 8.1-S (vs. 7.1) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Oct 2010 23:29:08 -0000 --AwNVUpjOmSj7UnwZ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Oct 20, 2010 at 04:50:10PM -0700, Julian Elischer wrote: > ... > try the 7.x machine but running the 8.x kernel.. i.e. change nothing,=20 > but boot the new kernel. I just started that test, to run over the weekend. I did run another test (3 iterations) with the 7.1 OS (kernel+userland) on the physical machine that normally runs 8.x. Here is a summary of the earlier results, along with the new ones: start stop real user sys host os 1287525363 1287546846 21483.13 82628.20 21703.09 ref-7.x 7.1-R+ 1287548005 1287569100 21094.63 82853.19 22185.02 ref-7.x 7.1-R+ 1287570300 1287591371 21071.33 82756.81 21943.22 ref-7.x 7.1-R+ 1287592592 1287614103 21511.23 82637.30 21849.90 ref-7.x 7.1-R+ 1287615323 1287636770 21446.42 82715.81 21708.97 ref-7.x 7.1-R+ 1287436357 1287461948 25590.99 81502.22 18115.07 ref-8.x 8.1-S@r214029 1287462797 1287488766 25969.26 81452.14 17920.14 ref-8.x 8.1-S@r214029 1287489641 1287515287 25645.84 81548.40 18256.52 ref-8.x 8.1-S@r214029 1287516151 1287541481 25329.64 81546.23 18294.10 ref-8.x 8.1-S@r214029 1287542355 1287568599 26244.59 81431.47 17902.39 ref-8.x 8.1-S@r214029 1287710312 1287732046 21733.20 82688.01 22108.95 ref-8.x 7.1-R+ 1287733360 1287754549 21188.88 82869.09 21890.83 ref-8.x 7.1-R+ 1287755881 1287777566 21684.09 82772.50 21933.74 ref-8.x 7.1-R+ I only ran 3 tests, as the purpose was merely a reality check -- to verify that we weren't actually seeing differences in hardware between the two machines. I believe that the above results provide adequate reason to believe that hardware is not the underlying issue: the 7.1 results on the ref-8.x machine are just about identical to the 7.1 results on the ref-7.x machine, and both are noticably different from the 8.1 results. > >FWIW, the workload is fairly CPU intensive during most of the run; the > >I/O done during (most of) the test appears to be very light, and the > >memory used is fairly modest. In each of the test machines, I have > >turned off HTT (HyperThreading Technology); hw.ncpu reports 8 for each. >=20 > try with HTT on for modern hardware.. > .... OK, I'll try that experiment at some point when I have the time and other resources, but what I'm trying to point out is that I believe I'm seeing something that appears to be a performance regression, comparing 7.1 (+ a few patches that were later committed to stable/7) to stable/8 @r214029, on identically configured hardware running the same workload (that happens to be critical for my employer). FWIW, I tried comparing the 7.1 (as above) vs. stable/7 a couple of months ago, and found no statistically significant difference in performance. Thus, I believe it would be possible for someone else to reproduce my results, comparing a fasirly recent stable/7 to a fairly recent stable/8. If someone does try that (with a workload primarily composed of building software), and gets results that differ significantly from what I'm reporting, I would appreciate an opportunity to compare notes, as it would then be apparent that I've managed to damage my stable/8 environment in some way. On the other hand, if someone else tries that, and gets results that corroborate what I'm seeing, we may have some work to do to get stable/8 whipped into shape before 8.2-RELEASE (for which the code freeze is currently scheduled for 28 Nov). Granted, performance isn't the only thing that drives releases, but I know that for the developers I support, having an 18% increase in elapsed time to perform a critical task, merely because of an "upgrade" to the OS, would be ... poorly received. And there's no way I could recommend doing that, unless the alternatives were demonstrably worse. I'm a bit concerned, here. Peace, david --=20 David H. Wolfskill david@catwhisker.org Depriving a girl or boy of an opportunity for education is evil. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --AwNVUpjOmSj7UnwZ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (FreeBSD) iEYEARECAAYFAkzCG4kACgkQmprOCmdXAD2lcACdGpkCS6H62SFB6yy8BtImiUtf GrEAnjXVVcVkRmATif+TsGBGXVLPlp+t =vp2g -----END PGP SIGNATURE----- --AwNVUpjOmSj7UnwZ-- From owner-freebsd-performance@FreeBSD.ORG Fri Oct 22 23:48:18 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8A3F5106564A for ; Fri, 22 Oct 2010 23:48:18 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (m209-73.dsl.rawbw.com [198.144.209.73]) by mx1.freebsd.org (Postfix) with ESMTP id 6137B8FC08 for ; Fri, 22 Oct 2010 23:48:18 +0000 (UTC) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.14.4/8.14.4) with ESMTP id o9MNmHtX063970 for ; Fri, 22 Oct 2010 16:48:17 -0700 (PDT) (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.14.4/8.14.4/Submit) id o9MNmH7r063969 for freebsd-performance@freebsd.org; Fri, 22 Oct 2010 16:48:17 -0700 (PDT) (envelope-from david) Date: Fri, 22 Oct 2010 16:48:17 -0700 From: David Wolfskill To: freebsd-performance@freebsd.org Message-ID: <20101022234817.GQ52404@albert.catwhisker.org> References: <20101020174854.GZ21226@albert.catwhisker.org> <4CBF8032.8000609@freebsd.org> <20101022231730.GP52404@albert.catwhisker.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="7a4X6VOqfbl9xMrG" Content-Disposition: inline In-Reply-To: <20101022231730.GP52404@albert.catwhisker.org> User-Agent: Mutt/1.4.2.3i Subject: Re: Possible evidence of performance regression for 8.1-S (vs. 7.1) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Oct 2010 23:48:18 -0000 --7a4X6VOqfbl9xMrG Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Oct 22, 2010 at 04:17:30PM -0700, David Wolfskill wrote: > On Wed, Oct 20, 2010 at 04:50:10PM -0700, Julian Elischer wrote: > > ... > > try the 7.x machine but running the 8.x kernel.. i.e. change nothing,= =20 > > but boot the new kernel. >=20 > I just started that test, to run over the weekend. > ... Moot point: the build fails. Peace, david --=20 David H. Wolfskill david@catwhisker.org Depriving a girl or boy of an opportunity for education is evil. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --7a4X6VOqfbl9xMrG Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (FreeBSD) iEYEARECAAYFAkzCIsAACgkQmprOCmdXAD2vdgCdGnGnshFfhjI6oFFfvI37e+Eh fzIAn0eE4OZ7xZ45xJvp2q7EmqG9bLMJ =aY7t -----END PGP SIGNATURE----- --7a4X6VOqfbl9xMrG-- From owner-freebsd-performance@FreeBSD.ORG Sat Oct 23 00:30:38 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E2A691065675 for ; Sat, 23 Oct 2010 00:30:38 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from out-0.mx.aerioconnect.net (out-0-32.mx.aerioconnect.net [216.240.47.92]) by mx1.freebsd.org (Postfix) with ESMTP id C1FB08FC13 for ; Sat, 23 Oct 2010 00:30:38 +0000 (UTC) Received: from idiom.com (postfix@mx0.idiom.com [216.240.32.160]) by out-0.mx.aerioconnect.net (8.13.8/8.13.8) with ESMTP id o9N071al013172; Fri, 22 Oct 2010 17:07:01 -0700 X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (h-67-100-89-137.snfccasy.static.covad.net [67.100.89.137]) by idiom.com (Postfix) with ESMTP id 88BCC2D6011; Fri, 22 Oct 2010 17:07:00 -0700 (PDT) Message-ID: <4CC22758.3040201@freebsd.org> Date: Fri, 22 Oct 2010 17:07:52 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.11) Gecko/20101013 Thunderbird/3.1.5 MIME-Version: 1.0 To: David Wolfskill References: <20101020174854.GZ21226@albert.catwhisker.org> <4CBF8032.8000609@freebsd.org> <20101022231730.GP52404@albert.catwhisker.org> <20101022234817.GQ52404@albert.catwhisker.org> In-Reply-To: <20101022234817.GQ52404@albert.catwhisker.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.67 on 216.240.47.51 Cc: freebsd-performance@freebsd.org Subject: Re: Possible evidence of performance regression for 8.1-S (vs. 7.1) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Oct 2010 00:30:39 -0000 On 10/22/10 4:48 PM, David Wolfskill wrote: > On Fri, Oct 22, 2010 at 04:17:30PM -0700, David Wolfskill wrote: >> On Wed, Oct 20, 2010 at 04:50:10PM -0700, Julian Elischer wrote: >>> ... >>> try the 7.x machine but running the 8.x kernel.. i.e. change nothing, >>> but boot the new kernel. >> I just started that test, to run over the weekend. >> ... > Moot point: the build fails. > that shouldn't happen maybe you need to set the uname vars so that the build THINKS it is on 7.x man 3 uname to find them. ENVIRONMENT UNAME_s If the environment variable UNAME_s is set, it will override the sysname member. UNAME_r If the environment variable UNAME_r is set, it will override the release member. UNAME_v If the environment variable UNAME_v is set, it will override the version member. UNAME_m If the environment variable UNAME_m is set, it will override the machine member. it's not unusual to have to do this when cross building with a different kernel. Running an 8.x kernel with the 7.x userland is a crucial part of debugging this. we really can't do much more until we have the results. If the build fails. it's either becasue the build detects the kernel rev (which we want to stop with this) or because of a bug in 8.1. It's supposed to be downwards compatible. > Peace, > david