From owner-freebsd-performance@FreeBSD.ORG Mon Oct 7 17:28:06 2013 Return-Path: Delivered-To: performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2FADE294 for ; Mon, 7 Oct 2013 17:28:06 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (mx.catwhisker.org [198.144.209.73]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D48DB23BD for ; Mon, 7 Oct 2013 17:28:05 +0000 (UTC) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.14.7/8.14.7) with ESMTP id r97HS4Hr008130 for ; Mon, 7 Oct 2013 10:28:04 -0700 (PDT) (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.14.7/8.14.7/Submit) id r97HS4O5008129 for performance@freebsd.org; Mon, 7 Oct 2013 10:28:04 -0700 (PDT) (envelope-from david) Date: Mon, 7 Oct 2013 10:28:04 -0700 From: David Wolfskill To: performance@freebsd.org Subject: Apparent performance regression 8.3@ -> 8.4@r255966? Message-ID: <20131007172804.GA7641@albert.catwhisker.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="BXVAT5kNtrzKuDFl" Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: performance@freebsd.org List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Oct 2013 17:28:06 -0000 --BXVAT5kNtrzKuDFl Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable At work, we have a bunch of machines that developers use to build some software. The machines presently run FreeBSD/amd64 8.3-STABLE @rxxxxxx (with a few local patches, which have since been committed to stable/8), and the software is built within a 32-bit jail. The hardware includes 2 packages of 6 physical cores each @3.47GHz (Intel X5690); SMT is enabled (so the scheduler sees hw.ncpu =3D=3D 24). The memory on the machines was recently increased from 6GB to 96GB. I am trying to set up a replacement host environment on my test machine; the current environment there is FreeBSD/amd64 8.4-STABLE @r255966; this environment achieves a couple of objectives: * It has no local patches. * The known problems (e.g., with mfiutil failing to report battery status accurately) are believed to be addressed appropriately. However: when I do comparison software builds, the new environment is taking about 12% longer to perform the same work (comparing against a fair sample of the deployed machines): Now, when I do these builds, I do so under /usr/bin/time, as well as using a bit of "scaffolding" I cobbled up (a few years back) that basically samples a bunch of sysctl OIDs periodically (by default, every 10 seconds). Once the build is done, I grab the file that has the sampled OID data and bring it to my desktop machine to post-process it; I generate graphs showing (aggregate and per-core) CPU utilization, as well as Load Averages over the course of the build. I can also generate graphs that show how the memory statistics that "top" displays vary during the course of the build, as well as just about any univariate OID, and quite a few simple multivariate OIDs (e.g., kern.cp_time, kern.cp_times, and vm.loadavg). After seeing the above results and poking around looking for somewhat-recent tuning information, I ran across a suggestion that the default of 2MB for vfs.ufs.dirhash_maxmem was probably on the low side. So I started sampling both vfs.ufs.dirhash_maxmem (mostly to make documentation of the configuration for a test run easier) and vfs.ufs.dirhash_mem (to see what we were actually using). And I tried quadrupling vfs.ufs.dirhash_maxmem (to 8MB). The next time I tried a test build, I found that vfs.ufs.dirhash_mem started at about 3.8MB, climbed fairly steadily, then "clipped" at 8MB, so I quadrupled it again (to 32MB), and found that it climbed to almost 12MB, then dropped precipitously to about 400KB (and oscillated between about 400KB & 20MB for the rest of the build, which appears to be the "packaging" phase). Despite that increase in vfs.ufs.dirhash_maxmem, this does not appear to have measurably affected the build times. In examining the CPU utilization graphs, the CPU generally looks about 5% busy for the first 15 minutes; this would be bmake determining dependency graphs, I expect. For the next 1:20, CPU is about 70% busy (~15% system; ~65% user/nice) for about 20 minutes, then drops to about 45% busy (~25% system; ~20% user/nice) for the next 20 minutes, and that pattern repeats once. We then see overall CPU use climb to about 60% (~20% system; ~40% user/nice) for about 1:20. Then there's a period of about 2:00 where overall CPU is at about 40% (~30% system; ~10% user/nice). Based on earlier work I did, where I was able to do a similar build in a native FreeBSD/i386 (no PAE) enviroment on the same hardware (but when it still only had 6GB RAM), and I managed to get the build done in 2:47, I believe that getting more work done in parallel in this 2:00 period is a key to improving performance: the 2:47 result showed that period to be a very busy one for the CPU. But I am at a loss to understand what might be preventing the work form getting done (in a timely fashion). I believe that there were some commits made to stable/9 (MFCed from head) a few months ago to significantly reduce the overhead of using jails or using nullfs (or both). And I'm looking forward to being able to test that -- but I need to get a "fixed" 8.x environment deployed first, and a 12% increase in build times is not something that is likely to be well-received. Help? Peace, david --=20 David H. Wolfskill david@catwhisker.org Taliban: Evil cowards with guns afraid of truth from a 14-year old girl. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --BXVAT5kNtrzKuDFl Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iEYEARECAAYFAlJS7yMACgkQmprOCmdXAD3WkQCcCErKrKm8i72ycj17dDo89KFO F0kAn2GF/T0fsJeLznJMyZn1ijQ90rfO =xORX -----END PGP SIGNATURE----- --BXVAT5kNtrzKuDFl-- From owner-freebsd-performance@FreeBSD.ORG Mon Oct 7 17:32:58 2013 Return-Path: Delivered-To: performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 961E74F7 for ; Mon, 7 Oct 2013 17:32:58 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-qe0-x22e.google.com (mail-qe0-x22e.google.com [IPv6:2607:f8b0:400d:c02::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 56A652433 for ; Mon, 7 Oct 2013 17:32:58 +0000 (UTC) Received: by mail-qe0-f46.google.com with SMTP id x7so5573081qeu.33 for ; Mon, 07 Oct 2013 10:32:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=kZOR1MZ7aOtqD9a9eGNnD8QgBdFlz+ytjwUeyIp4Y70=; b=Pq3eyanA9frKaNRIiJjhiQ0PYNB5gTEp+kCUP9yt4u7r4HjBD/cbBhNX02gOmN3Sqb UM9rF8XjaCu1d6U2qyYlSmUenxL2Rhqq2IBUhRfNKV/VUYF/BiiquDf6pAKZPAPUPK/R mcVesIY779R87XQuxJ0FmRQMp9hKUZHof4IswQSAiW0czhO27EmRd0Ia5RQgZjPE/ke+ pGpRasTMnyAgqoRLCMCCf4BlHSTxH5NYKujuJ8R4EbeAzU/E3kd3cZzg4T8VlspUfzdn iyfDczn85Q2Gfv/TDnKR9WcVQuvFmcgQSUh5Nvt1KiWlYha635NUyx0cxSXRB2FQMSD2 oVNw== MIME-Version: 1.0 X-Received: by 10.224.63.199 with SMTP id c7mr34342619qai.74.1381167177514; Mon, 07 Oct 2013 10:32:57 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.207.66 with HTTP; Mon, 7 Oct 2013 10:32:57 -0700 (PDT) Received: by 10.224.207.66 with HTTP; Mon, 7 Oct 2013 10:32:57 -0700 (PDT) In-Reply-To: <20131007172804.GA7641@albert.catwhisker.org> References: <20131007172804.GA7641@albert.catwhisker.org> Date: Mon, 7 Oct 2013 10:32:57 -0700 X-Google-Sender-Auth: NUSqrWGVYHVExrHEW61i4zxTrAc Message-ID: Subject: Re: Apparent performance regression 8.3@ -> 8.4@r255966? From: Adrian Chadd To: performance@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Oct 2013 17:32:58 -0000 On Oct 7, 2013 1:28 PM, "David Wolfskill" wrote: > > At work, we have a bunch of machines that developers use to build some > software. The machines presently run FreeBSD/amd64 8.3-STABLE @rxxxxxx > (with a few local patches, which have since been committed to stable/8), > and the software is built within a 32-bit jail. Well, whats the start commit? Xxxxxx isnt all that helpful. :-) Once you can establish that, we xan go over the commit logs to see what changed. Youre building exactly the same software on both, right? -adrian > The hardware includes 2 packages of 6 physical cores each @3.47GHz > (Intel X5690); SMT is enabled (so the scheduler sees hw.ncpu == > 24). The memory on the machines was recently increased from 6GB > to 96GB. > > I am trying to set up a replacement host environment on my test machine; > the current environment there is FreeBSD/amd64 8.4-STABLE @r255966; this > environment achieves a couple of objectives: > > * It has no local patches. > * The known problems (e.g., with mfiutil failing to report battery > status accurately) are believed to be addressed appropriately. > > However: when I do comparison software builds, the new environment is > taking about 12% longer to perform the same work (comparing against a > fair sample of the deployed machines): > > > Now, when I do these builds, I do so under /usr/bin/time, as well > as using a bit of "scaffolding" I cobbled up (a few years back) > that basically samples a bunch of sysctl OIDs periodically (by > default, every 10 seconds). Once the build is done, I grab the > file that has the sampled OID data and bring it to my desktop machine > to post-process it; I generate graphs showing (aggregate and per-core) > CPU utilization, as well as Load Averages over the course of the > build. I can also generate graphs that show how the memory statistics > that "top" displays vary during the course of the build, as well as just > about any univariate OID, and quite a few simple multivariate OIDs > (e.g., kern.cp_time, kern.cp_times, and vm.loadavg). > > After seeing the above results and poking around looking for > somewhat-recent tuning information, I ran across a suggestion that the > default of 2MB for vfs.ufs.dirhash_maxmem was probably on the low side. > So I started sampling both vfs.ufs.dirhash_maxmem (mostly to make > documentation of the configuration for a test run easier) and > vfs.ufs.dirhash_mem (to see what we were actually using). And I tried > quadrupling vfs.ufs.dirhash_maxmem (to 8MB). > > The next time I tried a test build, I found that vfs.ufs.dirhash_mem > started at about 3.8MB, climbed fairly steadily, then "clipped" at > 8MB, so I quadrupled it again (to 32MB), and found that it climbed > to almost 12MB, then dropped precipitously to about 400KB (and > oscillated between about 400KB & 20MB for the rest of the build, > which appears to be the "packaging" phase). > > Despite that increase in vfs.ufs.dirhash_maxmem, this does not > appear to have measurably affected the build times. > > In examining the CPU utilization graphs, the CPU generally looks > about 5% busy for the first 15 minutes; this would be bmake determining > dependency graphs, I expect. For the next 1:20, CPU is about 70% > busy (~15% system; ~65% user/nice) for about 20 minutes, then drops > to about 45% busy (~25% system; ~20% user/nice) for the next 20 > minutes, and that pattern repeats once. > > We then see overall CPU use climb to about 60% (~20% system; ~40% > user/nice) for about 1:20. > > Then there's a period of about 2:00 where overall CPU is at about 40% > (~30% system; ~10% user/nice). > > Based on earlier work I did, where I was able to do a similar build in a > native FreeBSD/i386 (no PAE) enviroment on the same hardware (but when > it still only had 6GB RAM), and I managed to get the build done in 2:47, > I believe that getting more work done in parallel in this 2:00 period is > a key to improving performance: the 2:47 result showed that period to be > a very busy one for the CPU. > > But I am at a loss to understand what might be preventing the work form > getting done (in a timely fashion). > > I believe that there were some commits made to stable/9 (MFCed from > head) a few months ago to significantly reduce the overhead of using > jails or using nullfs (or both). And I'm looking forward to being able > to test that -- but I need to get a "fixed" 8.x environment deployed > first, and a 12% increase in build times is not something that is likely > to be well-received. > > Help? > > Peace, > david > -- > David H. Wolfskill david@catwhisker.org > Taliban: Evil cowards with guns afraid of truth from a 14-year old girl. > > See http://www.catwhisker.org/~david/publickey.gpg for my public key. From owner-freebsd-performance@FreeBSD.ORG Mon Oct 7 17:49:31 2013 Return-Path: Delivered-To: performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 0382C99C; Mon, 7 Oct 2013 17:49:31 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (mx.catwhisker.org [198.144.209.73]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id AD8BA2532; Mon, 7 Oct 2013 17:49:30 +0000 (UTC) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.14.7/8.14.7) with ESMTP id r97HnTrY008260; Mon, 7 Oct 2013 10:49:29 -0700 (PDT) (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.14.7/8.14.7/Submit) id r97HnTwT008259; Mon, 7 Oct 2013 10:49:29 -0700 (PDT) (envelope-from david) Date: Mon, 7 Oct 2013 10:49:29 -0700 From: David Wolfskill To: Adrian Chadd Subject: Re: Apparent performance regression 8.3@ -> 8.4@r255966? Message-ID: <20131007174929.GO1611@albert.catwhisker.org> References: <20131007172804.GA7641@albert.catwhisker.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="31zvzas5NXT9fief" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: performance@freebsd.org X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Oct 2013 17:49:31 -0000 --31zvzas5NXT9fief Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Oct 07, 2013 at 10:32:57AM -0700, Adrian Chadd wrote: > On Oct 7, 2013 1:28 PM, "David Wolfskill" wrote: > > > > At work, we have a bunch of machines that developers use to build some > > software. The machines presently run FreeBSD/amd64 8.3-STABLE @rxxxxxx > > (with a few local patches, which have since been committed to stable/8), > > and the software is built within a 32-bit jail. >=20 > Well, whats the start commit? Xxxxxx isnt all that helpful. :-) r238914M -- sorry; I got impatient & sent th emessage off before I went through & plugged in the stuff I meant to plug in later.... :-( > Once you can establish that, we xan go over the commit logs to see what > changed. >=20 > Youre building exactly the same software on both, right? Yes: I have a script that checks out a "sandbox" from the repo, then tars it up. Each test iteration then: * Ensures there is nothing of an old sandbox (directory) around. * Unpacks the pristine copy of the sandbox. * Dives into the sandbox & performs a measured build. * Exits the sandbox (and removes it). > .... Here's the other bit I neglected to include: dwolf-fbsd(9.2-S)[30] /usr/bin/ministat -s baseline test_env x baseline + test_env +--------------------------------------------------------------------------= ----+ | xxxxx xx xxxxxxxx x xx x + = | |xxxxxxxx xx xxxxxxxx x xx xxxx x x +x *+ + ++ = x| | |___________A____________| = | | |____A____| = | +--------------------------------------------------------------------------= ----+ N Min Max Median Avg Stddev x 75 4.91961 6.16419 5.22432 5.2311616 0.20113192 + 7 5.68327 5.94078 5.86111 5.8563686 0.086090721 Difference at 95.0% confidence 0.625207 +/- 0.153262 11.9516% +/- 2.92979% (Student's t, pooled s =3D 0.194874) Peace, david --=20 David H. Wolfskill david@catwhisker.org Taliban: Evil cowards with guns afraid of truth from a 14-year old girl. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --31zvzas5NXT9fief Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iEYEARECAAYFAlJS9CgACgkQmprOCmdXAD22ugCdGUxYzWW+Q63QrNz1seNgIU8j anYAn0Q7+0yl9fREXDYPW92y8W2+bh53 =5vEF -----END PGP SIGNATURE----- --31zvzas5NXT9fief-- From owner-freebsd-performance@FreeBSD.ORG Mon Oct 7 19:27:14 2013 Return-Path: Delivered-To: performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 9AC83906 for ; Mon, 7 Oct 2013 19:27:14 +0000 (UTC) (envelope-from erik+lists@cederstrand.dk) Received: from csmtp12.one.com (csmtp12.one.com [195.47.247.112]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5B5EB2BD9 for ; Mon, 7 Oct 2013 19:27:14 +0000 (UTC) Received: from [192.168.1.82] (unknown [176.222.238.90]) by csmtp12.one.com (Postfix) with ESMTPA id BFDBB4000B701 for ; Mon, 7 Oct 2013 19:19:36 +0000 (UTC) Received: from [192.168.1.82] ([UNAVAILABLE]. [176.222.238.90]) (using TLSv1 with cipher AES128-SHA) by 0.0.0.0:587 (trex/4.8.87); Mon, 07 Oct 2013 19:09:25 GMT Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\)) Subject: Re: Apparent performance regression 8.3@ -> 8.4@r255966? From: Erik Cederstrand In-Reply-To: <20131007172804.GA7641@albert.catwhisker.org> Date: Mon, 7 Oct 2013 21:19:38 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <172AF9A7-53F7-42CF-A164-084CFB418EE9@cederstrand.dk> References: <20131007172804.GA7641@albert.catwhisker.org> To: performance@freebsd.org X-Mailer: Apple Mail (2.1510) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Oct 2013 19:27:14 -0000 Den 07/10/2013 kl. 19.28 skrev David Wolfskill : > In examining the CPU utilization graphs, the CPU generally looks > about 5% busy for the first 15 minutes; this would be bmake = determining > dependency graphs, I expect. Is that one process using 100% of one core, or many processes using 5% = total? > Based on earlier work I did, where I was able to do a similar build in = a > native FreeBSD/i386 (no PAE) enviroment on the same hardware (but when > it still only had 6GB RAM), and I managed to get the build done in = 2:47, > I believe that getting more work done in parallel in this 2:00 period = is > a key to improving performance: the 2:47 result showed that period to = be > a very busy one for the CPU. You need to know where your bottlenecks are during the build. Since you = have lots of RAM, you could try to rule out differences in filesystem = access by placing your jail on an mfs and building your software off = that. If that improves build times, then you're IO bound at least some = of the time. You should be logging disk access along with CPU and memory = during the build. Erik= From owner-freebsd-performance@FreeBSD.ORG Mon Oct 7 19:44:13 2013 Return-Path: Delivered-To: performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 950E8E48 for ; Mon, 7 Oct 2013 19:44:13 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (mx.catwhisker.org [198.144.209.73]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 3DC962CE9 for ; Mon, 7 Oct 2013 19:44:12 +0000 (UTC) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.14.7/8.14.7) with ESMTP id r97Ji6b0009084; Mon, 7 Oct 2013 12:44:06 -0700 (PDT) (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.14.7/8.14.7/Submit) id r97Ji5ej009083; Mon, 7 Oct 2013 12:44:05 -0700 (PDT) (envelope-from david) Date: Mon, 7 Oct 2013 12:44:05 -0700 From: David Wolfskill To: Erik Cederstrand Subject: Re: Apparent performance regression 8.3@ -> 8.4@r255966? Message-ID: <20131007194405.GP1611@albert.catwhisker.org> References: <20131007172804.GA7641@albert.catwhisker.org> <172AF9A7-53F7-42CF-A164-084CFB418EE9@cederstrand.dk> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Vhqu5qQ3bjg0ZI9i" Content-Disposition: inline In-Reply-To: <172AF9A7-53F7-42CF-A164-084CFB418EE9@cederstrand.dk> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: performance@freebsd.org X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Oct 2013 19:44:13 -0000 --Vhqu5qQ3bjg0ZI9i Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Oct 07, 2013 at 09:19:38PM +0200, Erik Cederstrand wrote: > ...=20 > > In examining the CPU utilization graphs, the CPU generally looks > > about 5% busy for the first 15 minutes; this would be bmake determining > > dependency graphs, I expect. >=20 > Is that one process using 100% of one core, or many processes using 5% to= tal? It's closer to the first, but "which core" varies a bit over time. (In one specific instance, core 17, then 13 are at near 100% within that first 15-minute interval.) > > Based on earlier work I did, where I was able to do a similar build in a > > native FreeBSD/i386 (no PAE) enviroment on the same hardware (but when > > it still only had 6GB RAM), and I managed to get the build done in 2:47, > > I believe that getting more work done in parallel in this 2:00 period is > > a key to improving performance: the 2:47 result showed that period to be > > a very busy one for the CPU. >=20 > You need to know where your bottlenecks are during the build. Indeed. :-} > Since you have lots of RAM, you could try to rule out differences in file= system access by placing your jail on an mfs and building your software off= that. That's an experiment that I've been advocating, but 96GB isn't enough. > If that improves build times, then you're IO bound at least some of the t= ime. You should be logging disk access along with CPU and memory during the= build. FWIW, I'm seeing very little CPU time spent in interrupt mode. I'll see if I can find a reasonable way to munge the output of iostat to be usable for the purpose, I suppose; thanks. Peace, david --=20 David H. Wolfskill david@catwhisker.org Taliban: Evil cowards with guns afraid of truth from a 14-year old girl. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --Vhqu5qQ3bjg0ZI9i Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iEYEARECAAYFAlJTDwQACgkQmprOCmdXAD2nfACdGhiKwCZi8hR6WKHcNIFXaTWP CbUAn04e5LE/qsBw+14sNxZLakDGVFNZ =kGWq -----END PGP SIGNATURE----- --Vhqu5qQ3bjg0ZI9i-- From owner-freebsd-performance@FreeBSD.ORG Thu Oct 10 14:03:10 2013 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 4109C75C for ; Thu, 10 Oct 2013 14:03:10 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from plane.gmane.org (plane.gmane.org [80.91.229.3]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id F0B8C2E88 for ; Thu, 10 Oct 2013 14:03:09 +0000 (UTC) Received: from list by plane.gmane.org with local (Exim 4.69) (envelope-from ) id 1VUGpR-0002V7-7X for freebsd-performance@freebsd.org; Thu, 10 Oct 2013 16:03:01 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 10 Oct 2013 16:03:01 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 10 Oct 2013 16:03:01 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Subject: Re: Apparent performance regression 8.3@ -> 8.4@r255966? Date: Thu, 10 Oct 2013 16:02:47 +0200 Lines: 64 Message-ID: References: <20131007172804.GA7641@albert.catwhisker.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="tan4eEgcA0ARhxdF6Q22T9nn4d6xgi4j1" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.0 In-Reply-To: <20131007172804.GA7641@albert.catwhisker.org> X-Enigmail-Version: 1.5.2 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Oct 2013 14:03:10 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --tan4eEgcA0ARhxdF6Q22T9nn4d6xgi4j1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 07/10/2013 19:28, David Wolfskill wrote:> At work, we have a bunch of machines that developers use to build some > software. The machines presently run FreeBSD/amd64 8.3-STABLE @rxxxxxx= > (with a few local patches, which have since been committed to stable/8)= , > and the software is built within a 32-bit jail. >=20 > The hardware includes 2 packages of 6 physical cores each @3.47GHz > (Intel X5690); SMT is enabled (so the scheduler sees hw.ncpu =3D=3D > 24). The memory on the machines was recently increased from 6GB > to 96GB. >=20 > I am trying to set up a replacement host environment on my test machine= ; > the current environment there is FreeBSD/amd64 8.4-STABLE @r255966; thi= s > environment achieves a couple of objectives: >=20 > * It has no local patches. > * The known problems (e.g., with mfiutil failing to report battery > status accurately) are believed to be addressed appropriately. >=20 > However: when I do comparison software builds, the new environment is > taking about 12% longer to perform the same work (comparing against a > fair sample of the deployed machines): So, the test machine is exactly the same as the old machines? Does the hardware upgrade coincide with 8.4-STABLE upgrade? At a guess, you also might be hitting a problem with either NUMA (which would mean the difference you encountered is pretty much random, depending on how the memory from your processes was allocated), or a generic scheduler issue (IIRC, FreeBSD 9 series was found to be much more scalable for > 16 CPUs). Just a thought - you *could* set up an 8-STABLE jail in a 9-STABLE environment if you need the 8-STABLE libraries for your software. --tan4eEgcA0ARhxdF6Q22T9nn4d6xgi4j1 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.21 (FreeBSD) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iKYEARECAGYFAlJWs4dfFIAAAAAALgAoaXNzdWVyLWZwckBub3RhdGlvbnMub3Bl bnBncC5maWZ0aGhvcnNlbWFuLm5ldDYxNDE4MkQ3ODMwNDAwMDJFRUIzNDhFNUZE MDhENTA2M0RGRjFEMkMACgkQ/QjVBj3/HSxBPACfY+scBQjkMxxPywFfWftacJeE G30AniKt8IrNTlfzGdFb19ANuKwrQIv6 =d/s4 -----END PGP SIGNATURE----- --tan4eEgcA0ARhxdF6Q22T9nn4d6xgi4j1-- From owner-freebsd-performance@FreeBSD.ORG Fri Oct 11 03:41:51 2013 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id B03CE987; Fri, 11 Oct 2013 03:41:51 +0000 (UTC) (envelope-from julian@elischer.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 82C8022E4; Fri, 11 Oct 2013 03:41:51 +0000 (UTC) Received: from jre-mbp.elischer.org (ppp121-45-246-96.lns20.per2.internode.on.net [121.45.246.96]) (authenticated bits=0) by vps1.elischer.org (8.14.7/8.14.7) with ESMTP id r9B3eqQ2048754 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Thu, 10 Oct 2013 20:40:54 -0700 (PDT) (envelope-from julian@elischer.org) Message-ID: <52577342.4090801@elischer.org> Date: Fri, 11 Oct 2013 11:40:50 +0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130801 Thunderbird/17.0.8 MIME-Version: 1.0 To: Ivan Voras Subject: Re: Apparent performance regression 8.3@ -> 8.4@r255966? References: <20131007172804.GA7641@albert.catwhisker.org> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org, David Wolfskill X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Oct 2013 03:41:51 -0000 On 10/10/13 10:02 PM, Ivan Voras wrote: > On 07/10/2013 19:28, David Wolfskill wrote:> At work, we have a bunch of > machines that developers use to build some >> software. The machines presently run FreeBSD/amd64 8.3-STABLE @rxxxxxx >> (with a few local patches, which have since been committed to stable/8), >> and the software is built within a 32-bit jail. >> >> The hardware includes 2 packages of 6 physical cores each @3.47GHz >> (Intel X5690); SMT is enabled (so the scheduler sees hw.ncpu == >> 24). The memory on the machines was recently increased from 6GB >> to 96GB. >> >> I am trying to set up a replacement host environment on my test machine; >> the current environment there is FreeBSD/amd64 8.4-STABLE @r255966; this >> environment achieves a couple of objectives: >> >> * It has no local patches. >> * The known problems (e.g., with mfiutil failing to report battery >> status accurately) are believed to be addressed appropriately. >> >> However: when I do comparison software builds, the new environment is >> taking about 12% longer to perform the same work (comparing against a >> fair sample of the deployed machines): > So, the test machine is exactly the same as the old machines? Does the > hardware upgrade coincide with 8.4-STABLE upgrade? > > At a guess, you also might be hitting a problem with either NUMA (which > would mean the difference you encountered is pretty much random, > depending on how the memory from your processes was allocated), or a > generic scheduler issue (IIRC, FreeBSD 9 series was found to be much > more scalable for > 16 CPUs). > > Just a thought - you *could* set up an 8-STABLE jail in a 9-STABLE > environment if you need the 8-STABLE libraries for your software. > > > OR, take the new kernel and use it to boot the old system then compare times. From owner-freebsd-performance@FreeBSD.ORG Fri Oct 11 04:57:56 2013 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id F33714D8; Fri, 11 Oct 2013 04:57:55 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (mx.catwhisker.org [198.144.209.73]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 973D025EA; Fri, 11 Oct 2013 04:57:55 +0000 (UTC) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.14.7/8.14.7) with ESMTP id r9B4vsGi036050; Thu, 10 Oct 2013 21:57:54 -0700 (PDT) (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.14.7/8.14.7/Submit) id r9B4vs2X036049; Thu, 10 Oct 2013 21:57:54 -0700 (PDT) (envelope-from david) Date: Thu, 10 Oct 2013 21:57:54 -0700 From: David Wolfskill To: Ivan Voras Subject: Re: Apparent performance regression 8.3@ -> 8.4@r255966? Message-ID: <20131011045754.GB1611@albert.catwhisker.org> References: <20131007172804.GA7641@albert.catwhisker.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="WxvyIW12Aj7m0eTB" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-performance@freebsd.org X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Oct 2013 04:57:56 -0000 --WxvyIW12Aj7m0eTB Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Oct 10, 2013 at 04:02:47PM +0200, Ivan Voras wrote: > On 07/10/2013 19:28, David Wolfskill wrote:> At work, we have a bunch of > machines that developers use to build some > > software. The machines presently run FreeBSD/amd64 8.3-STABLE @rxxxxxx > > (with a few local patches, which have since been committed to stable/8), > > and the software is built within a 32-bit jail. > .... > So, the test machine is exactly the same as the old machines? Does the > hardware upgrade coincide with 8.4-STABLE upgrade? The only hardware upgrade was to increase RAM from 6GB to 96GB, which was done for all of the machines being discussed. The machine I'm using for testing configurations now is the one I used to test configurations before we bought the machines we've now deployed. I cite 8.4 as a (vague) reference point; more specifically, I used a snapshot of stable/8. More precisely, it's stab le/8 @r255978. > At a guess, you also might be hitting a problem with either NUMA (which > would mean the difference you encountered is pretty much random, > depending on how the memory from your processes was allocated), or a > generic scheduler issue (IIRC, FreeBSD 9 series was found to be much > more scalable for > 16 CPUs). I'm hoping to demonstrate that -- but first, I need to get demonstrate something that fixes some known bugs while not saddling the developers with a 12% increase in build times. And I've been trying (when my test machine has been available to me) to get this done for nearly a year, now. (There is also a perception that "jumping" from 8.x to 9.x is "scary". I am reminded of the "It's just a leaf!" sequence near the beginning of "A Bug's Life.") > Just a thought - you *could* set up an 8-STABLE jail in a 9-STABLE > environment if you need the 8-STABLE libraries for your software. The "32-bit jail" is actually a 7.1-R [+ a few patches] environment as it is. Thanks for responding. :-} Peace, david --=20 David H. Wolfskill david@catwhisker.org Taliban: Evil cowards with guns afraid of truth from a 14-year old girl. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --WxvyIW12Aj7m0eTB Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iEYEARECAAYFAlJXhVEACgkQmprOCmdXAD1LXQCfS6CMqp2loPP6ymKADk960/IZ 6KgAn3BsKmX1gsSJYTK9IZuKtvDq41MU =vCRv -----END PGP SIGNATURE----- --WxvyIW12Aj7m0eTB-- From owner-freebsd-performance@FreeBSD.ORG Fri Oct 11 05:20:55 2013 Return-Path: Delivered-To: performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 367278CB for ; Fri, 11 Oct 2013 05:20:55 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-qc0-x233.google.com (mail-qc0-x233.google.com [IPv6:2607:f8b0:400d:c01::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id EA91B26D7 for ; Fri, 11 Oct 2013 05:20:54 +0000 (UTC) Received: by mail-qc0-f179.google.com with SMTP id l4so2580086qcv.24 for ; Thu, 10 Oct 2013 22:20:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=Hrh2y09H1d4lOQKBfdMER/oABW2piKRkqIDzNDn4izY=; b=LDzYcDnJur5CXIHoIfZV9QfwbEa0vbmm5KsFvAm9KM+wwb0kTVsQFLNoMLUtN5STHr 2+o2r9MNM+VtNzSywN9k3HP7hNQkzzgdXJ+G84uR5FZgZ0R/G2Ska8ngHMgX7jWN6CYQ sZdHLJich8Y+ln9UBujbcWQDR3nODzB8ushZaCKQcrbLmvt8P+uCZKwvl3nwFqE97x4r MPZH70X+VVpWqh2u9l4iDmWD38feVsob7p0vA4IVjZvDzimMH6CS6i9tiItrcPa+akTr gRoWLaLdUd/2G8I320jaRsDjCZZYoiwOvV4j3sCRVKcpmf7kXlRCGQWDFqJOmh0Mterz Qmmg== MIME-Version: 1.0 X-Received: by 10.49.3.3 with SMTP id 3mr5357265qey.56.1381468854137; Thu, 10 Oct 2013 22:20:54 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.207.66 with HTTP; Thu, 10 Oct 2013 22:20:53 -0700 (PDT) In-Reply-To: <20131007194405.GP1611@albert.catwhisker.org> References: <20131007172804.GA7641@albert.catwhisker.org> <172AF9A7-53F7-42CF-A164-084CFB418EE9@cederstrand.dk> <20131007194405.GP1611@albert.catwhisker.org> Date: Thu, 10 Oct 2013 22:20:53 -0700 X-Google-Sender-Auth: _v8IvblGbRftXjOAMBAouZfgZsQ Message-ID: Subject: Re: Apparent performance regression 8.3@ -> 8.4@r255966? From: Adrian Chadd To: David Wolfskill Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: performance@freebsd.org, Erik Cederstrand X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Oct 2013 05:20:55 -0000 Hi David, How about just picking a revision of stable/8 half way between the good and not good version, then re-test? It'd help to narrow down the range of commits that could've caused problems. -adrian