From owner-freebsd-performance@FreeBSD.ORG Wed Jun 12 22:58:51 2013 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id CC0A2D48 for ; Wed, 12 Jun 2013 22:58:51 +0000 (UTC) (envelope-from obrien@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id AC36C192D; Wed, 12 Jun 2013 22:58:51 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r5CMwpaA024261; Wed, 12 Jun 2013 22:58:51 GMT (envelope-from obrien@freefall.freebsd.org) Received: (from obrien@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r5CMwpJw024258; Wed, 12 Jun 2013 22:58:51 GMT (envelope-from obrien) Date: Wed, 12 Jun 2013 15:58:49 -0700 From: "David O'Brien" To: freebsd-performance@freebsd.org Subject: Scaling and performance issues with FreeBSD 9 (& 10) on 4 socket systems Message-ID: <20130612225849.GA2858@dragon.NUXI.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.20 (2009-06-14) Cc: Simon Gerraty X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: obrien@freebsd.org List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Jun 2013 22:58:51 -0000 Hi all, $WORK is looking to see what throwing money at a build can do. We've been building the product on FreeBSD 8.3/amd64 in a 7.1/i386 jail, on on Supermicro X8DT3 http://www.supermicro.com/products/motherboard/qpi/5500/x8dt3.cfm w/Xeon X5690 3.47GHz: "2 package(s) x 6 core(s) x 2 SMT threads". These are nicknamed "Montana-2" or "M2". The new machines are Supermicro X9QR7-TF+/X9QRi-F+ http://www.supermicro.com/products/motherboard/Xeon/C600/X9QR7-TF_.cfm w/Xeon E5-4650 2.70GHz: "4 package(s) x 8 core(s) x 2 SMT threads". These are nicknamed "jbm" (build machine) during their testing phase. As part of this we also upgraded the host FreeBSD to 9.1 from 8.3. We've found that a "make -j28" build on the M2 machines is considerably faster than on the jbm machines. While somewhat faster might not be surprising given the faster CPU's, the build is 2x longer on jbm. (jbm runs are FreeBSD 9.1/amd64 with the same 7.1/i386 jail installation as the M2) In addition, we've found the scaling on jbm to be quite bad as additional jobs or threads or processes are run. ----------%<----------%<----------%<----------%<----------%<---------- In order to better quantify this, a co-worker who's been playing with Bitcoin suggested the Bitcoin "vanitygen" multi-threaded application as a good test. We have results of 1-64 threads on a jbm machine running (1) FreeBSD 9.1/amd64, (2) FreeBSD 10-current/amd64 [no Witness], (3) FreeBSD 8.4/amd64, (4) FreeBSD 10-current/amd64 SCHED_4BSD, & (5) Fedora 18/amd64. The results are graphed at http://people.freebsd.org/~obrien/jbm/vanitygen/vanity-perf-graph.png We found FreeBSD 8.4 to perform better than FreeBSD 9.1, and Linux considerably better than both on the same machine. ----------%<----------%<----------%<----------%<----------%<---------- To get more data, I ran /usr/ports/benchmarks/sysbench/ and see similar concerning results between the M2 machines and jbm machines (on the native amd64 host, not within the i386 jail). This is a multi-threaded application. The graph for that is http://people.freebsd.org/~obrien/jbm/sysbench/sysbench.png ----------%<----------%<----------%<----------%<----------%<---------- Simon also did some benchmarking using an integrity file signing server. This one is a multi-process benchmark doing 2k RSA+SHA1 calculations. The graph for that is http://people.freebsd.org/~obrien/jbm/sigs/sigs.png ----------%<----------%<----------%<----------%<----------%<---------- We've tried various things and haven't been able to explain why FreeBSD isn't scaling on the new hardware. Nor why it performs so much worse than FreeBSD on the older "M2" machines. Thoughts? -- -- David (obrien@FreeBSD.org) P.S. vanitygen is from 'git clone git://github.com/samr7/vanitygen.git' and builds with this patch: --- Makefile.ORIG 2013-06-06 21:39:10.000000000 -0700 +++ Makefile 2013-06-06 21:38:45.000000000 -0700 @@ -1,2 +1,2 @@ -LIBS=-lpcre -lcrypto -lm -lpthread -CFLAGS=-ggdb -O3 -Wall +LIBS=-L/usr/local/lib -lpcre -lcrypto -lm -lpthread +CFLAGS=-ggdb -O3 -Wall -I/usr/local/include