From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 12:56:03 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 04486106564A; Thu, 15 Dec 2011 12:56:03 +0000 (UTC) (envelope-from sjg@evilcode.net) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 9B0658FC12; Thu, 15 Dec 2011 12:56:02 +0000 (UTC) Received: by iakl21 with SMTP id l21so5157427iak.13 for ; Thu, 15 Dec 2011 04:56:02 -0800 (PST) MIME-Version: 1.0 Received: by 10.42.161.10 with SMTP id r10mr2375060icx.22.1323952367276; Thu, 15 Dec 2011 04:32:47 -0800 (PST) Received: by 10.50.163.106 with HTTP; Thu, 15 Dec 2011 04:32:47 -0800 (PST) In-Reply-To: References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> Date: Thu, 15 Dec 2011 05:32:47 -0700 Message-ID: From: "Samuel J. Greear" To: Adrian Chadd Content-Type: text/plain; charset=ISO-8859-1 Cc: FreeBSD Stable Mailing List , freebsd-performance@freebsd.org, Current FreeBSD , "O. Hartmann" , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 12:56:03 -0000 > Well, the only way it's going to get fixed is if someone sits down, > replicates it, and starts to document exactly what it is that these > benchmarks are/aren't doing. > I think you will find that investigation is largely a waste of time, because not only are some of these benchmarks just downright silly, there are huge differences in the environments (compiler versions), etc., etc. leading to a largely apples/oranges comparison. But also the the analysis and reporting of the results by Phoronix is simply moronic to the point of being worse than useful, they are spreading misinformation. Take the first test as an example, Blogbench read. This doesn't raise any red flags, right? At least not until you realize that Blogbench isn't a read test, it's a read/write test. So what they have done here is run a read/write test and then thrown away the write results for both platforms and reported only the read results. If you dig down into the actual results, http://openbenchmarking.org/result/1112113-AR-ORACLELIN37 -- you will see two Blogbench numbers, one for read and another for write. These were both taken from the same Blogbench run, so FreeBSD optimizes writes over reads, that's probably a good thing for your data but a bad thing when someone totally misrepresents benchmark results. Other benchmarks in the Phoronix suite and their representations are similarly flawed, _ALL_ of these results should be ignored and no time should be wasted by any FreeBSD committer further evaluating this garbage. (Yes, I have been down this rabbit hole). Best, Sam