From owner-freebsd-current@FreeBSD.ORG Wed Dec 21 22:34:07 2011 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3C032106566B for ; Wed, 21 Dec 2011 22:34:07 +0000 (UTC) (envelope-from mozolevsky@gmail.com) Received: from mail-pz0-f54.google.com (mail-pz0-f54.google.com [209.85.210.54]) by mx1.freebsd.org (Postfix) with ESMTP id 0F04D8FC18 for ; Wed, 21 Dec 2011 22:34:06 +0000 (UTC) Received: by dakp5 with SMTP id p5so7405285dak.13 for ; Wed, 21 Dec 2011 14:34:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type; bh=6TjhqpSWVgLVcHiKr1TZqlLxb2ZeQ6Kidx1BqyS1nu4=; b=umbKknPOESSPo5nMJsEGeEEt/keDF3Di3it7d+PfqE7Kmjt2qHrzh9mRTzGgZ49lFa +OTTMF19SPHZ+WDr7V/60d2CERWj2EHiwb7OiH0N/3GNoQ8DesK8DE5F3vxHSxlIV22+ FiRnzqTdXKkQXzC6ZUtZfpmTKLKotnGQc7Ju0= Received: by 10.68.74.227 with SMTP id x3mr16811455pbv.114.1324506846670; Wed, 21 Dec 2011 14:34:06 -0800 (PST) MIME-Version: 1.0 Sender: mozolevsky@gmail.com Received: by 10.68.12.199 with HTTP; Wed, 21 Dec 2011 14:33:24 -0800 (PST) In-Reply-To: References: <4EF25468.9040204@gmail.com> From: Igor Mozolevsky Date: Wed, 21 Dec 2011 22:33:24 +0000 X-Google-Sender-Auth: g4u_P9THMuKh09I7IrHg0woBnV8 Message-ID: To: Freddie Cash Content-Type: text/plain; charset=UTF-8 Cc: FreeBSD Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Dec 2011 22:34:07 -0000 On 21 December 2011 22:03, Freddie Cash wrote: > On Wed, Dec 21, 2011 at 1:49 PM, Johan Hendriks wrote: >> Nice page, but one thing i do not get is the following. >> >> [quote] >> If you compare FreeBSD / GCC 4.2.1 against, for example, Ubuntu / GCC 4.7 >> then the results are unlikely to tell you anything meaningful about FreeBSD >> vs Ubuntu. >> [/quote] >> >> That is a little strange in my opinion. >> It tells me that FreeBSD falls more and more behind on Linux. >> The reason is or could be that FreeBSD cannot or will not include GCC 4.7 >> and that FreeBSD will not be on par with Linux anymore. > > When benchmarking two systems, you need to make sure that everything > possible is the same (constants) and that the only differences between > the two systems are what you want to benchmark (variables). Yes and no, but to be perfectly frank, the statement, as it stands, is a bit of a nonsense. Let me illustrate in a different way. This is macro~ vs micro~comparison of systems and depends on what you are trying to get out of the benchmark. Using the same argument one can say that Ferrari F430 vs Toyota Prius is a meaningless comparison because the under-the-hood equipment is different. Now, it is absolutely correct to say that in A vs B comparisons, only one thing should be changed and the rest should remain constant. The important thing is, however, to determine the scope of your benchmark: you are not benchmarking a component of A vs a component of B, but you are benchmarking A as a whole system and B as a whole system. Thus, the thing that changes is the system itself. Going back to F430 vs Prius, you first decide what you want to benchmark (acceleration, top speed, fuel consumption, ride comfort, &c) then you measure that aspect in each of the system---you are not looking at the wiring, engine, wheels, &c individually but *at a whole system*. You use the same route, time of day, driver, drive pattern, weather conditions, &c, the only thing that changes is the car! Similarly, FreeBSD vs Linux, you want to a) determine what metric you want to benchmark (NFS throughput, HTTP client handling, SMPT throughput, prime number computation) and b) *scientifically* measure the system against that metric... This would essentially amount to having identical set up and tests, and only changing the hard disks (one containing Linux and another one containing FreeBSD). I don't see why this is such a difficult concept to grasp. Cheers, -- Igor M. :-)