Date: Fri, 14 Feb 2003 06:53:09 -0800 From: "Sam Leffler" <sam@errno.com> To: "Brad Knowles" <brad.knowles@skynet.be> Cc: "Chris BeHanna" <chris@pennasoft.com>, <current@FreeBSD.ORG> Subject: Re: 5-STABLE Roadmap Message-ID: <26dd01c2d438$d37f0810$52557f42@errno.com> References: <200302140036.h1E0aK3q071051@freefall.freebsd.org> <a05200f0dba72122437d7@[10.0.1.2]> <25c301c2d3e1$8f2e3e30$52557f42@errno.com> <200302140028.21669.chris@pennasoft.com> <265d01c2d3ec$912bbae0$52557f42@errno.com> <a05200f0fba72952f3afd@[10.0.1.2]>
next in thread | previous in thread | raw e-mail | index | archive | help
> At 9:47 PM -0800 2003/02/13, Sam Leffler wrote:
>
> >> SpecFS (NFS ops/sec benchmark)
> >
> > List price on SPEC SFS97 R1 is $900. And my recollection is that it
was
> > involved to setup and run.
>
> $450 for educational organizations. Wouldn't the FreeBSD
> Foundation qualify?
>
The point was that they cost $$$. Not an option for many developers.
> > Benchmarks must be unencumbered; be easy to setup+run by one person;
and not
> > require lots of equipment. For the most part we are looking for
benchmarks
> > that will help tune system performance; not generate press releases.
>
> Some of the more interesting benchmarks take more hardware to
> properly run. They're not necessarily particularly hard to setup,
> but they do want client machines to be used to generate test load.
>
> Rick Jones had to use 20 client machines with netperf in order to
> find the limits of performance for Nominum Authoritative Name Server
> (ANS), and he works for HP.
>
>
> I think you need to decide just how thorough you want your
> testing to be. If it's all going to be just single people running
> benchmarks on single machines, I think that there are going to be a
> lot of things you may miss.
>
> I have this problem myself with the benchmarks I've been running
> in conjunction with the invited talks I did at LISA 2002 and BSDCon
> Europe 2002, and people have repeatedly called me to task on this
> issue.
Benchmarks for tuning are frequently different in nature than benchmarks for
comparing systems. When tuning or for regression testing you usually want
something that's easy to setup and runs fast enough to give you results
quickly. For comparisons you usually care more about things like coverage,
how well it models real-world behaviour, etc.
My point was that we're presently trying to identify good benchmarks to use
in comparing performance between -stable and -current. But these must also
be ones that we can use for tuning either by reducing the configuration or
otherwise directing the activity. Microbenchmarks are valuable here and
have already been heavily used. We're at the point where we need something
that exercises the system on a bit larger scale. Eventually we'll get to
the point where large-scale benchmarks are worth running. Note that these
issues have nothing to do with single machine vs. multiple machines.
Of course what we really need more than benchmarks are people to actually
follow through on the results and fix the problems...
Sam
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-current" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?26dd01c2d438$d37f0810$52557f42>