Date: Mon, 20 Sep 1999 19:59:11 +0930 (CST) From: Mark Newton <newton@atdot.dotat.org> To: freebsd-hackers@freebsd.org Subject: RE: Some more commentary and results on 'postmark' (fwd) Message-ID: <199909201029.TAA48175@atdot.dotat.org>
next in thread | raw e-mail | index | archive | help
I forwarded Brad Knowles' comments about the PostMark benchmarking suite to a NetApp engineer I happen to have known since my teens. Selected portions of his comments are included below (with permission) for your entertainment and elucidation. - mark A NetApp person wrote: > Brad Knowles wrote: > > > Both the client and server systems were 90%+ *IDLE* during all tests. > > I/O bound is fun! :) > > > The benchmark has a number of problems. The 'postmark' program > > isn't forking at all, so there is a serious bottleneck in the process > > itself, especially whenever a read is issued. It doesn't really give > > us an accurate representation of a multi-tasking load. Most > > NFS servers have a multitasking load so it isn't really a fair test. > > I don't think Jeff was really all that interested in accurate comparisons of > multi-tasking NFS performance -- for that, we've got SPEC SFS, which has > been the standard method of comparing NFS performance for more than a > decade. > > > The benchmark shows pretty clearly the inefficiency of large UFS > > directories. Putting 20000 files in a single directory is not fun, > > and it seriously skews the test results considering what the benchmark > > is supposed to be testing. > > Jeff's favourite problem domain is mail services, which have traditionally > been lots-of-small-files-in-a-directory stuff. His benchmark reflects that > focus. Thankfully, you can tweak the options to make it reflect some (but > not all) other problem domains. > > > It seems pretty clear to me that this benchmark has been designed > > to show-off the netapp in the best possible light and its competitors > > in the worst possible light. Well, ok, that may be an overly-harsh > > assessment, but it is still true to some degree. > > Actually, it's not true to *any* degree, and I know because I talked to Jeff > Katcher (and his boss at the time, Andy Watson) whilst he was developing it > and writing the white paper. > > PostMark was written to shame Sun and HP into improving their single-task > NFS CLIENT performance by showing them how much better FreeBSD and Linux > performed. Pure and simple. > > Jeff stopped coding as soon as he had something which was vaguely tunable to > reflect different application loads (you don't have to have a fixed ratio of > files to transactions, guys) and which showed the kinds of performance > problems we'd seen with real live applications and the latest revisions of > the commercial NFS clients which, thanks to introduction of a few more > internal abstraction layers, were considerably slower than their > predecessors. > > > The benchmark is seriously flawed. > > <laughter> To paraphrase a hundred posts in freebsd-security: don't complain > unless you're willing to write the code that fixes the problem, or at least > suggest implementable solutions to the author. I'm sure Jeff will be more > than happy to revise the benchmark if time permits, and I'll be forwarding > the posts to him so he's got some impetus. :) -------------------------------------------------------------------- I tried an internal modem, newton@atdot.dotat.org but it hurt when I walked. Mark Newton ----- Voice: +61-4-1620-2223 ------------- Fax: +61-8-82231777 ----- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199909201029.TAA48175>