Date: Wed, 12 Aug 2009 13:29:35 +0100 (BST) From: Robert Watson <rwatson@FreeBSD.org> To: current@FreeBSD.org Subject: Call for regression and performance testing - 8.0 Message-ID: <alpine.BSF.2.00.0908121329130.66335@fledge.watson.org>
next in thread | raw e-mail | index | archive | help
Dear all: As we're approaching 8.0 BETA3, now would be a really good time to start identifying functional and performance regressions from the 7.x series. We've done a mixed job at this in the past, but when it comes to "I'll do better some day", there's no time like the present :-). Some notes: Functional testing We actually have a sizable regression test suite in src/tools/regression -- some of these tools will have broken as a result of 8-CURRENT development, so the first task may be to fix them. The next is to run them on 7-STABLE and 8-CURRENT and decide if things have gotten worse -- or maybe we have a bug to fix in both. Pick the tool of your choice, and give it a spin. More than one person per tool is fine, because that way we get more diverse testing. Performance testing On the performance front, life is always a bit more tricky -- performance testing is a subtle art. However, the most clear lessons are that (a) testing with diverse workloads and diverse environments is extremely important, (b) you should do multiple runs and use ministat(1) to analyze results, and (c) that you want to compare apples with apples -- use the same hardware/configuration wherever possible. Watch out for annoying nits such as partition layout affecting I/O throughput for two different installs on the same disk. Pick something you think is important to you: bytes/sec over TCP on loopback, web hits/sec, NFS ops/sec, disk I/O transactions/sec, and do some comparison between 7.2 and 8.0. If you find improvement -- great! If you find a regression, please start a thread on current@ to help get it diagnosed. And if you want help doing performance measurement for a particular workload that isn't well studied, send some e-mail to performance@ to ask for advice on how best to measure it. Thanks, Robert N M Watson Computer Laboratory University of Cambridge
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.2.00.0908121329130.66335>