From owner-freebsd-stable Fri Oct 18 0:46:21 2002 Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C003E37B401 for ; Fri, 18 Oct 2002 00:46:19 -0700 (PDT) Received: from topperwein.dyndns.org (acs-24-154-51-184.zoominternet.net [24.154.51.184]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4D48343E42 for ; Fri, 18 Oct 2002 00:45:18 -0700 (PDT) (envelope-from behanna@zbzoom.net) Received: from [192.168.168.10] ([192.168.168.10]) by topperwein.dyndns.org (8.12.6/8.12.5) with ESMTP id g9I7jI3x063885 for ; Fri, 18 Oct 2002 03:45:18 -0400 (EDT) (envelope-from behanna@zbzoom.net) Date: Fri, 18 Oct 2002 03:45:13 -0400 (EDT) From: Chris BeHanna Reply-To: Chris BeHanna To: FreeBSD-Stable Subject: Re: freebsd test matrix In-Reply-To: <20021017234320.GA1664@panix.com> Message-ID: <20021018034039.F1275-100000@topperwein.dyndns.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-stable@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Thu, 17 Oct 2002, David Kleiner wrote: > On Thu, Oct 17, 2002 at 09:04:54AM +0200, Fischer, Oliver wrote: > > Chris BeHanna wrote: > > > > > > I'm currently doing QA on a (very) large software project, and all > > >of those things are important. Some of the testing uses existing > > >industry test suites and benchmarking tools, and some of it (much of > > >it) is custom. Being able to compile, install, and boot is just the > > >tip of the iceberg. > > > > I agree, it is only the top of it. I have an idea how to do this for a > > lot of programms, but no one how to do it with the more system specific. > > Does anyone know how commercial os vendors do it? > > As something I am involved with (not to mention any names) - and I > don't know if any of it has been looked at in the open-source U*x > world: > > nightly build cycles with basic (short stress, basic api sanity, > some standards compliance) > > bi-weekly - more involved api, abi, stress, standards > > beta-candidates, release candidates and so on - even more > longer-term test suites. > > As we, most likely, cannot use anything out of the Commercial World, > all this has to be built from close to scratch. I may be wrong here > but it is a likelier scenario. > > Ideas, anyone? There are some standardized test suites that can be used (e.g., POSIX compliance, SFS, SVVS, gABI). Some test standards-compliance, while others stress the system (SFS, for example, is the gold standard for testing filesystem performance). The thing that bites you in the butt is test harnesses. Invariably, the commonly-available (free) harnesses will lack features desired by the QA team; therefore, they end up rolling their own. In a volunteer project like this, that's a big, but necessary, first step to take. Ideally, each developer will have a set of unit tests for what (s)he is working on, that could, with some massaging, be plugged into a larger harness to do nightly regression testing (perhaps via tinderbox). The trouble is keeping the tests up-to-date. Invariably, what the tests are testing ends up changing underneath them, leading to spurious failures in the test report. -- Chris BeHanna http://www.pennasoft.com Principal Consultant PennaSoft Corporation chris@pennasoft.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message