From owner-freebsd-arch@FreeBSD.ORG Fri Jan 18 01:36:58 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C072216A417; Fri, 18 Jan 2008 01:36:58 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 9EE6213C4DD; Fri, 18 Jan 2008 01:36:58 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id 42D581A4D8C; Thu, 17 Jan 2008 17:33:31 -0800 (PST) Date: Thu, 17 Jan 2008 17:33:31 -0800 From: Alfred Perlstein To: John Baldwin Message-ID: <20080118013331.GP99258@elvis.mu.org> References: <200801171731.09873.jhb@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200801171731.09873.jhb@freebsd.org> User-Agent: Mutt/1.4.2.3i Cc: arch@freebsd.org Subject: Re: Simple framework for regression tests in src/tools/regression X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Jan 2008 01:36:58 -0000 * John Baldwin [080117 14:28] wrote: > When I converted the test program for the posix shm stuff over to a regression > test I added some framework bits to manage the output format prove(1) > expects. You can find it in src/tools/regression/posixshm/test.[ch]. I was > curious if we wanted to perhaps make it a library in the base system > (libbsdtest.a or some such) that our regression tools could use. > > The basic organization of the current test.[ch] is that you have a linker set > of tests which it uses to number the tests and determine the test count. The > framework provides routines for the test to indicate its result like so: > > pass() - indicates test passed > fail() - indicates test failed with no error message > fail_err(fmt, ...) - failure with printf(3) error message > fail_errno(fmt, ...) - failure with error message followed by strerror(3) > output similar to err(3) > skip(const char *) - test skipped with reason for skip > todo(const char *) - test marked as TODO > > If a given test doesn't call any of the status indicators above, then the main > test loop will fail the test with an error of "unknown status". No effort is > made to handle multiple status indications for a given test. I assume > prove(1) would handle complaining about such anomalies in the test output. > > Tests consist of a void foo(void) function along with a description and each > test is created via the TEST() macro like so: > > void > mytest(void) > { > struct passwd *pwd; > > pwd = getpwuid(getuid()); > if (strcmp(pwd->pw_name, "jhb") == 0) { > fail_err("great evil detected"); > return; > } > pass(); > } > TEST(mytest, "check for evil users"); > > To invoke the tests, one executes the 'run_tests()' method provided by the > framework. It outputs the header expected by prove(1) and then executes each > test in succession. I specifically didn't put main() in the framework in > case a test utility wants to take optional arguments, etc. A simple test > utility (like posixshm.c) just invokes run_tests() in its main(). > > One design decision that I'm sure will be debated is that I went with the > functions to indicate results (fail, etc.) rather than having the test method > return a value because it becomes more complicated to handle error messages > otherwise. If this were C++ the tests could return a std::pair<> of a status > code and some sort of object with a toString() method or something (or if it > was python I would return a tuple). However, this is C. :) It's also nicer > to be able to just pass the va args to vprintf() directly w/o having to print > them into strings and print them out later, etc. > > Anyways, comments, etc.? There is C++ in the base system. You can return a pair by returning a structure which is supported in C. -Alfred