Date: Thu, 17 Jan 2008 18:40:25 -0500 From: David Schultz <das@FreeBSD.ORG> To: John Baldwin <jhb@FreeBSD.ORG> Cc: arch@FreeBSD.ORG Subject: Re: Simple framework for regression tests in src/tools/regression Message-ID: <20080117234025.GA86029@VARK.MIT.EDU> In-Reply-To: <200801171731.09873.jhb@freebsd.org> References: <200801171731.09873.jhb@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Jan 17, 2008, John Baldwin wrote:
> When I converted the test program for the posix shm stuff over to a regression
> test I added some framework bits to manage the output format prove(1)
> expects. You can find it in src/tools/regression/posixshm/test.[ch]. I was
> curious if we wanted to perhaps make it a library in the base system
> (libbsdtest.a or some such) that our regression tools could use.
>
> The basic organization of the current test.[ch] is that you have a linker set
> of tests which it uses to number the tests and determine the test count. The
> framework provides routines for the test to indicate its result like so:
>
> pass() - indicates test passed
> fail() - indicates test failed with no error message
> fail_err(fmt, ...) - failure with printf(3) error message
> fail_errno(fmt, ...) - failure with error message followed by strerror(3)
> output similar to err(3)
> skip(const char *) - test skipped with reason for skip
> todo(const char *) - test marked as TODO
I think it would be great if some day we had an easy way to set up
tools/regression so that all the tests get run by tinderbox
automatically. I don't know if that has any impact on the design
of what you propose.
It would be nice if the library also provided an option (perhaps
via an env variable) to make it possible to choose between
a) printing a failure message and continuing
b) calling abort() to trap into the debugger
Also, add a drop-in replacement for assert() to the list
above. Some of my tests call assert() because that happened to be
convenient at the time, but that's not the right thing to do for
prove(1).
Also on my wish list:
- A test infrastructure where you can type 'make' on level of the
hierarchy and have it run all the tests under that directory;
prove -r isn't flexible enough. It should be possible to skip
tests that must be run as root and tests that only apply to
certain architectures[1]. It should be possible to skip
"long-running" tests, and it should be possible to specify some
sort of abstract complexity parameter to tests that can run for
arbitrary amounts of time. For example, I have a lot of libm
tests that could run all day, but you wouldn't want to run them
for more than a few seconds unless you had nothing better to do
with the machine.
- A good way to set up performance regression tests. This, of
course, requires that the tests be run repeatedly on the same
hardware. But on the library front, this means having a way to
output measurements to a file somewhere and compare to previous
measurements. (It might also mean convenience functions to
gather timing and memory usage data, and routines to output
platform configuration data automatically.) The Solaris kernel
group has something like this. As I recall, they ran it on the
same box nightly, and you could do things like generate a graph
of fork latency for the previous few years, with different lines
for different VCS branches. Not that we want to go that far, but
it's something to think about...
[1] A 'skip' library call might not be enough. Some tests might
not even compile if they are for a different architecture.
Another question is what if a test requires that a particular
port be installed before it will run. Currently we simply
don't allow dependencies like that, so for example, my libm
tests that use ports/math/mpfr simply aren't checked in.
But you could imagine being able to type
'make install-test-depends' and having all the right software
get installed automagically; otherwise, the test infrastructure
would notice that it is missing and refrain from attempting to
compile the test.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20080117234025.GA86029>