Date: Thu, 17 Jan 2008 17:31:09 -0500 From: John Baldwin <jhb@freebsd.org> To: arch@freebsd.org Subject: Simple framework for regression tests in src/tools/regression Message-ID: <200801171731.09873.jhb@freebsd.org>
next in thread | raw e-mail | index | archive | help
When I converted the test program for the posix shm stuff over to a regression
test I added some framework bits to manage the output format prove(1)
expects. You can find it in src/tools/regression/posixshm/test.[ch]. I was
curious if we wanted to perhaps make it a library in the base system
(libbsdtest.a or some such) that our regression tools could use.
The basic organization of the current test.[ch] is that you have a linker set
of tests which it uses to number the tests and determine the test count. The
framework provides routines for the test to indicate its result like so:
pass() - indicates test passed
fail() - indicates test failed with no error message
fail_err(fmt, ...) - failure with printf(3) error message
fail_errno(fmt, ...) - failure with error message followed by strerror(3)
output similar to err(3)
skip(const char *) - test skipped with reason for skip
todo(const char *) - test marked as TODO
If a given test doesn't call any of the status indicators above, then the main
test loop will fail the test with an error of "unknown status". No effort is
made to handle multiple status indications for a given test. I assume
prove(1) would handle complaining about such anomalies in the test output.
Tests consist of a void foo(void) function along with a description and each
test is created via the TEST() macro like so:
void
mytest(void)
{
struct passwd *pwd;
pwd = getpwuid(getuid());
if (strcmp(pwd->pw_name, "jhb") == 0) {
fail_err("great evil detected");
return;
}
pass();
}
TEST(mytest, "check for evil users");
To invoke the tests, one executes the 'run_tests()' method provided by the
framework. It outputs the header expected by prove(1) and then executes each
test in succession. I specifically didn't put main() in the framework in
case a test utility wants to take optional arguments, etc. A simple test
utility (like posixshm.c) just invokes run_tests() in its main().
One design decision that I'm sure will be debated is that I went with the
functions to indicate results (fail, etc.) rather than having the test method
return a value because it becomes more complicated to handle error messages
otherwise. If this were C++ the tests could return a std::pair<> of a status
code and some sort of object with a toString() method or something (or if it
was python I would return a tuple). However, this is C. :) It's also nicer
to be able to just pass the va args to vprintf() directly w/o having to print
them into strings and print them out later, etc.
Anyways, comments, etc.?
--
John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200801171731.09873.jhb>