Date: Wed, 11 Apr 2001 03:12:57 -0400 (EDT) From: Robert Watson <rwatson@FreeBSD.ORG> To: Peter Jeremy <peter.jeremy@alcatel.com.au> Cc: freebsd-arch@FreeBSD.ORG Subject: Re: Features to facilitate correctness and regression testing Message-ID: <Pine.NEB.3.96L.1010411030800.84384B-100000@fledge.watson.org> In-Reply-To: <20010411161553.U66243@gsmx07.alcatel.com.au>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, 11 Apr 2001, Peter Jeremy wrote: > > It has been > > suggested to me that these would make a great addition to a new > > regression/ CVS sub-tree, that they should be included under > > appropriate existing parts of the tree reflecting what they test, > > and that they would be best in the form of a port. > > I'd prefer to see any test code in the tree near the code being tested > - this (marginally) improves the chances of tests being updated to > cover new functionality. A number of parts of the system already > include stand-alone self-checks which could probably be invoked as > part of an overall regression test. Pushing the tests into a > separate port makes it more difficult for developers to use the > tests and would appear to make it harder to update the tests. > > Any generic test skeleton files probably belong in a "regression" > tree under /usr/src. Where do you think that kernel regression tests should be placed? Often, these tests will be initiated and managed from userland, so they are probably inappropriate to drop in sys/? > Since you are talking about changing the security-related behaviour of > the system, I'd prefer it to require an "options REGRESSION" (and maybe > then set a sysctl), rather than be a KLD. I'm not at all keen on the > idea of having it stashed away as a separately maintained port. One of my primary motivations for looking at importing the features into the base kernel was to avoid both the non-use, bitrot and synchronization problems currently inherent to ports. Ports with kld's suffer breakage in a number of ways, due to lack of synchronization with the base tree, difficulty in maintaining up-to-date kld builds when the system is a moving target, etc. (For example, I rebuild my workstation almost daily on -CURRENT -- and I have to rebuild my vmware kld's about as frequently or risk panics). For many consumers sitting on a release, this may be less of a problem, but for a development tool on the -CURRENT branch, it's a serious problem. Does this mean we need a <sys/regression.h> which provides constants and prototypes for regression-specific interfaces which are enabled using options REGRESSION? (This is how I have it implemented in my local source tree, but I'm interested in whether this seems like a good idea to anyone else :-). > > Is the idea of helper functions for regression testing simply > > philosophically wrong, or does it serve a useful function > > I think it's a valid part of white-box testing. In any piece of > software, there are going to be code paths that are very difficult or > impossible to exercise using the `normal' interfaces. Providing > interfaces that are intended solely for testing the correctness of the > code seems perfectly reasonable. This is no different to the JTAG ports > on virtually every modern LSI chip. Sounds good to me -- I think we agree on almost all points. Robert N M Watson FreeBSD Core Team, TrustedBSD Project robert@fledge.watson.org NAI Labs, Safeport Network Services To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.NEB.3.96L.1010411030800.84384B-100000>