Date: Wed, 11 Apr 2001 16:15:53 +1000 From: Peter Jeremy <peter.jeremy@alcatel.com.au> To: Robert Watson <rwatson@FreeBSD.ORG> Cc: freebsd-arch@FreeBSD.ORG Subject: Re: Features to facilitate correctness and regression testing Message-ID: <20010411161553.U66243@gsmx07.alcatel.com.au> In-Reply-To: <Pine.NEB.3.96L.1010410230318.81804A-100000@fledge.watson.org>; from rwatson@FreeBSD.ORG on Tue, Apr 10, 2001 at 11:15:14PM -0400 References: <Pine.NEB.3.96L.1010410230318.81804A-100000@fledge.watson.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2001-Apr-10 23:15:14 -0400, Robert Watson <rwatson@FreeBSD.ORG> wrote: >1) To what extent are we interested in including correctness and > regression testing suites in the base development tree? I'd say it's an excellent idea. Currently our standard test is "make world", but that doesn't do a particularly good job of exercising the overall system. > It has been > suggested to me that these would make a great addition to a new > regression/ CVS sub-tree, that they should be included under > appropriate existing parts of the tree reflecting what they test, > and that they would be best in the form of a port. I'd prefer to see any test code in the tree near the code being tested - this (marginally) improves the chances of tests being updated to cover new functionality. A number of parts of the system already include stand-alone self-checks which could probably be invoked as part of an overall regression test. Pushing the tests into a separate port makes it more difficult for developers to use the tests and would appear to make it harder to update the tests. Any generic test skeleton files probably belong in a "regression" tree under /usr/src. > In any case, I'm > very interested in the idea of a "make regression" being easily > accessible to developers, to allow them to run fairly intensive > correctness tests after making changes. I think there would need to be lots of knobs. Things like the libgmp test suite take a long time to run and are probably overkill for checking that a change in a totally unrelated part of the system works. >2) Some helper functions may allow processes to modify system state in > such a way as to potentially violate normal limits on system behavior: ... > should these > features be included in the base system, then be optionally > compiled? Since you are talking about changing the security-related behaviour of the system, I'd prefer it to require an "options REGRESSION" (and maybe then set a sysctl), rather than be a KLD. I'm not at all keen on the idea of having it stashed away as a separately maintained port. > Is the idea of helper functions for regression testing simply > philosophically wrong, or does it serve a useful function I think it's a valid part of white-box testing. In any piece of software, there are going to be code paths that are very difficult or impossible to exercise using the `normal' interfaces. Providing interfaces that are intended solely for testing the correctness of the code seems perfectly reasonable. This is no different to the JTAG ports on virtually every modern LSI chip. Peter To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20010411161553.U66243>