Date: Wed, 22 Dec 2010 10:08:28 +0000 From: "Robert N. M. Watson" <rwatson@FreeBSD.org> To: Erik Cederstrand <erik@cederstrand.dk> Cc: mdf@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Schedule for releases Message-ID: <96AE5824-D19C-4CD2-9DC1-98D7D314CE1A@FreeBSD.org> In-Reply-To: <DB4D8AC7-25D6-4901-BBF9-77BEB956840B@cederstrand.dk> References: <AANLkTi=_mHDz3LZ1SAuCsz6kmvqCdZBx3Q5ZTyQQO1%2BP@mail.gmail.com> <201012211500.16131.jhb@freebsd.org> <alpine.BSF.2.00.1012212215320.36028@fledge.watson.org> <DB4D8AC7-25D6-4901-BBF9-77BEB956840B@cederstrand.dk>
next in thread | previous in thread | raw e-mail | index | archive | help
On 22 Dec 2010, at 07:21, Erik Cederstrand wrote: > Den 21/12/2010 kl. 23.28 skrev Robert Watson: >=20 >> Looking at 7.x, I'm struck by how much it has slowed down. There's a = significant user community, but not a significant developer community.=20= >=20 > Which pretty much sums up a dilemma in the development of FreeBSD, I = think. Developers want users to try out their new shiny stuff, but users = don't want to spend time upgrading. >=20 > I think one of many things that would be great to do is to improve the = usability and coverage of the regression tests. This would take at least = some of the burden off developers who want to MFC their work. We already = have the tinderboxes, Coverity and Clang Static Analyzer, but apart from = pho's stress tests we don't have any automated runtime testing (as far = as I know). We have an increasing quantity of moderately decent regression tests in = our tools/regression tree, but they are only run selectively and = moderately rarely. A very nice contribution someone could make would be: (1) Make it so a recursive regression run is easily accessible -- "make = tests ; make runtests" -- in the root or similar. (2) Set up a "test tinderbox", perhaps using a hosted server at Sentex, = and perhaps using virtual machines so that many versions can be run. (3) Arrange a report website and report e-mails identifying the = introduction of regressions in branches. One known problem with our regression suite is that it includes tests = that crash certain versions (and ideally not later versions), so we'd = need something flexible enough to skip tests known to fail on particular = branches in some usefully configurable way. All this would be extremely useful -- we can then start pulling in more = regression tests written by other communities (other *BSD, etc), and = push people to produce more tests when they fix bugs. It would be = especially nice to run our security regression tests regularly (testing = permissions, ACL semantics, inter-process protections, etc). Robert=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?96AE5824-D19C-4CD2-9DC1-98D7D314CE1A>