Date: Thu, 19 Jun 2008 19:06:36 +0200 From: Kris Kennaway <kris@FreeBSD.org> To: Wesley Shields <wxs@FreeBSD.org> Cc: pfgshield-freebsd@yahoo.com, freebsd-ports@freebsd.org Subject: Re: regression-test Message-ID: <485A921C.80602@FreeBSD.org> In-Reply-To: <20080619163501.GB12160@atarininja.org> References: <676047.76296.qm@web32706.mail.mud.yahoo.com> <20080619163501.GB12160@atarininja.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Wesley Shields wrote: > On Thu, Jun 19, 2008 at 08:54:17AM -0700, pfgshield-freebsd@yahoo.com wrote: >> Christian Weisgerber wrote: >> ... >>> In particular, running tests of what duration would be considered >>> appropriate? E.g., building audio/flac takes a minute here, running >>> the tests sixteen, which I suspect would be excessive. >> FWIW, I started adding those regression-test targets to many ports >> because they are extremely useful: while trying to update to the >> latest coreutils I found everything apparently built but everything >> was actually broken. Now I am finding bugs in some ports that I >> thought were fine. > > Which is why I would recommend wrapping them in checks for > MAINTAINER_MODE or something similar. This way you, as the maintainer, > can turn them on when you want but they don't affect anything else. Our experience has shown that it is very common for tests to succeed when run by the maintainer, but to fail in the package cluster environment because of differences in the environment, configuration, default assumptions, etc. i.e. the regression tests found bugs, which is the entire point. We *want* regression tests to be available unconditionally -- that is why I added the hook to the package build process to run them if present. Please don't hack around this because you think you're doing us a favour :-) Kris
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?485A921C.80602>