From owner-freebsd-ports@FreeBSD.ORG  Thu Jun 19 17:06:37 2008
Return-Path: <owner-freebsd-ports@FreeBSD.ORG>
Delivered-To: freebsd-ports@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 6676B106567B;
	Thu, 19 Jun 2008 17:06:37 +0000 (UTC)
	(envelope-from kris@FreeBSD.org)
Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28])
	by mx1.freebsd.org (Postfix) with ESMTP id 36C608FC26;
	Thu, 19 Jun 2008 17:06:36 +0000 (UTC)
	(envelope-from kris@FreeBSD.org)
Message-ID: <485A921C.80602@FreeBSD.org>
Date: Thu, 19 Jun 2008 19:06:36 +0200
From: Kris Kennaway <kris@FreeBSD.org>
User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421)
MIME-Version: 1.0
To: Wesley Shields <wxs@FreeBSD.org>
References: <676047.76296.qm@web32706.mail.mud.yahoo.com>
	<20080619163501.GB12160@atarininja.org>
In-Reply-To: <20080619163501.GB12160@atarininja.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: pfgshield-freebsd@yahoo.com, freebsd-ports@freebsd.org
Subject: Re: regression-test
X-BeenThere: freebsd-ports@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Porting software to FreeBSD <freebsd-ports.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-ports>,
	<mailto:freebsd-ports-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-ports>
List-Post: <mailto:freebsd-ports@freebsd.org>
List-Help: <mailto:freebsd-ports-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-ports>,
	<mailto:freebsd-ports-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 19 Jun 2008 17:06:37 -0000

Wesley Shields wrote:
> On Thu, Jun 19, 2008 at 08:54:17AM -0700, pfgshield-freebsd@yahoo.com wrote:
>> Christian Weisgerber wrote:
>> ...
>>> In particular, running tests of what duration would be considered
>>> appropriate?  E.g., building audio/flac takes a minute here, running
>>> the tests sixteen, which I suspect would be excessive.
>> FWIW, I started adding those regression-test targets to many ports
>> because they are extremely useful: while trying to update to the
>> latest coreutils I found everything apparently built but everything
>> was actually broken. Now I am finding bugs in some ports that I
>> thought were fine.
> 
> Which is why I would recommend wrapping them in checks for
> MAINTAINER_MODE or something similar.  This way you, as the maintainer,
> can turn them on when you want but they don't affect anything else.

Our experience has shown that it is very common for tests to succeed 
when run by the maintainer, but to fail in the package cluster 
environment because of differences in the environment, configuration, 
default assumptions, etc.  i.e. the regression tests found bugs, which 
is the entire point.

We *want* regression tests to be available unconditionally -- that is 
why I added the hook to the package build process to run them if 
present.  Please don't hack around this because you think you're doing 
us a favour :-)

Kris