From owner-freebsd-virtualization@FreeBSD.ORG Sat Oct 25 20:20:07 2014 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7E16FD6B; Sat, 25 Oct 2014 20:20:07 +0000 (UTC) Received: from mail-yh0-x236.google.com (mail-yh0-x236.google.com [IPv6:2607:f8b0:4002:c01::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 104C3366; Sat, 25 Oct 2014 20:20:07 +0000 (UTC) Received: by mail-yh0-f54.google.com with SMTP id 29so3165889yhl.13 for ; Sat, 25 Oct 2014 13:20:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=tynCnMKKC0eHZKShqxl5XFhNw74YLNxljWlRLqIOxA4=; b=Pub3QTq2WHLOKa8SdiQLMVF4SbTO2qnYUbR4XG+2yQllUwJPdQbUCWpRbuFy8nd9s8 r8pwHdCwy4EzGeMTtXHkwS7L9TqxbdbXcRNskwWWpPzDxCgil7raUIrZvitMCPnBJphI Y1aY/Zi3+hdvH1rNCXGNYqff8v+MHVALWchY3YxImcLw/VjXavNp1PD56P1vXrpdIR/Z fNNMe+v+M9AwqgBmfgQ9X1246rzuf5eDr+tEAGC/wTNiDB3kXn+PYKvfnmE6ht4LgoqI R6UiXjiXhNkLA2IPvRR1Lih9V4YEYtUgdDH/EuOz9rXOeEbbIGkRh4Xq4xXKIVvCB19C ksfw== MIME-Version: 1.0 X-Received: by 10.170.113.214 with SMTP id f205mr15326025ykb.10.1414268405509; Sat, 25 Oct 2014 13:20:05 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.170.82.197 with HTTP; Sat, 25 Oct 2014 13:20:05 -0700 (PDT) In-Reply-To: References: <20141024053636.GH11222@dft-labs.eu> <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> <544B46BA.4000008@freebsd.org> Date: Sat, 25 Oct 2014 13:20:05 -0700 X-Google-Sender-Auth: Tz1pjYcN82aDbop02VCfh_GmGLU Message-ID: Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins From: "K. Macy" To: Garrett Cooper Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-testing@freebsd.org" , FreeBSD stable , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Oct 2014 20:20:07 -0000 >>> Alan also suggested against integrating the test suite as-is, because a= s he said, "Remember, don't run these tests on a production system. They W= ILL cause panics and deadlocks, and they may cause data loss too.=E2=80=9D >>> >>> Cheers, >>> -Garrett >> >> Wait, we want to sweep those bugs under the rug? What exactly is wrong = with making a test harness that can very easily reproduce a known problem? = The chances are that anyone will dive into it once the bug is easily repro= ducible. > > Sweeping bugs under the rug is not what I plan on doing; I=E2=80= =99m marking these as expected failures, as opposed to having them continua= lly panic a machine. Once a ZFS dev takes a look at the issue and resolves = them, then the ZFS dev can remove the =E2=80=9Cbail=E2=80=9D calls I=E2=80= =99m adding to the testcases. > Cheers, > -Garrett Yes, disabling tests that fail leads to an ineffectual test suite. A test suite that never has any failures is not very useful. However, there are two factors to take in to account in this context: a) frequent failures can lead users to stop running a test suite leading to further regressions b) long-term repeated failures can desensitize users leading them to ignore *new* failures facilitating further regressions Thus it's really a question of what context you're talking about running the test suite in. For purposes of Jenkins we want full visibility in to what is passing and what is failing and how long this has been going on for. Cheers. -K