From owner-freebsd-testing@FreeBSD.ORG Sat Oct 25 16:49:50 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 30F1BC36; Sat, 25 Oct 2014 16:49:50 +0000 (UTC) Received: from mail-la0-x229.google.com (mail-la0-x229.google.com [IPv6:2a00:1450:4010:c03::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1F886CA9; Sat, 25 Oct 2014 16:49:48 +0000 (UTC) Received: by mail-la0-f41.google.com with SMTP id pn19so4268604lab.0 for ; Sat, 25 Oct 2014 09:49:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=cgm9lfEHWfB6qAKCJlmjuQQ3hPSKCH27q8M4kK3lgUg=; b=t/zm/tpixCC8iLm0cepTwA62bBcKH00ujOBaTZjgFYl0Bo3gLBEwSzKg/CrjCEIWGS GiogtLPx0yMwoo2c6DZct0U0cUATsYJRJfO57owSpHlO3UMj8kJP3GZl9qXgyxRfCamL UjorKSEMtacmJRLL2upi0sTRhku8LSGxeMws02Ke0b7ZHxjat8ujAHmPaPX6lSypOII0 R1cK0S1ssTesQBpX78IwggzDqUkfjAknXFHUTfjM8wXoUnQXXAQUNiLAgulq/n9hN7dA 3sVj+i/FHxqB9SPwJhRlQC2IQcG4ZNYW6RtBZjNIGzx3Rx7RO+kvUSPNCnNPe3yRNatW OsLg== MIME-Version: 1.0 X-Received: by 10.112.57.227 with SMTP id l3mr11901372lbq.68.1414255786825; Sat, 25 Oct 2014 09:49:46 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.112.84.197 with HTTP; Sat, 25 Oct 2014 09:49:46 -0700 (PDT) In-Reply-To: <544B46BA.4000008@freebsd.org> References: <20141024053636.GH11222@dft-labs.eu> <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> <544B46BA.4000008@freebsd.org> Date: Sat, 25 Oct 2014 09:49:46 -0700 X-Google-Sender-Auth: GIlVT8zHlo5sp3t_C8L_ce6W7_A Message-ID: Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins From: Craig Rodrigues To: Alfred Perlstein Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-testing@freebsd.org" , FreeBSD stable , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Oct 2014 16:49:50 -0000 On Fri, Oct 24, 2014 at 11:44 PM, Alfred Perlstein wrote: > > On 10/24/14 9:45 PM, Garrett Cooper wrote: > >> I think getting tools/regression/zfs working first would be a better idea >> (which means that ZFS developers will need to go debug/fix the issue noted >> in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191574 ). I'll go >> ahead and commit my fixes to head from my github fork so it runs. >> > If tools/regression/zfs has Kyuafiles that makes it easier to run under the kyua tool, that would greatly facilitate running under Jenkins automation, and would be useful. I notice some of the fixes you are applying to the regression/zfs tests is to make certain tests not run on FreeBSD because they cause known kernel panics such as this: https://lists.freebsd.org/pipermail/svn-src-all/2014-October/093671.html I'm not entirely convinced that this is a good way to "fix" a test. If a test causes a panic, then that's what it does, it it should not be swept under the rug, as Alfred has pointed out. Printing out a warning with a pointer to the PR just before running this type of test is OK, though. > >> Alan also suggested against integrating the test suite as-is, because as >> he said, "Remember, don't run these tests on a production system. They >> WILL cause panics and deadlocks, and they may cause data loss too." >> >> Cheers, >> -Garrett >> > > Wait, we want to sweep those bugs under the rug? What exactly is wrong > with making a test harness that can very easily reproduce a known problem? > The chances are that anyone will dive into it once the bug is easily > reproducible. > I agree with Alfred on this. Even though Alan's test suite may kernel panic or cause problems, there is still value in running it and making the results visible on jenkins.freebsd.org. Running these tests inside a VM which is generated during the build will allow these types of test to run, but still keep the test machine usable, even if the VM gets corrupted while running the tests. If we have test suites for ZFS, but no one runs them, then no one will bother to investigate and fix the bugs. Running the test suites under automation that is visible on jenkins.freebsd.org is going to force developers to see problems much sooner than they do now. Just having the tests in the tree and hoping that people run them and care to look into the problems is not enough. -- Craig