Date: Sat, 25 Oct 2014 13:46:05 -0700 From: Matt Olander <matt@ixsystems.com> To: "K. Macy" <kmacy@freebsd.org> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org>, FreeBSD stable <freebsd-stable@freebsd.org>, "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org>, Garrett Cooper <yaneurabeya@gmail.com> Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins Message-ID: <CAK6u07Xovg_RqsFB%2B-3OsFDjcLQU=pD-bvv%2BokLOd_z=Q83zjg@mail.gmail.com> In-Reply-To: <CAHM0Q_O_mr3X7bAZLEDJ78VgBv5yb8Gh2OwiXbiM3-00wYX46g@mail.gmail.com> References: <CAG=rPVe-hCYiH5YuC%2BrzrucJbHJvEFmik0RAA%2Brq%2BXQ5K_A0Ww@mail.gmail.com> <20141024053636.GH11222@dft-labs.eu> <CAHM0Q_MOLoYGVhVOwAHfxKmMdX8bBK0Y=OoiR0TR=t3kQyYtVQ@mail.gmail.com> <CAG=rPVcRkCtwjNdzO2p6PuMVTLTFh7qKN=pxPVDrE0DM=R_a9w@mail.gmail.com> <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> <544B46BA.4000008@freebsd.org> <FBD107B5-519F-4BB5-80DD-050E9FE22ABC@gmail.com> <CAHM0Q_O_mr3X7bAZLEDJ78VgBv5yb8Gh2OwiXbiM3-00wYX46g@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Oct 25, 2014 at 1:20 PM, K. Macy <kmacy@freebsd.org> wrote: >>>> Alan also suggested against integrating the test suite as-is, because = as he said, "Remember, don't run these tests on a production system. They = WILL cause panics and deadlocks, and they may cause data loss too.=E2=80=9D >>>> >>>> Cheers, >>>> -Garrett >>> >>> Wait, we want to sweep those bugs under the rug? What exactly is wrong= with making a test harness that can very easily reproduce a known problem?= The chances are that anyone will dive into it once the bug is easily repr= oducible. >> >> Sweeping bugs under the rug is not what I plan on doing; I=E2=80= =99m marking these as expected failures, as opposed to having them continua= lly panic a machine. Once a ZFS dev takes a look at the issue and resolves = them, then the ZFS dev can remove the =E2=80=9Cbail=E2=80=9D calls I=E2=80= =99m adding to the testcases. >> Cheers, >> -Garrett > > Yes, disabling tests that fail leads to an ineffectual test suite. A > test suite that never has any failures is not very useful. However, > there are two factors to take in to account in this context: > a) frequent failures can lead users to stop running a test suite > leading to further regressions > b) long-term repeated failures can desensitize users leading them to > ignore *new* failures facilitating further regressions > > Thus it's really a question of what context you're talking about > running the test suite in. For purposes of Jenkins we want full > visibility in to what is passing and what is failing and how long this > has been going on for. Agreed. We're talking about doing an OpenZFS bug-tracker and maybe we could have tests post there automatically, once the bug is opened. I am going to put this on the board at MeetBSD Ca. next week, so we can have a discussion about it in real-time with some of the people present. Cheers, -matt
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAK6u07Xovg_RqsFB%2B-3OsFDjcLQU=pD-bvv%2BokLOd_z=Q83zjg>