Date: Sat, 25 Oct 2014 13:20:05 -0700 From: "K. Macy" <kmacy@freebsd.org> To: Garrett Cooper <yaneurabeya@gmail.com> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org>, FreeBSD stable <freebsd-stable@freebsd.org>, "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org> Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins Message-ID: <CAHM0Q_O_mr3X7bAZLEDJ78VgBv5yb8Gh2OwiXbiM3-00wYX46g@mail.gmail.com> In-Reply-To: <FBD107B5-519F-4BB5-80DD-050E9FE22ABC@gmail.com> References: <CAG=rPVe-hCYiH5YuC%2BrzrucJbHJvEFmik0RAA%2Brq%2BXQ5K_A0Ww@mail.gmail.com> <20141024053636.GH11222@dft-labs.eu> <CAHM0Q_MOLoYGVhVOwAHfxKmMdX8bBK0Y=OoiR0TR=t3kQyYtVQ@mail.gmail.com> <CAG=rPVcRkCtwjNdzO2p6PuMVTLTFh7qKN=pxPVDrE0DM=R_a9w@mail.gmail.com> <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> <544B46BA.4000008@freebsd.org> <FBD107B5-519F-4BB5-80DD-050E9FE22ABC@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
>>> Alan also suggested against integrating the test suite as-is, because a= s he said, "Remember, don't run these tests on a production system. They W= ILL cause panics and deadlocks, and they may cause data loss too.=E2=80=9D >>> >>> Cheers, >>> -Garrett >> >> Wait, we want to sweep those bugs under the rug? What exactly is wrong = with making a test harness that can very easily reproduce a known problem? = The chances are that anyone will dive into it once the bug is easily repro= ducible. > > Sweeping bugs under the rug is not what I plan on doing; I=E2=80= =99m marking these as expected failures, as opposed to having them continua= lly panic a machine. Once a ZFS dev takes a look at the issue and resolves = them, then the ZFS dev can remove the =E2=80=9Cbail=E2=80=9D calls I=E2=80= =99m adding to the testcases. > Cheers, > -Garrett Yes, disabling tests that fail leads to an ineffectual test suite. A test suite that never has any failures is not very useful. However, there are two factors to take in to account in this context: a) frequent failures can lead users to stop running a test suite leading to further regressions b) long-term repeated failures can desensitize users leading them to ignore *new* failures facilitating further regressions Thus it's really a question of what context you're talking about running the test suite in. For purposes of Jenkins we want full visibility in to what is passing and what is failing and how long this has been going on for. Cheers. -K
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAHM0Q_O_mr3X7bAZLEDJ78VgBv5yb8Gh2OwiXbiM3-00wYX46g>