Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 25 Oct 2014 14:55:28 -0700
From:      "K. Macy" <kmacy@freebsd.org>
To:        Garrett Cooper <yaneurabeya@gmail.com>
Cc:        "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org>, Alfred Perlstein <alfred@freebsd.org>, FreeBSD stable <freebsd-stable@freebsd.org>, "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org>
Subject:   Re: Automatically running /usr/tests on stable/10 branch under Jenkins
Message-ID:  <CAHM0Q_PwAiUTZirAuB7Q4NiY=H3YOJ522qcxoQ7gNvtZ9Lxk0g@mail.gmail.com>
In-Reply-To: <74FE3F75-43D2-4CFC-8B0F-56EF886F4748@gmail.com>
References:  <CAG=rPVe-hCYiH5YuC%2BrzrucJbHJvEFmik0RAA%2Brq%2BXQ5K_A0Ww@mail.gmail.com> <20141024053636.GH11222@dft-labs.eu> <CAHM0Q_MOLoYGVhVOwAHfxKmMdX8bBK0Y=OoiR0TR=t3kQyYtVQ@mail.gmail.com> <CAG=rPVcRkCtwjNdzO2p6PuMVTLTFh7qKN=pxPVDrE0DM=R_a9w@mail.gmail.com> <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> <544B46BA.4000008@freebsd.org> <FBD107B5-519F-4BB5-80DD-050E9FE22ABC@gmail.com> <CAHM0Q_O_mr3X7bAZLEDJ78VgBv5yb8Gh2OwiXbiM3-00wYX46g@mail.gmail.com> <74FE3F75-43D2-4CFC-8B0F-56EF886F4748@gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Oct 25, 2014 at 2:46 PM, Garrett Cooper <yaneurabeya@gmail.com> wro=
te:
> On Oct 25, 2014, at 13:20, K. Macy <kmacy@freebsd.org> wrote:
>
>>>>> Alan also suggested against integrating the test suite as-is, because=
 as he said, "Remember, don't run these tests on a production system.  They=
 WILL cause panics and deadlocks, and they may cause data loss too.=E2=80=
=9D
>>>>>
>>>>> Cheers,
>>>>> -Garrett
>>>>
>>>> Wait, we want to sweep those bugs under the rug?  What exactly is wron=
g with making a test harness that can very easily reproduce a known problem=
?  The chances are that anyone will dive into it once the bug is easily rep=
roducible.
>>>
>>>        Sweeping bugs under the rug is not what I plan on doing; I=E2=80=
=99m marking these as expected failures, as opposed to having them continua=
lly panic a machine. Once a ZFS dev takes a look at the issue and resolves =
them, then the ZFS dev can remove the =E2=80=9Cbail=E2=80=9D calls I=E2=80=
=99m adding to the testcases.
>>
>> Yes, disabling tests that fail leads to an ineffectual test suite. A
>> test suite that never has any failures is not very useful. However,
>> there are two factors to take in to account in this context:
>> a) frequent failures can lead users to stop running a test suite
>> leading to further regressions
>> b) long-term repeated failures can desensitize users leading them to
>> ignore *new* failures facilitating further regressions
>>
>> Thus it's really a question of what context you're talking about
>> running the test suite in. For purposes of Jenkins we want full
>> visibility in to what is passing and what is failing and how long this
>> has been going on for.
>
> (seeing as how my other post isn=E2=80=99t in the -testing archives yet..=
)
>         Panicking a node (what the tests are doing before last night) and=
 exiting with a non-zero exit code (what I=E2=80=99m making them do with th=
e bail outs in tools/regression/zfs) are both considered test failures. The=
 difference being that I can safely run all of the tests on a production or=
 a test machine without having to panic/reboot the box and I get greater co=
verage in one fell swoop. If a developer wants they can always delete the l=
ines that bail out of the tests to get the desired panic.

I don't think there is a RIGHT answer. I just ask that it be noisy
about the choice that's being made. In other words, it should be very
explicit that it's running a "safe" subset of the tests because
FreeBSD's ZFS can't actually pass all of them, and that there be a
flag to enable those of us who want to see the failure to see it
without modifying any scripts or config files. A test framework that
requires me to muck with settings to run failing tests isn't living up
to its full potential.


-K



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAHM0Q_PwAiUTZirAuB7Q4NiY=H3YOJ522qcxoQ7gNvtZ9Lxk0g>