Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 25 Oct 2014 14:46:46 -0700
From:      Garrett Cooper <yaneurabeya@gmail.com>
To:        "K. Macy" <kmacy@freebsd.org>
Cc:        "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org>, FreeBSD stable <freebsd-stable@freebsd.org>, "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org>
Subject:   Re: Automatically running /usr/tests on stable/10 branch under Jenkins
Message-ID:  <74FE3F75-43D2-4CFC-8B0F-56EF886F4748@gmail.com>
In-Reply-To: <CAHM0Q_O_mr3X7bAZLEDJ78VgBv5yb8Gh2OwiXbiM3-00wYX46g@mail.gmail.com>
References:  <CAG=rPVe-hCYiH5YuC%2BrzrucJbHJvEFmik0RAA%2Brq%2BXQ5K_A0Ww@mail.gmail.com> <20141024053636.GH11222@dft-labs.eu> <CAHM0Q_MOLoYGVhVOwAHfxKmMdX8bBK0Y=OoiR0TR=t3kQyYtVQ@mail.gmail.com> <CAG=rPVcRkCtwjNdzO2p6PuMVTLTFh7qKN=pxPVDrE0DM=R_a9w@mail.gmail.com> <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> <544B46BA.4000008@freebsd.org> <FBD107B5-519F-4BB5-80DD-050E9FE22ABC@gmail.com> <CAHM0Q_O_mr3X7bAZLEDJ78VgBv5yb8Gh2OwiXbiM3-00wYX46g@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help

--Apple-Mail=_B288C26C-0C84-4CCF-A5EC-92AEB18549DC
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=windows-1252

On Oct 25, 2014, at 13:20, K. Macy <kmacy@freebsd.org> wrote:

>>>> Alan also suggested against integrating the test suite as-is, =
because as he said, "Remember, don't run these tests on a production =
system.  They WILL cause panics and deadlocks, and they may cause data =
loss too.=94
>>>>=20
>>>> Cheers,
>>>> -Garrett
>>>=20
>>> Wait, we want to sweep those bugs under the rug?  What exactly is =
wrong with making a test harness that can very easily reproduce a known =
problem?  The chances are that anyone will dive into it once the bug is =
easily reproducible.
>>=20
>>        Sweeping bugs under the rug is not what I plan on doing; I=92m =
marking these as expected failures, as opposed to having them =
continually panic a machine. Once a ZFS dev takes a look at the issue =
and resolves them, then the ZFS dev can remove the =93bail=94 calls I=92m =
adding to the testcases.
>=20
> Yes, disabling tests that fail leads to an ineffectual test suite. A
> test suite that never has any failures is not very useful. However,
> there are two factors to take in to account in this context:
> a) frequent failures can lead users to stop running a test suite
> leading to further regressions
> b) long-term repeated failures can desensitize users leading them to
> ignore *new* failures facilitating further regressions
>=20
> Thus it's really a question of what context you're talking about
> running the test suite in. For purposes of Jenkins we want full
> visibility in to what is passing and what is failing and how long this
> has been going on for.

(seeing as how my other post isn=92t in the -testing archives yet..)
	Panicking a node (what the tests are doing before last night) =
and exiting with a non-zero exit code (what I=92m making them do with =
the bail outs in tools/regression/zfs) are both considered test =
failures. The difference being that I can safely run all of the tests on =
a production or a test machine without having to panic/reboot the box =
and I get greater coverage in one fell swoop. If a developer wants they =
can always delete the lines that bail out of the tests to get the =
desired panic.
Cheers,
-Garrett

--Apple-Mail=_B288C26C-0C84-4CCF-A5EC-92AEB18549DC
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP using GPGMail

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iQEcBAEBCgAGBQJUTBpGAAoJEMZr5QU6S73eroQH/0EUjiiTQEnkaDgG29GxsSP4
pmerYl9PXlSJiN1t0ApDz7HxxWh0qTsV8+5OIdrWOU4gvmK3DEaq+HOMqVboVtS5
WLFy+/bljFhnEbF5b3UqPUyNFNaCKV7Rdr96oLnA9NVJRvEo6aNHF0+rZKyTDeMd
Qb7YDscluRqyqFCugb3rcatFWVHLycQKZdiUSx5Mdc7PnXiJ4nK9nK9nq6gU3lAA
u9F3wKFZLHuzI9Ko2MEUp9jqwoK8kL2wx3Q5YBvyCUUI7OmoIf2GNhhxOZwa4Tga
lfrTGH4gOcRLzjGmOGCOqXLAG3j1JpPetzJywdYjfZIVu+iEp+FmX4vH6WjF8nw=
=a/Nc
-----END PGP SIGNATURE-----

--Apple-Mail=_B288C26C-0C84-4CCF-A5EC-92AEB18549DC--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?74FE3F75-43D2-4CFC-8B0F-56EF886F4748>