Date: Sat, 25 Oct 2014 11:59:59 -0700 From: Garrett Cooper <yaneurabeya@gmail.com> To: Craig Rodrigues <rodrigc@freebsd.org> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org>, Alfred Perlstein <alfred@freebsd.org> Subject: Panicking UUTs and integrating the ZFS test suite into the default run Message-ID: <86E89F83-179D-4301-9EC0-0ECC3909B924@gmail.com> In-Reply-To: <CAG=rPVexyG_ZW8%2BXPWUu6Dt7--ZjwKG7Wtga3esR0JqpJPJtPw@mail.gmail.com> References: <CAG=rPVe-hCYiH5YuC%2BrzrucJbHJvEFmik0RAA%2Brq%2BXQ5K_A0Ww@mail.gmail.com> <20141024053636.GH11222@dft-labs.eu> <CAHM0Q_MOLoYGVhVOwAHfxKmMdX8bBK0Y=OoiR0TR=t3kQyYtVQ@mail.gmail.com> <CAG=rPVcRkCtwjNdzO2p6PuMVTLTFh7qKN=pxPVDrE0DM=R_a9w@mail.gmail.com> <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> <544B46BA.4000008@freebsd.org> <CAG=rPVexyG_ZW8%2BXPWUu6Dt7--ZjwKG7Wtga3esR0JqpJPJtPw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_F769DEA7-56B3-46B3-BAFA-2F238869E838 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 (moving to -testing and renaming because this is turning into a bikeshed = discussion) On Oct 25, 2014, at 9:49, Craig Rodrigues <rodrigc@freebsd.org> wrote: > On Fri, Oct 24, 2014 at 11:44 PM, Alfred Perlstein = <alfred@freebsd.org> wrote: ... > If tools/regression/zfs has Kyuafiles that makes it easier to > run under the kyua tool, that would greatly facilitate running > under Jenkins automation, and would be useful. > I notice some of the fixes you are applying to the regression/zfs > tests is to make certain tests not run on FreeBSD because they > cause known kernel panics such as this: >=20 > = https://lists.freebsd.org/pipermail/svn-src-all/2014-October/093671.html Yes. > I'm not entirely convinced that this is a good way to "fix" a test. > If a test causes a panic, then that's what it does, it it should not = be > swept under the rug, as Alfred has pointed out. > Printing out a warning with a pointer to the PR just before running = this > type of test is OK, though. I highly disagree. Neither of us can guarantee that these tests aren=92t = being run on either production systems (this would be extremely = foolhardy, but I=92ve seen it happen before several times on the LTP = list), test systems that other developers/testers are working on where = the focus is not ZFS, or that the end-user would care about ZFS in their = test run. Continually panicking the box because of known issues is silly = and wastes developer/tester cycles. Furthermore, running the tests and = getting a subset of coverage [because the box panicked] doesn=92t help = further the goals of testing the overall system. Good developers and = testers care about coverage in addition to =93how green are my tests?=94, = =93how far is feature X to completion?=94. Idioms like the following are in my opinion sufficient for working = around issues. 6=20 7 [ "${os}" =3D "FreeBSD" ] && die "panics FreeBSD; see bug # 194589" 8 1. The issue is documented, so developers and testers know where to get = the bug information 2. The test is marked as a failure. Both of these items encourage testers and other developers to go prod = developers with knowledge and cycles to go fix the bugs so things = eventually turn green. > I agree with Alfred on this. Even though Alan's test suite > may kernel panic or cause problems, there is still value in running it > and making the results visible on jenkins.freebsd.org. > Running these tests inside a VM which is generated during the build > will allow these types of test to run, but still keep the test machine = usable, > even if the VM gets corrupted while running the tests. Again, because of some of the things that Alan said about corruption and = panicking, until the underlying issues are fixed in ZFS, there=92s = absolutely no reason why it should be run by default. As a sidenote, the FreeBSD test suite needs a couple features: 1. Additional configuration options, similar to = =93allow_sysctl_side_effects=94 used by = tests/sys/kern/kern_descrip_test.c (say =93causes_corruption=94 or = =93might_panic=94). If a. These safety belts existed b. Were off by default c. Could be queried via all of the handlers (ATF, TAP, Plain) = via some API (atf_config, etc). then I wouldn't have an issue with someone integrating them in to = the Jenkins runs without the bail outs. 1. and 2. are easy to take care = of, but 3. is missing in all but the ATF handler. 2. A hook to say =93this feature/module is present=94, e.g. =93this test = requires zfs, so if it=92s not kldloaded/statically compiled into the = kernel, don=92t run this test=94 > If we have test suites for ZFS, but no one runs them, then no one will > bother to investigate and fix the bugs. I understand but someone needs to go fix the bugs too before stuff is = turned on by default, otherwise that person will bear the brunt of the = support issues that come up with Jane/Joe user decides to go and run the = test suite and it goes and blows up their machine. > Running the test suites under automation that is visible on = jenkins.freebsd.org is going to force developers to see problems > much sooner than they do now. Just having the tests in the tree > and hoping that people run them and care to look into the problems is > not enough. Cheers. --Apple-Mail=_F769DEA7-56B3-46B3-BAFA-2F238869E838 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUS/MvAAoJEMZr5QU6S73eY1oH/2o0ujMD4+18uW8SInjf6K/D O/PysbLhkcrNKW4Qg7HKOioFjC+EDgx7u/u5+kKnECHRxEf1P5b8Ezzu4Mv7Bm4J OWvONXhS0RDki6eRgGJuVgGzBORdWvgm2xfp6h0Apga7E9yGI9ZvQqtH4hWJ6O88 85Y+PlJLp5obElOG5/y4KAWHnLinI17M6yBv2xXSNvOwMqnsw16xeCv6xIAfkjdO ZaxPCNxrZtIbDHe0DinbO+ajpOa6B5gkhFw5YykRouEiItisR6KlyjiBI/iBSxOG whXgycurIKa6xVChsxJs+NqFUNT+JtWm3BABQlB4B5VtW06Khs5cf6Ho6KxA/+c= =FO8v -----END PGP SIGNATURE----- --Apple-Mail=_F769DEA7-56B3-46B3-BAFA-2F238869E838--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?86E89F83-179D-4301-9EC0-0ECC3909B924>