Date: Tue, 4 Nov 2014 10:38:03 -0800 From: Garrett Cooper <yaneurabeya@gmail.com> To: "freebsd-testing@freebsd.org" <freebsd-testing@FreeBSD.org> Subject: Kyua/ATF as a test framework discussion Message-ID: <C3091631-0E35-40F4-BEF0-12AB68EF6B97@gmail.com>
next in thread | raw e-mail | index | archive | help
--Apple-Mail=_5C25921D-35F7-462F-B10A-B36F2089D9EC Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 (developers BCCed; please continue discussion on -testing@) Hi all, A few questions came up yesterday at the BSD Vendor Summit in = the Q&A session after rodrigc=92s talk about Kyua/ATF related to testing = on FreeBSD, as related to the testing framework (there are a handful of = other questions I=92ll bring up in other threads ;)..). In particular, = some of the questions that stood out in my mind were: 1. What type of testing does Kyua do? Can it be used in = benchmark testing, stress testing, etc? 2. Is there a structured way of checking for feature <X> before = running my tests? 3. Can I run tests with custom arguments, e.g. point something = at a filesystem path, etc? How do I do that? I=92ll try to answer these questions based on experience and = input I=92ve received from several people both internal and external to = EMC/Isilon. Answer to 1. - Kyua/ATF excels as a unit testing framework and only does = non-distributed testing. - Kyua/ATF should not be used in benchmark testing or stress testing. There is a key component missing from kyua that allows it to run = functional tests (both on FreeBSD and in general). In particular it does = not have per test program setup/cleanup hooks that other frameworks do = (cmake, JUnit, LTP, python unittest), and it does not have a = per-testcase setup hook. This is something that I called out several = years ago when I first looked at importing ATF into the base tree (see = the =93Known Issues=94 section in = https://wiki.freebsd.org/TestingFreeBSD ). Not having setup/cleanup = hooks has made some testing at EMC/Isilon painful because of how some = testers have written tests, in particular there are testcases that were = added to functionally test out mountd, mount_nfs, syslogd, etc that I = had to disable because of the unnecessary complexity involved in trying = to manage state in setting up/tearing down daemons, running syslogd, = etc, and because of the external state involved the testcases would work = sometimes if run end-to-end, not other times if executed one-by-one, and = not other times if run end-to-end. Adding these hooks isn=92t impossible and should be the next = step in improving the test Kyua/ATF infrastructure (I ran out of time = before trying to implement this, and I=92m not a C++ whiz so I got lost = trying to figure out how to make everything work). Answer to 2. There isn=92t currently a structured API for asking questions = like =93is this driver loaded?", =93is this sysctl set?=94, =93am I = running on UFS/ZFS?=94, etc. In general test cases from = tools/regression, etc have invented homegrown logic for handling this, = so this isn=92t a new issue. I=92ve done some initial prototyping of a C/shell library for = testing for feature requirements on FreeBSD, but the work isn=92t = complete/in the tree yet (please see the mailing list threads below for = more details): - = https://lists.freebsd.org/pipermail/freebsd-testing/2014-June/000439.html - = https://lists.freebsd.org/pipermail/freebsd-testing/2014-July/000467.html I want to complete this sometime before/around the start of the = 2015, in particular because this blocks several of the testcases at = EMC/Isilon from being upstreamed in a good, working state. Answer to 3. Yes and no(-ish). For ATF tests: You can pass through custom arguments on the command line like = so: % kyua test -v mykey=3Dmyvalue This information is that passed through the atf APIs so you can = query the value with either atf-config or atf_tc_config (unfortunately = the mechanism I=92m used to differs enough from what is currently in = place, so I=92ll need to go look up what the current mechanism/structure = is for this, but basically it involves specifying variables kyua.conf). For non-ATF tests: You have to resort to using environment variables today, like I = did with pjdfstest: = https://svnweb.freebsd.org/base/head/tests/sys/pjdfstest/tests/conf?revisi= on=3D274016&view=3Dmarkup#l5 , as I described in the README file: = https://svnweb.freebsd.org/base/head/share/doc/pjdfstest/README?revision=3D= 274016&view=3Dmarkup . I=92d _really_ like to avoid doing this though = because then we=92ll have 40 some different ways to do things around the = tree instead of a single, supported way to do things. There was an enhancement I opened on GitHub to add = =93configuration integration=94 for the other test types (Plain, TAP) = [via environment variables or command querying], but that wasn=92t = completed: https://github.com/jmmv/kyua/issues/84 . I=92ll reopen the = discussion. I=92ll try to add/migrate relevant links for things noted in = this email on the wiki. Thank you! --Apple-Mail=_5C25921D-35F7-462F-B10A-B36F2089D9EC Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUWR0LAAoJEMZr5QU6S73eUiEH/1xeeonM4LbOCYkAWujkhZwA rgib/2fzwJnvxOi92lLLGupvRzXJE7Dy3Dru7+FsDXs9KQ1WIUyKdjbR7wi/mXAY kNSzsG8138Oe6oAc/ow2rQxW0mn6PZCA9GAmmZZ/qJ/5EDlGAqLJmiRDUPqL8AG1 m5HtLqFgHlozJyK5mpxrBCtjSOISCsF9oip6HSFT1AFFuNgZSHseUzVFnbpUrcJG NUvkrkQdOn1CeftpmFu41PtAD+tcDcfhlUpYDN8eRcne9Pnr2Ed+3p6z73qs0rMB fFVD5rBy8gd5WkpUvVaOifr238kanenA9UFaXYtIvgzoRbYgoWf4dSOKl2BLHRY= =POKQ -----END PGP SIGNATURE----- --Apple-Mail=_5C25921D-35F7-462F-B10A-B36F2089D9EC--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?C3091631-0E35-40F4-BEF0-12AB68EF6B97>