Date: Tue, 4 Nov 2014 12:16:31 -0700 From: Alan Somers <asomers@freebsd.org> To: Garrett Cooper <yaneurabeya@gmail.com> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org> Subject: Re: Kyua/ATF as a test framework discussion Message-ID: <CAOtMX2jpSDTkwmJYmAvGSuXEyhyJVDS=%2Bpfm3hiXaw6LeVTL-w@mail.gmail.com> In-Reply-To: <C3091631-0E35-40F4-BEF0-12AB68EF6B97@gmail.com> References: <C3091631-0E35-40F4-BEF0-12AB68EF6B97@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Nov 4, 2014 at 11:38 AM, Garrett Cooper <yaneurabeya@gmail.com> wro= te: > (developers BCCed; please continue discussion on -testing@) > Hi all, > > A few questions came up yesterday at the BSD Vendor Summit in the= Q&A session after rodrigc=E2=80=99s talk about Kyua/ATF related to testing= on FreeBSD, as related to the testing framework (there are a handful of ot= her questions I=E2=80=99ll bring up in other threads ;)..). In particular, = some of the questions that stood out in my mind were: > > 1. What type of testing does Kyua do? Can it be used in benchmark= testing, stress testing, etc? > 2. Is there a structured way of checking for feature <X> before r= unning my tests? > 3. Can I run tests with custom arguments, e.g. point something at= a filesystem path, etc? How do I do that? > > I=E2=80=99ll try to answer these questions based on experience an= d input I=E2=80=99ve received from several people both internal and externa= l to EMC/Isilon. > > Answer to 1. > > - Kyua/ATF excels as a unit testing framework and only does non-distribut= ed testing. > - Kyua/ATF should not be used in benchmark testing or stress testing. > > There is a key component missing from kyua that allows it to run = functional tests (both on FreeBSD and in general). In particular it does no= t have per test program setup/cleanup hooks that other frameworks do (cmake= , JUnit, LTP, python unittest), and it does not have a per-testcase setup h= ook. This is something that I called out several years ago when I first loo= ked at importing ATF into the base tree (see the =E2=80=9CKnown Issues=E2= =80=9D section in https://wiki.freebsd.org/TestingFreeBSD ). Not having set= up/cleanup hooks has made some testing at EMC/Isilon painful because of how= some testers have written tests, in particular there are testcases that we= re added to functionally test out mountd, mount_nfs, syslogd, etc that I ha= d to disable because of the unnecessary complexity involved in trying to ma= nage state in setting up/tearing down daemons, running syslogd, etc, and be= cause of the external state involved the testcases would work sometimes if = run end-to-end, not other times if executed one-by-one, and not other times= if run end-to-end. > > Adding these hooks isn=E2=80=99t impossible and should be the nex= t step in improving the test Kyua/ATF infrastructure (I ran out of time bef= ore trying to implement this, and I=E2=80=99m not a C++ whiz so I got lost = trying to figure out how to make everything work). I partially disagree with this answer. While it's true that per-test program setup and teardown steps would be convenient, they are by no means required. Spectra Logic has many functional and stress tests that work just fine. Code duplication due to lack of per-test program setup and teardown steps is minimal. For example, see sys/netinet/fibs_test on a CURRENT system. The ZFS test suite ( base/projects/zfsd/head/tests/sys/cddl/zfs ) was adapted from STF, which uses per-test program setup and cleanups. But it turned out that it didn't really need them, and was using them wrongly as often as not. For example, many test cases could only be run in a certain order because of interactions in their side effects. I converted all of the setup and teardown steps to per-testcase. OTOH, I do agree that Kyua should not be used for benchmarking. Its reports are much too coarse. For one thing, a Kyua test can only pass or fail (or skip, expected fail, etc), whereas a benchmark normally returns one or more real numbers. For another, in order to compare benchmarks from different systems, it is necessary to collect a great deal of information about the configuration of those systems. How much RAM? What model of CPU? What were the network card's sysctl settings? Kyua doesn't try to do that, and I don't think that it ever should. > > Answer to 2. > > There isn=E2=80=99t currently a structured API for asking questio= ns like =E2=80=9Cis this driver loaded?", =E2=80=9Cis this sysctl set?=E2= =80=9D, =E2=80=9Cam I running on UFS/ZFS?=E2=80=9D, etc. In general test ca= ses from tools/regression, etc have invented homegrown logic for handling t= his, so this isn=E2=80=99t a new issue. > > I=E2=80=99ve done some initial prototyping of a C/shell library f= or testing for feature requirements on FreeBSD, but the work isn=E2=80=99t = complete/in the tree yet (please see the mailing list threads below for mor= e details): > > - https://lists.freebsd.org/pipermail/freebsd-testing/2014-June/000439.ht= ml > - https://lists.freebsd.org/pipermail/freebsd-testing/2014-July/000467.ht= ml > > I want to complete this sometime before/around the start of the 2= 015, in particular because this blocks several of the testcases at EMC/Isil= on from being upstreamed in a good, working state. > > Answer to 3. > > Yes and no(-ish). > > For ATF tests: > > You can pass through custom arguments on the command line like so= : > > % kyua test -v mykey=3Dmyvalue > > This information is that passed through the atf APIs so you can q= uery the value with either atf-config or atf_tc_config (unfortunately the m= echanism I=E2=80=99m used to differs enough from what is currently in place= , so I=E2=80=99ll need to go look up what the current mechanism/structure i= s for this, but basically it involves specifying variables kyua.conf). Also, you can put those settings in /usr/local/etc/kyua/kyua.conf instead of on the kyua command line. > > For non-ATF tests: > > You have to resort to using environment variables today, like I d= id with pjdfstest: https://svnweb.freebsd.org/base/head/tests/sys/pjdfstest= /tests/conf?revision=3D274016&view=3Dmarkup#l5 , as I described in the READ= ME file: https://svnweb.freebsd.org/base/head/share/doc/pjdfstest/README?re= vision=3D274016&view=3Dmarkup . I=E2=80=99d _really_ like to avoid doing th= is though because then we=E2=80=99ll have 40 some different ways to do thin= gs around the tree instead of a single, supported way to do things. > > There was an enhancement I opened on GitHub to add =E2=80=9Cconfi= guration integration=E2=80=9D for the other test types (Plain, TAP) [via en= vironment variables or command querying], but that wasn=E2=80=99t completed= : https://github.com/jmmv/kyua/issues/84 . I=E2=80=99ll reopen the discussi= on. > > I=E2=80=99ll try to add/migrate relevant links for things noted i= n this email on the wiki. > > Thank you! -Alan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2jpSDTkwmJYmAvGSuXEyhyJVDS=%2Bpfm3hiXaw6LeVTL-w>