Date: Wed, 5 Nov 2014 16:41:12 -0500 From: Julio Merino <jmmv@freebsd.org> To: Garrett Cooper <yaneurabeya@gmail.com> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org> Subject: Re: Kyua/ATF as a test framework discussion Message-ID: <CAFY7cWBm9FKrpZr5aeN6y_Po21wADuDNyh33cnMPbsAS=2SvVA@mail.gmail.com> In-Reply-To: <C3091631-0E35-40F4-BEF0-12AB68EF6B97@gmail.com> References: <C3091631-0E35-40F4-BEF0-12AB68EF6B97@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Nov 4, 2014 at 1:38 PM, Garrett Cooper <yaneurabeya@gmail.com> wrot= e: > (developers BCCed; please continue discussion on -testing@) > Hi all, > > A few questions came up yesterday at the BSD Vendor Summit in the= Q&A session after rodrigc=E2=80=99s talk about Kyua/ATF related to testing= on FreeBSD, as related to the testing framework (there are a handful of ot= her questions I=E2=80=99ll bring up in other threads ;)..). In particular, = some of the questions that stood out in my mind were: > > 1. What type of testing does Kyua do? Can it be used in benchmark= testing, stress testing, etc? > 2. Is there a structured way of checking for feature <X> before r= unning my tests? > 3. Can I run tests with custom arguments, e.g. point something at= a filesystem path, etc? How do I do that? > > I=E2=80=99ll try to answer these questions based on experience an= d input I=E2=80=99ve received from several people both internal and externa= l to EMC/Isilon. > > Answer to 1. > > - Kyua/ATF excels as a unit testing framework and only does non-distribut= ed testing. > - Kyua/ATF should not be used in benchmark testing or stress testing. This is a fair description. > There is a key component missing from kyua that allows it to run = functional tests (both on FreeBSD and in general). In particular it does no= t have per test program setup/cleanup hooks that other frameworks do (cmake= , JUnit, LTP, python unittest), and it does not have a per-testcase setup h= ook. Well, this is specifically ATF's fault. The more I look at ATF and other testing libraries, the more I'm convinced that ATF's approach of forcing every test case be in a completely separate process is suboptimal because it prevents having fixtures altogether. If you write your own test program without ATF, then you can easily implement your own global setup and teardown routines. (In fact, the alternative library I'm playing with in https://github.com/jmmv/shtk/tree/unittest actually supports per-test setup/teardown and it'd be trivial to add per-program setup/teardown as well.) Of course, "write your own test program" is poor advice given that doing so from scratch, without support libraries, sucks. The problem is that I do not believe that the ATF interface is fixable to support these cases; we'd need to write a different library, and at that point... is it worth doing? What else is there? You should be able to easily plug existing, more advanced libraries like pyUnit or googletest into Kyua -- and if there is a problem with doing such integration, we should fix it. > Answer to 2. > > There isn=E2=80=99t currently a structured API for asking questio= ns like =E2=80=9Cis this driver loaded?", =E2=80=9Cis this sysctl set?=E2= =80=9D, =E2=80=9Cam I running on UFS/ZFS?=E2=80=9D, etc. In general test ca= ses from tools/regression, etc have invented homegrown logic for handling t= his, so this isn=E2=80=99t a new issue. Having such functionality would be great, but IMHO the majority of it is so tied to FreeBSD that it has to live in the FreeBSD source tree. > Answer to 3. > > Yes and no(-ish). A major goal when designing ATF back then was precisely to avoid variability in test execution: all test programs should behave the same way so that their execution can be scripted and so that the caller (user) needn't know how to run the test program (usually poorly documented). Aren't configuration variables passed through the environment sufficient? Kyua 0.11 should do that. (Note that you say above that #84 wasn't completed, but I'm not sure why.)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFY7cWBm9FKrpZr5aeN6y_Po21wADuDNyh33cnMPbsAS=2SvVA>