Date: Tue, 4 Nov 2014 11:40:34 -0800 From: Craig Rodrigues <rodrigc@FreeBSD.org> To: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org> Subject: Re: Kyua/ATF as a test framework discussion Message-ID: <CAG=rPVf3YhTc3GmZmjjRAEdJ9HC9nF7QA%2BEjHjEr_XO=dJAOfg@mail.gmail.com> In-Reply-To: <C3091631-0E35-40F4-BEF0-12AB68EF6B97@gmail.com> References: <C3091631-0E35-40F4-BEF0-12AB68EF6B97@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Nov 4, 2014 at 10:38 AM, Garrett Cooper <yaneurabeya@gmail.com> wrote: > There is a key component missing from kyua that allows it to run > functional tests (both on FreeBSD and in general). In particular it does > not have per test program setup/cleanup hooks that other frameworks do > (cmake, JUnit, LTP, python unittest), and it does not have a per-testcase > setup hook. > This is something that I called out several years ago when I first looked > at importing ATF into the base tree (see the "Known Issues" section in > https://wiki.freebsd.org/TestingFreeBSD ). Not having setup/cleanup hooks > has made some testing at EMC/Isilon painful because of how some testers > have written tests, in particular there are testcases that were added to > functionally test out mountd, mount_nfs, syslogd, etc that I had to disable > because of the unnecessary complexity involved in trying to manage state in > setting up/tearing down daemons, running syslogd, etc, and because of the > external state involved the testcases would work sometimes if run > end-to-end, not other times if executed one-by-one, and not other times if > run end-to-end. > > I've raised this issue on kyua-discuss and Julio is not in favor of it: https://groups.google.com/forum/#!msg/kyua-discuss/6DmYvtLN66I/MuG528I1V1AJ I agree with you and disagree with Julio. In some cases, having the hooks in the test framework to do setup/cleanup outside of the test itself is useful. My experience is that people who write tests don't always write perfect code, and do not always do proper setup/teardown inside the testcase. In FreeBSD and NetBSD, we can enforce this and force people to fix the tests. In large products based on FreeBSD, my experience is that this is not often possible in real-world QA environments. Having hooks in the test framework to do setup/teardown like in other unit test frameworks would be useful. -- Craig
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAG=rPVf3YhTc3GmZmjjRAEdJ9HC9nF7QA%2BEjHjEr_XO=dJAOfg>