Date: Wed, 5 Nov 2014 18:02:18 -0800 From: Craig Rodrigues <rodrigc@FreeBSD.org> To: Julio Merino <jmmv@freebsd.org> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org> Subject: Re: Kyua/ATF as a test framework discussion Message-ID: <CAG=rPVcMXX1LcNeXE%2BG2UnLQ%2B7AhkvVOGH8tSz5MMcsxLTkb6Q@mail.gmail.com> In-Reply-To: <CAFY7cWCHq4RM5qw0vsPDVjZj65=m6YGhwS5sx-hFCBtwAAiMqg@mail.gmail.com> References: <C3091631-0E35-40F4-BEF0-12AB68EF6B97@gmail.com> <CAG=rPVf3YhTc3GmZmjjRAEdJ9HC9nF7QA%2BEjHjEr_XO=dJAOfg@mail.gmail.com> <CAFY7cWBusbyXNK0Sq-LRa1X5qieMxY4SsNaADwpGa5p4hg%2B5Zw@mail.gmail.com> <CAG=rPVeQbv0kiw-f1NO1LbQ4hmYnuPfVQbkWJYnx1eMgS2W1XA@mail.gmail.com> <CAFY7cWCHq4RM5qw0vsPDVjZj65=m6YGhwS5sx-hFCBtwAAiMqg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Nov 5, 2014 at 3:45 PM, Julio Merino <jmmv@freebsd.org> wrote: > See my other mail in this thread. Basically, I believe the current ATF > interface is not worth fixing to support these cases and we should > look at other existing, unit-testing libraries to see if they fit our > needs. Kyua should be the glue to bring them together. In fact, you > should be able to use all the libraries you mentioned with Kyua > already (or at least that's the goal); and, if not, that's a bug that > we should fix. > I'm confused by your e-mail. Are you not interested in fixing/enhancing ATF, and want to move to "something else"? Is "something else" an incremental improvement on ATF, or something completely different and incompatible? After Garrett's recent work to import NetBSD tests into FreeBSD, we now have over 3000 tests in FreeBSD, many of which are ATF tests. I hope to see people writing more tests using ATF, but they will not be inclined to do so if it is perceived as a dead end. Also, keep in mind, we have a few companies building BSD-based products who have started using ATF and have some interest in it. kyua as it exists right now is mostly OK, and the "kyua + Jenkins" integration has been working out quite well, at least for jenkins.freebsd.org. At the recent FreeBSD Devsummit, I presented what I did: https://wiki.freebsd.org/201411DevAndVendorSummit?action=AttachFile&do=view&target=kyua_jenkins.pdf There was a lot of interest in that work. There are a few companies that want to look at that and use it as a model for testing their own products which are based on BSD. Integrating non-ATF test frameworks in kyua is not seamless right now. If I treat each non-ATF test framework as a "plain_test_program" in kyua, that works mostly OK. However, with "plain_test_program", I don't get the ability to have multiple distinct test cases in a single program, like I do with ATF, which are visible via "kyua list, kyua debug", etc. Improving things in this area is doable, for example taking Alan's Python unittest backend to kyua ( https://lists.freebsd.org/pipermail/freebsd-testing/2013-November/000104.html ) and polishing it up. Also, with "plain_test_program", if I want custom setup/teardown, I need to implement that setup/teardown inside the test program. I don't always want that, and being able to maybe specify the custom setup/teardown routines in the Kyuafile would actually be useful. ATF and Kyua are not perfect, but they are actually very, very good tools, and are taking BSD in a very good direction with respect to testing. Fixing some of the issues in ATF, and making Kyua integrate better with other frameworks will only make it better. I hope you are still interested in doing some of these things. -- Craig
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAG=rPVcMXX1LcNeXE%2BG2UnLQ%2B7AhkvVOGH8tSz5MMcsxLTkb6Q>