Date: Tue, 17 Jan 2017 16:11:18 -0700 From: Alan Somers <asomers@freebsd.org> To: "Ngie Cooper (yaneurabeya)" <yaneurabeya@gmail.com> Cc: "freebsd-testing@freebsd.org" <testing@freebsd.org> Subject: Re: Looking at replacing ATF/Kyua (in a limited fashion) with Google Test/shunit2 Message-ID: <CAOtMX2i5ChWgLz7mMwVAoG=YS0H7zRXgDpBwuSMSoHva-wUfug@mail.gmail.com> In-Reply-To: <45D23581-C780-4C55-80CF-19A81813D672@gmail.com> References: <45D23581-C780-4C55-80CF-19A81813D672@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Jan 17, 2017 at 4:01 PM, Ngie Cooper (yaneurabeya) <yaneurabeya@gmail.com> wrote: > Hello all, > > I had an initial discussion with a handful of other test stakehol= ders and due to the number of caveats with ATF/Kyua and a variety of issues= in contributing back to the ATF/Kyua projects (time on maintainer=E2=80=99= s end, legal issues, technical issues, etc), I'm seriously considering repl= acing parts of ATF/Kyua with GoogleTest and shunit2. In particular, I want = to do these things [better]: > > - Detecting kernels features and modules. It is ad hoc and done o= n a per-testcase basis or in ad hoc common routines, which bloats tests unn= ecessarily. > - Detecting network interfaces. My employer desires that =E2=80= =94 we had something in use internally for detecting interfaces in tests, b= ut it was really messy; it would be nice if there was a clean, generalized = way to do this. > - Adding test fixtures to ease testing on a per-suite basis. For = example, setting up a service like bsnmpd for testing, running pjdfstest in= different configurations, (a temporarily setup UFS or ZFS filesystem, etc)= , collecting core files after tests for analysis purposes, etc. > Why these frameworks? > > GoogleTest: > - It is easier to extend upon. > - It is better documented than ATF/Kyua. > - It has more brevity in terms of writing testcases. > - It has features that I (and others who write tests) really want= . > - It is very similar to atf-c-api in core ways, in terms of macro= s. > - It has been accepted by multiple open source projects, like cla= ng/llvm, so including them in the FreeBSD Test Suite is more trivial after = the porting and integrating work. > - It is lightweight and a good candidate for inclusion in the bas= e system. > - Google and other groups are actively maintaining the project. > > shunit2: > - It has more brevity in terms of writing testcases. > - It has features that I (and others who write tests) really want= . > - It has syntactic sugar/API construction patterned after JUnit/X= Unit in other ways, so it=E2=80=99s familiar to testers who have written JU= nit/XUnit tests. > > Some concerns that have been brought up: > > Q: Why those frameworks? Have you considered other options? > A: Yes. A number of users have provided a matrix on Wikipedia and= I=E2=80=99ve looked at the list to try and determine : > a. C: https://en.wikipedia.org/wiki/List_of_unit_testing_= frameworks#C > b. C++: https://en.wikipedia.org/wiki/List_of_unit_testin= g_frameworks#C.2B.2B > c. sh: https://en.wikipedia.org/wiki/List_of_unit_testing= _frameworks#Shell > > Q: Google sometimes tinkers with projects, then abandons them lat= er. Is GoogleTest one of those projects? > A: As far as I can tell, no. This is widely used internally and d= eveloped on by their engineers. It is used external to Google by larger pro= jects like LLVM, so even if Google disappeared tomorrow I believe there=E2= =80=99s enough inertia and interest to keep things going here. > > Q: What about C test integration? > A: I=E2=80=99m trying to see how far I can get with GoogleTest in= tegration on the C API side, and if replacing atf-c-api is possible/doable = using C integration in libc++ (the syntax/API construction for atf-c-api an= d GoogleTest is similar in some regards). > > I=E2=80=99m really early on in the evaluation phase, so if anyone= has any input to provide with the direction I=E2=80=99m going, I would rea= lly love the input. Writing tests for bsnmp will be my first proof of conce= pt (I hope to use both GoogleTest and shunit2). > > Thank you, > -Ngie Before you get to far, you should ask yourself how much test case isolation you desire. Googletest is great, but I think it'll be hard to get the strong isolation between testcases provided by ATF. Perhaps it'll be find if GoogleTest tests provide no per-testcase isolation; most C/C++ programmers don't expect it anyway. But many of FreeBSD's atf-sh tests depend on it. Also, how are you planning to hook up GoogleTest and shunit2 to Kyua? -Alan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2i5ChWgLz7mMwVAoG=YS0H7zRXgDpBwuSMSoHva-wUfug>