Date: Wed, 5 Apr 2017 11:44:51 -0700 From: "Ngie Cooper (yaneurabeya)" <yaneurabeya@gmail.com> To: Craig Rodrigues <rodrigc@freebsd.org> Cc: "freebsd-testing@freebsd.org" <testing@freebsd.org> Subject: Re: Looking at replacing ATF/Kyua (in a limited fashion) with Google Test/shunit2 Message-ID: <A66D2B01-59CB-4F4A-8A6D-0C79F15B5D49@gmail.com> In-Reply-To: <CAG=rPVegLMBhv3hXsabhs_QZ1069HPmPsRS9i2605GZ3qP9izQ@mail.gmail.com> References: <45D23581-C780-4C55-80CF-19A81813D672@gmail.com> <CAG=rPVegLMBhv3hXsabhs_QZ1069HPmPsRS9i2605GZ3qP9izQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_77D8F390-D83C-45B5-AC95-0F2CDC22B0CD
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
(Apologies for taking so long to reply; the longer the reply, the longer =
it takes for me to write a thoughtful response sometimes ;)..)
> On Jan 17, 2017, at 16:09, Craig Rodrigues <rodrigc@freebsd.org> =
wrote:
>=20
> On Tue, Jan 17, 2017 at 3:01 PM, Ngie Cooper (yaneurabeya) <
> yaneurabeya@gmail.com> wrote:
>=20
>> Hello all,
>>=20
>> I had an initial discussion with a handful of other test
>> stakeholders and due to the number of caveats with ATF/Kyua and a =
variety
>> of issues in contributing back to the ATF/Kyua projects (time on
>> maintainer=E2=80=99s end, legal issues, technical issues, etc), I'm =
seriously
>> considering replacing parts of ATF/Kyua with GoogleTest and shunit2. =
In
>> particular, I want to do these things [better]:
>=20
> I'll provide my perspective on your proposal.
> For me, the nice thing about ATF/Kyua was that Julio pushed unit tests =
to
> live along-side the code in FreeBSD,
> and provided a consistent structure by which these tests could run and
> provide test output.
> The fact that he was able to modify the Makefile system to get this =
working
> in FreeBSD *and* NetBSD
> is amazing. I never would have had the energy to do that!!
>=20
> I was hoping that ATF/Kyua would attract interest from projects other =
than
> FreeBSD and NetBSD,
> so that there would be an ecosystem of projects that would use this =
stuff
> and drive this forward.
> I updated the Mac Homebrew port of Kyua a few times, and also tried to
> submit a Debian package
> for ATF and Kyua but got stalled there.
> Unfortunately, ATF/Kyua never really took off outside of FreeBSD and =
NetBSD.
Yes. That=E2=80=99s a good summary of where we=E2=80=99ve come from.
> Also, the implementation details of modern C++ and Lua for Kyua, and =
shell
> script and C for ATF
> is a high barrier of entry for people wanting to work on the code and
> extend it.
> There are enough developers in FreeBSD who could probably pick up the =
C and
> ATF parts.
> For the C++ and Lua parts, I think the barrier is quite high, and =
there are
> not that many developers
> in the FreeBSD community with the skills or interest to work on that.
I=E2=80=99d have to look at how easy divorcing libatf-c from libatf-c++ =
would be, but I suspect it=E2=80=99s non-trivial. The ATF runners in =
kyua seem to depend on both libraries (although they can be obtained =
from ports):
$ ldd /usr/local/tests/kyua/engine/atf_* =
/usr/local/tests/kyua/bootstrap/atf_helpers | grep c++
libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 =
(0x80117b000)
libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x8015ae000)
libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 =
(0x80119f000)
libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x8015d2000)
libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 =
(0x8011a2000)
libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x8015d5000)
libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 =
(0x8011db000)
libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x80160e000)
libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 =
(0x800823000)
libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x800c56000)
> I have a selfish perspective in that I like to invest time in =
libraries and
> technologies that
> help me get jobs, even in companies that are not FreeBSD.
> In my last job at a company making a FreeBSD appliance, I tried to =
push
> adoption of ATF/Kyua
> but wasn't able to convince folks to pick it up. The developers gave =
me
> feedback that the ATF/Kyua
> documentation didn't give them confidence. It didn't help that I =
couldn't
> point to other projects
> outside of FreeBSD/NetBSD using this stuff.
> At least for me, I cannot say that ATF/Kyua is a marketable skillset =
which
> opens doors for jobs. :(
>=20
> Moving forward, if you try to steer towards libraries and technologies =
that
> have active
> user communities *even outside of FreeBSD*, that would be nice.
> Testing software isn't unique to FreeBSD, and there is a lot of good =
work
> going on in these
> other communities.
>=20
> You mentioned that GoogleTest is actively worked on by Google,
> but there at last one non-Google community (LLVM) is using it, and has =
a
> selfish interest to
> keep it going. That is one plus in my opinion.
Bingo. Hence, one of the motivators behind GoogleTest (it has more =
mindshare in the open source community than ATF/Kyua does).
> GoogleTest uses a 2-clause BSD license, so that is OK.
> GoogleTest requires signing a Contributor License Agreement for people =
who
> want to submit patches upstream to them. That's no different from =
Kyua.
> That's a bit annoying, but maybe something we can live with.
Yes.
> So I have no problem with GoogleTest on those points.
>=20
> My questions for you are:
> (1) GoogleTest is very C++-focused. ATF has API's for C, C++, and
> shell.
> How successful will you be at testing a pile of C code with a
> C++-focused library?
> It is technically possible (
> =
https://meekrosoft.wordpress.com/2009/11/09/unit-testing-c-code-with-the-g=
oogletest-framework/
> )
> but what are the gotchas?
That I need to quite frankly investigate. It might come down to the fact =
that tests need to be written in C++ =E2=80=94 which I=E2=80=99m not =
really keen on, but if the C++ is simple, and the boilerplate is simple =
enough, then the C++ shouldn=E2=80=99t need to get too fancy, and hence =
will look at lot like the existing, simple GoogleTest examples.
> (2) What do we do with existing tests? Do they need to be rewritten =
to
> the GoogleTest API?
> Or do we have infrastructure that allows existing ATF tests to =
run
> with Google Test?
Same thing as before:
1. Write new tests in the preferred target framework.
2. Don=E2=80=99t rewrite existing tests, unless there=E2=80=99s a good =
reason to do so (needless refactoring introduces risk).
As far as the second question (=E2=80=9COr do we=E2=80=A6=E2=80=9D), =
yes, my goal was to write equivalent macros, if at all possible, for =
using GoogleTest in lieu of ATF. The ownership for this task falls on =
me. I might need to write some compatibility functions/macros to ease =
the transition away from ATF, if my investigation demonstrates it's =
feasible/possible.
> (3) If this is adopted, are new tests written in GoogleTest API, or =
in
> the ATF API?
The GoogleTest APIs.
> (4) Kyua is the tool currently used for running the tests, and
> providing test result output.
> Kyua has test-case isolation, and the potential to parallelize =
the
> execution of test-cases.
> Are the Kyua features of test-case isolation and =
parallelization
> of test-cases must-haves,
> or can we live without them?
> Do you want to move away from using Kyua, or modify it to work
> with GoogleTest?
As a first step, I want to mock ATF via GoogleTest. Although =
contributing back to Google (as a corporation =E2=80=94 thanks $lawyers) =
has proved an impedance, we can always patch the port to add in the =
helper.
> If you want to move away from Kyua, how do you want people to =
run
> the GoogleTests and
> get the test results?
I really don=E2=80=99t want to change the inputs/outputs if at all =
possible =E2=80=94 just add a bit more =E2=80=9Cwindow dressing=E2=80=9D =
to denote setup/teardown fixtures and other gaps that ATF doesn=E2=80=99t =
have.
If it doesn=E2=80=99t prove fruitful, I=E2=80=99ll send out an update =
email describing the work I=E2=80=99ve done, the roadblocks/tradeoffs =
I=E2=80=99ve made in getting to the decision, and ask for some consensus =
in terms of driving the effort forward, further.
> I have no objection to your proposal. Using a more mainstream testing
> framework with a larger user community,
> and some momentum to keep things going would be nice.
Thanks!
-Ngie
--Apple-Mail=_77D8F390-D83C-45B5-AC95-0F2CDC22B0CD
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=signature.asc
Content-Type: application/pgp-signature;
name=signature.asc
Content-Description: Message signed with OpenPGP using GPGMail
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org
iQIcBAEBCgAGBQJY5TsjAAoJEPWDqSZpMIYVdY0QAI5eemX4HJVupRXPyKfx0bJc
uZ5fY2TnBj+A3y4FycPKsr+hj47SZs/IGVoosn8U4i8KCJKebPQn7Ck9YD+cRUyp
U8ihpvVxTbTCPVm4B+GatdN1dIr8jBAAHWZVyR/Uwi/OFu3U+r86/E2wJQEsjutM
VXVhzEj9PhxqhmAxHhN1p9ZRmE3hJNhlu+zy3YWn4pR8Xk4C/k0edZTuozyZCOkc
LX0Q36m+XIgPjlskMTp8xbQ2X8hnPgXz1YAJwO6hdl0F4Loucyy4FBvlgoYu5RGp
QBEukBt77io/wekCLdYICHdxg7x2LMBKlOAz+mPbL+O0Q/CrP4BCnGu1FYMHCqdH
bajZzme8ojRGTJiBJ5lzavoJjx8lj7SzVOrot986Wmwmta2mh4kmiIEJhbILkqW+
ULJG7TBudfMmJPjJJYB5oh1oliLySf6dDokPlqUeokyWHOj0Pky0uIp3nSJKlvfp
67XEsL2Z9sFG2CVTeLvNq9vCZ0rogdN1kPM3u1nPzNLXa99gIXx0UflfrH/CMi5e
NvDPLq+P7g1AIoTL4ioEVc4u4UbrObUrKvY5QSy+vIzv6Hs6b259Ox8IDYvQiMIQ
0FBcOwq4d3c+tMB6k78AQ9k/vgJJqR56rpJESgHnhUq4c6wmTbLzX3rx/dKdh9mb
O9Dy/2Bj3YD8IOHGEOgn
=Ej3a
-----END PGP SIGNATURE-----
--Apple-Mail=_77D8F390-D83C-45B5-AC95-0F2CDC22B0CD--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A66D2B01-59CB-4F4A-8A6D-0C79F15B5D49>