From owner-freebsd-testing@freebsd.org Wed Apr 5 18:44:57 2017 Return-Path: Delivered-To: freebsd-testing@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 48FCBD308E5 for ; Wed, 5 Apr 2017 18:44:57 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 224457DA for ; Wed, 5 Apr 2017 18:44:57 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 1EBCCD308E2; Wed, 5 Apr 2017 18:44:57 +0000 (UTC) Delivered-To: testing@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1E63AD308E1 for ; Wed, 5 Apr 2017 18:44:57 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: from mail-pg0-x22b.google.com (mail-pg0-x22b.google.com [IPv6:2607:f8b0:400e:c05::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E1ACF7CA; Wed, 5 Apr 2017 18:44:53 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: by mail-pg0-x22b.google.com with SMTP id 21so12550274pgg.1; Wed, 05 Apr 2017 11:44:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:mime-version:from:in-reply-to:date:cc:message-id:references :to; bh=1U5g6l6GXsxRgT2dvzWyEWJnUpHvPgL/G3gBeDUXVfk=; b=rsVhjr6uV7wtSbwTL9X72IMGMguVRlBOak9Yew8EliQ/N3M1zrW6VwYDgxeNJBKvA1 2F7kaaGb1IgKpB6o8SpLIA3h/1BJAUVuzH+NkOM921NUTES81foH/mljaU1Yyf1bAP9Q H8GQsJpCVKHpDBmdTjbawRibMeaqa4Ojp/bsdZi2CBGLF/RGhIBDiyWEM27SxkLiS2WJ cP234kD7aIOrOEGypylnwh3IDCWWA/bO9Rln/NlkADfDR8X1Zw7COCBfoMRt4nw1vbt3 P9Mab6xg308ruXzohPo6hU/SmSfXgzgWVvKqiMYqlL8rEa5U+rwyPbtCon1GhAmhsNek /Dsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:mime-version:from:in-reply-to:date:cc :message-id:references:to; bh=1U5g6l6GXsxRgT2dvzWyEWJnUpHvPgL/G3gBeDUXVfk=; b=nRQpZUt2EdAiYdv2Z4FCAyWpIWeVjD7Uv6vq8JJaEDJhSw9EvwRfqSSoSdn8nBDmaA o731z4liCrZbJ+WPNiDX0j62mDaRInphzZisxRooEIeQ5NvmLenT1o7IBhCanuPUpK03 J8ndWjE0a6jm26tFWyg8rMVzLQXh5r3xff7aJKRJt8NU+HR0DOR8+5mJzxX53toJ8lTs 5r5sIEnLrEIPj8H4jgCMp0YHZ5OH7r51oPg8Q2MIV9msUQdbXxcIhS0MbeJwVW670m6a emxAF8/Fv02qk0XkvzCxiMGGinroKLxi/qmXRP5lw0hN9Fpbu1TwSr7siCXD3Xdl8SLh u+bg== X-Gm-Message-State: AFeK/H3OdJGN59I5W7xBO4KkR9OaZQvhyZpipKMlf09yIFXBb0xTqJdHR9l+RlZCSXpNbg== X-Received: by 10.84.129.3 with SMTP id 3mr38305065plb.150.1491417893024; Wed, 05 Apr 2017 11:44:53 -0700 (PDT) Received: from pinklady.local (c-73-19-52-228.hsd1.wa.comcast.net. [73.19.52.228]) by smtp.gmail.com with ESMTPSA id b128sm38939794pfg.54.2017.04.05.11.44.52 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 05 Apr 2017 11:44:52 -0700 (PDT) Subject: Re: Looking at replacing ATF/Kyua (in a limited fashion) with Google Test/shunit2 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Content-Type: multipart/signed; boundary="Apple-Mail=_77D8F390-D83C-45B5-AC95-0F2CDC22B0CD"; protocol="application/pgp-signature"; micalg=pgp-sha512 X-Pgp-Agent: GPGMail From: "Ngie Cooper (yaneurabeya)" In-Reply-To: Date: Wed, 5 Apr 2017 11:44:51 -0700 Cc: "freebsd-testing@freebsd.org" Message-Id: References: <45D23581-C780-4C55-80CF-19A81813D672@gmail.com> To: Craig Rodrigues X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Apr 2017 18:44:57 -0000 --Apple-Mail=_77D8F390-D83C-45B5-AC95-0F2CDC22B0CD Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 (Apologies for taking so long to reply; the longer the reply, the longer = it takes for me to write a thoughtful response sometimes ;)..) > On Jan 17, 2017, at 16:09, Craig Rodrigues = wrote: >=20 > On Tue, Jan 17, 2017 at 3:01 PM, Ngie Cooper (yaneurabeya) < > yaneurabeya@gmail.com> wrote: >=20 >> Hello all, >>=20 >> I had an initial discussion with a handful of other test >> stakeholders and due to the number of caveats with ATF/Kyua and a = variety >> of issues in contributing back to the ATF/Kyua projects (time on >> maintainer=E2=80=99s end, legal issues, technical issues, etc), I'm = seriously >> considering replacing parts of ATF/Kyua with GoogleTest and shunit2. = In >> particular, I want to do these things [better]: >=20 > I'll provide my perspective on your proposal. > For me, the nice thing about ATF/Kyua was that Julio pushed unit tests = to > live along-side the code in FreeBSD, > and provided a consistent structure by which these tests could run and > provide test output. > The fact that he was able to modify the Makefile system to get this = working > in FreeBSD *and* NetBSD > is amazing. I never would have had the energy to do that!! >=20 > I was hoping that ATF/Kyua would attract interest from projects other = than > FreeBSD and NetBSD, > so that there would be an ecosystem of projects that would use this = stuff > and drive this forward. > I updated the Mac Homebrew port of Kyua a few times, and also tried to > submit a Debian package > for ATF and Kyua but got stalled there. > Unfortunately, ATF/Kyua never really took off outside of FreeBSD and = NetBSD. Yes. That=E2=80=99s a good summary of where we=E2=80=99ve come from. > Also, the implementation details of modern C++ and Lua for Kyua, and = shell > script and C for ATF > is a high barrier of entry for people wanting to work on the code and > extend it. > There are enough developers in FreeBSD who could probably pick up the = C and > ATF parts. > For the C++ and Lua parts, I think the barrier is quite high, and = there are > not that many developers > in the FreeBSD community with the skills or interest to work on that. I=E2=80=99d have to look at how easy divorcing libatf-c from libatf-c++ = would be, but I suspect it=E2=80=99s non-trivial. The ATF runners in = kyua seem to depend on both libraries (although they can be obtained = from ports): $ ldd /usr/local/tests/kyua/engine/atf_* = /usr/local/tests/kyua/bootstrap/atf_helpers | grep c++ libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 = (0x80117b000) libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x8015ae000) libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 = (0x80119f000) libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x8015d2000) libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 = (0x8011a2000) libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x8015d5000) libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 = (0x8011db000) libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x80160e000) libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 = (0x800823000) libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x800c56000) > I have a selfish perspective in that I like to invest time in = libraries and > technologies that > help me get jobs, even in companies that are not FreeBSD. > In my last job at a company making a FreeBSD appliance, I tried to = push > adoption of ATF/Kyua > but wasn't able to convince folks to pick it up. The developers gave = me > feedback that the ATF/Kyua > documentation didn't give them confidence. It didn't help that I = couldn't > point to other projects > outside of FreeBSD/NetBSD using this stuff. > At least for me, I cannot say that ATF/Kyua is a marketable skillset = which > opens doors for jobs. :( >=20 > Moving forward, if you try to steer towards libraries and technologies = that > have active > user communities *even outside of FreeBSD*, that would be nice. > Testing software isn't unique to FreeBSD, and there is a lot of good = work > going on in these > other communities. >=20 > You mentioned that GoogleTest is actively worked on by Google, > but there at last one non-Google community (LLVM) is using it, and has = a > selfish interest to > keep it going. That is one plus in my opinion. Bingo. Hence, one of the motivators behind GoogleTest (it has more = mindshare in the open source community than ATF/Kyua does). > GoogleTest uses a 2-clause BSD license, so that is OK. > GoogleTest requires signing a Contributor License Agreement for people = who > want to submit patches upstream to them. That's no different from = Kyua. > That's a bit annoying, but maybe something we can live with. Yes. > So I have no problem with GoogleTest on those points. >=20 > My questions for you are: > (1) GoogleTest is very C++-focused. ATF has API's for C, C++, and > shell. > How successful will you be at testing a pile of C code with a > C++-focused library? > It is technically possible ( > = https://meekrosoft.wordpress.com/2009/11/09/unit-testing-c-code-with-the-g= oogletest-framework/ > ) > but what are the gotchas? That I need to quite frankly investigate. It might come down to the fact = that tests need to be written in C++ =E2=80=94 which I=E2=80=99m not = really keen on, but if the C++ is simple, and the boilerplate is simple = enough, then the C++ shouldn=E2=80=99t need to get too fancy, and hence = will look at lot like the existing, simple GoogleTest examples. > (2) What do we do with existing tests? Do they need to be rewritten = to > the GoogleTest API? > Or do we have infrastructure that allows existing ATF tests to = run > with Google Test? Same thing as before: 1. Write new tests in the preferred target framework. 2. Don=E2=80=99t rewrite existing tests, unless there=E2=80=99s a good = reason to do so (needless refactoring introduces risk). As far as the second question (=E2=80=9COr do we=E2=80=A6=E2=80=9D), = yes, my goal was to write equivalent macros, if at all possible, for = using GoogleTest in lieu of ATF. The ownership for this task falls on = me. I might need to write some compatibility functions/macros to ease = the transition away from ATF, if my investigation demonstrates it's = feasible/possible. > (3) If this is adopted, are new tests written in GoogleTest API, or = in > the ATF API? The GoogleTest APIs. > (4) Kyua is the tool currently used for running the tests, and > providing test result output. > Kyua has test-case isolation, and the potential to parallelize = the > execution of test-cases. > Are the Kyua features of test-case isolation and = parallelization > of test-cases must-haves, > or can we live without them? > Do you want to move away from using Kyua, or modify it to work > with GoogleTest? As a first step, I want to mock ATF via GoogleTest. Although = contributing back to Google (as a corporation =E2=80=94 thanks $lawyers) = has proved an impedance, we can always patch the port to add in the = helper. > If you want to move away from Kyua, how do you want people to = run > the GoogleTests and > get the test results? I really don=E2=80=99t want to change the inputs/outputs if at all = possible =E2=80=94 just add a bit more =E2=80=9Cwindow dressing=E2=80=9D = to denote setup/teardown fixtures and other gaps that ATF doesn=E2=80=99t = have. If it doesn=E2=80=99t prove fruitful, I=E2=80=99ll send out an update = email describing the work I=E2=80=99ve done, the roadblocks/tradeoffs = I=E2=80=99ve made in getting to the decision, and ask for some consensus = in terms of driving the effort forward, further. > I have no objection to your proposal. Using a more mainstream testing > framework with a larger user community, > and some momentum to keep things going would be nice. Thanks! -Ngie --Apple-Mail=_77D8F390-D83C-45B5-AC95-0F2CDC22B0CD Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJY5TsjAAoJEPWDqSZpMIYVdY0QAI5eemX4HJVupRXPyKfx0bJc uZ5fY2TnBj+A3y4FycPKsr+hj47SZs/IGVoosn8U4i8KCJKebPQn7Ck9YD+cRUyp U8ihpvVxTbTCPVm4B+GatdN1dIr8jBAAHWZVyR/Uwi/OFu3U+r86/E2wJQEsjutM VXVhzEj9PhxqhmAxHhN1p9ZRmE3hJNhlu+zy3YWn4pR8Xk4C/k0edZTuozyZCOkc LX0Q36m+XIgPjlskMTp8xbQ2X8hnPgXz1YAJwO6hdl0F4Loucyy4FBvlgoYu5RGp QBEukBt77io/wekCLdYICHdxg7x2LMBKlOAz+mPbL+O0Q/CrP4BCnGu1FYMHCqdH bajZzme8ojRGTJiBJ5lzavoJjx8lj7SzVOrot986Wmwmta2mh4kmiIEJhbILkqW+ ULJG7TBudfMmJPjJJYB5oh1oliLySf6dDokPlqUeokyWHOj0Pky0uIp3nSJKlvfp 67XEsL2Z9sFG2CVTeLvNq9vCZ0rogdN1kPM3u1nPzNLXa99gIXx0UflfrH/CMi5e NvDPLq+P7g1AIoTL4ioEVc4u4UbrObUrKvY5QSy+vIzv6Hs6b259Ox8IDYvQiMIQ 0FBcOwq4d3c+tMB6k78AQ9k/vgJJqR56rpJESgHnhUq4c6wmTbLzX3rx/dKdh9mb O9Dy/2Bj3YD8IOHGEOgn =Ej3a -----END PGP SIGNATURE----- --Apple-Mail=_77D8F390-D83C-45B5-AC95-0F2CDC22B0CD--