From owner-freebsd-testing@freebsd.org Wed Jan 18 00:09:24 2017 Return-Path: Delivered-To: freebsd-testing@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5A957CB5F54 for ; Wed, 18 Jan 2017 00:09:24 +0000 (UTC) (envelope-from crodr001@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 384631E85 for ; Wed, 18 Jan 2017 00:09:24 +0000 (UTC) (envelope-from crodr001@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 37A2DCB5F53; Wed, 18 Jan 2017 00:09:24 +0000 (UTC) Delivered-To: testing@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3744ACB5F52 for ; Wed, 18 Jan 2017 00:09:24 +0000 (UTC) (envelope-from crodr001@gmail.com) Received: from mail-wm0-x22a.google.com (mail-wm0-x22a.google.com [IPv6:2a00:1450:400c:c09::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BA76F1E84 for ; Wed, 18 Jan 2017 00:09:23 +0000 (UTC) (envelope-from crodr001@gmail.com) Received: by mail-wm0-x22a.google.com with SMTP id f73so42925080wmf.1 for ; Tue, 17 Jan 2017 16:09:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to; bh=Zy4z5rlJueHb5EFmuJo8CghbtKmkqjiFE/D8l5jRI0k=; b=bZNzuewuZPPf2B+HYHP6TyN4ZF1/imGteA6LxnwgV8SHT0F8wBSagls3XUjLqVNT26 gIPzPmyPLWNoYJclWD0BtAHx0ccv7YCjc4KZoONExE0rKb2rjbM2RnqCw5G31ELQwgYf B4udlvaGR9WdR9fInxNARmkaNYy0nCloKZvv9BjdS8PVVc/pqapWDzuKzX7Nu7Nbci6x XSL9WdkW4kHcTW5SxZss2/B0D6xqCNc9miJpVURE6Svcs5OWTGmu4NKgKUjhGI5b90V5 kpp3jt4+p64QyCllTzcD3LYHzWLVUjaWvwpf+L87FVLXkF2H5MWaPT8QmvHIN93jxDx/ wTmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to; bh=Zy4z5rlJueHb5EFmuJo8CghbtKmkqjiFE/D8l5jRI0k=; b=LwtrirnLA/Y6ESvzlwTH0+8dHrivXRqwW9msUCR6DzOhLo9cFtDXoJ8VIX8K1tp3Ne prU+e3pMfB0xxzsn0x9PdL3cUK+Uz1gT/RE4aW2rWYtV8lga9wH5cSuFw4No8xJ73tHn oaj+aRsEPWWM6X9tIBNs5hM5QBBq88aZrOtf3bRq8h6YvCh9gijbnOg+Dtb96ipEkeaM hqwMDwlWuKXG+1zFcT1M3EqN9dGfCvGia8E0OkTahimV+3vudBJglyaqAv1rWnWhVGEi JhaF5+rKFcmmL44NrBRbCzj7UW+7DBReRo/DVk0Nnnmm4eKEUZaSfsF6JgwSVYiRqoej /yag== X-Gm-Message-State: AIkVDXLfdlJ4sfNnNeAr03f462qmwGSWzf6LbKZ48wFOxu4KHnyB86Kvxsm+tcr/2QBjwNCU0aLv6vnXYe/p6Q== X-Received: by 10.223.164.203 with SMTP id h11mr201724wrb.153.1484698162058; Tue, 17 Jan 2017 16:09:22 -0800 (PST) MIME-Version: 1.0 Sender: crodr001@gmail.com Received: by 10.28.163.195 with HTTP; Tue, 17 Jan 2017 16:09:21 -0800 (PST) In-Reply-To: <45D23581-C780-4C55-80CF-19A81813D672@gmail.com> References: <45D23581-C780-4C55-80CF-19A81813D672@gmail.com> From: Craig Rodrigues Date: Tue, 17 Jan 2017 16:09:21 -0800 X-Google-Sender-Auth: n7SdTBcdf0JLbwVUe5x0XcKZBzI Message-ID: Subject: Re: Looking at replacing ATF/Kyua (in a limited fashion) with Google Test/shunit2 To: "freebsd-testing@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jan 2017 00:09:24 -0000 On Tue, Jan 17, 2017 at 3:01 PM, Ngie Cooper (yaneurabeya) < yaneurabeya@gmail.com> wrote: > Hello all, > > I had an initial discussion with a handful of other test > stakeholders and due to the number of caveats with ATF/Kyua and a variety > of issues in contributing back to the ATF/Kyua projects (time on > maintainer=E2=80=99s end, legal issues, technical issues, etc), I'm serio= usly > considering replacing parts of ATF/Kyua with GoogleTest and shunit2. In > particular, I want to do these things [better]: I'll provide my perspective on your proposal. For me, the nice thing about ATF/Kyua was that Julio pushed unit tests to live along-side the code in FreeBSD, and provided a consistent structure by which these tests could run and provide test output. The fact that he was able to modify the Makefile system to get this working in FreeBSD *and* NetBSD is amazing. I never would have had the energy to do that!! I was hoping that ATF/Kyua would attract interest from projects other than FreeBSD and NetBSD, so that there would be an ecosystem of projects that would use this stuff and drive this forward. I updated the Mac Homebrew port of Kyua a few times, and also tried to submit a Debian package for ATF and Kyua but got stalled there. Unfortunately, ATF/Kyua never really took off outside of FreeBSD and NetBSD= . Also, the implementation details of modern C++ and Lua for Kyua, and shell script and C for ATF is a high barrier of entry for people wanting to work on the code and extend it. There are enough developers in FreeBSD who could probably pick up the C and ATF parts. For the C++ and Lua parts, I think the barrier is quite high, and there are not that many developers in the FreeBSD community with the skills or interest to work on that. I have a selfish perspective in that I like to invest time in libraries and technologies that help me get jobs, even in companies that are not FreeBSD. In my last job at a company making a FreeBSD appliance, I tried to push adoption of ATF/Kyua but wasn't able to convince folks to pick it up. The developers gave me feedback that the ATF/Kyua documentation didn't give them confidence. It didn't help that I couldn't point to other projects outside of FreeBSD/NetBSD using this stuff. At least for me, I cannot say that ATF/Kyua is a marketable skillset which opens doors for jobs. :( Moving forward, if you try to steer towards libraries and technologies that have active user communities *even outside of FreeBSD*, that would be nice. Testing software isn't unique to FreeBSD, and there is a lot of good work going on in these other communities. You mentioned that GoogleTest is actively worked on by Google, but there at last one non-Google community (LLVM) is using it, and has a selfish interest to keep it going. That is one plus in my opinion. GoogleTest uses a 2-clause BSD license, so that is OK. GoogleTest requires signing a Contributor License Agreement for people who want to submit patches upstream to them. That's no different from Kyua. That's a bit annoying, but maybe something we can live with. So I have no problem with GoogleTest on those points. My questions for you are: (1) GoogleTest is very C++-focused. ATF has API's for C, C++, and shell. How successful will you be at testing a pile of C code with a C++-focused library? It is technically possible ( https://meekrosoft.wordpress.com/2009/11/09/unit-testing-c-code-with-the-go= ogletest-framework/ ) but what are the gotchas? (2) What do we do with existing tests? Do they need to be rewritten to the GoogleTest API? Or do we have infrastructure that allows existing ATF tests to run with Google Test? (3) If this is adopted, are new tests written in GoogleTest API, or in the ATF API? (4) Kyua is the tool currently used for running the tests, and providing test result output. Kyua has test-case isolation, and the potential to parallelize the execution of test-cases. Are the Kyua features of test-case isolation and parallelization of test-cases must-haves, or can we live without them? Do you want to move away from using Kyua, or modify it to work with GoogleTest? If you want to move away from Kyua, how do you want people to run the GoogleTests and get the test results? I have no objection to your proposal. Using a more mainstream testing framework with a larger user community, and some momentum to keep things going would be nice. -- Craig