From owner-freebsd-testing@FreeBSD.ORG Wed Nov 5 21:41:39 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 47CC764C for ; Wed, 5 Nov 2014 21:41:39 +0000 (UTC) Received: from mail-qg0-f53.google.com (mail-qg0-f53.google.com [209.85.192.53]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 062E8B95 for ; Wed, 5 Nov 2014 21:41:38 +0000 (UTC) Received: by mail-qg0-f53.google.com with SMTP id z107so22146qgd.40 for ; Wed, 05 Nov 2014 13:41:32 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=pl4ClTB4I6q7PIF/N3IK/YFphKplo5J4r2TevsyDMqs=; b=lLWanwkbXB4KetgmfW4RJSynBo0GvrLX5YOU5OVDkQfzLRc0YxNcbXj/5E6dfMDnUe agpUoV6W71kml6MeU9Wi7Z54B14bFzw/ZsFg9YVe/0rcYNvu8JDZlMcB01nSosbjIPlE +D0IQ3Ej5jOUHW1pewugwDPGSATiwk5LFhniLmnF1HsyM9G1Lek5SuA2wZ4MbCjHj6r6 eo8kKpE29ATPN3DSMP4LOB9W41uSJW8CDFvkyw4FDaMuopkb1EQlrIVfBT2Vh+x8zceW kWpmPvgXvKfgZ0VcVHd3jtA4E3iwhInuwXa7JyG18sUUcFfE9Fvy14RmQTd5OQ6s4oRD tpVg== X-Gm-Message-State: ALoCoQkW16jCGCIzi9QwnfzMJBA7OMDo69GsLXZVA1B8P4Q+yrGsMXOfA7IIC/QXy2c3EmdZvvB4 X-Received: by 10.140.108.182 with SMTP id j51mr41345qgf.27.1415223692304; Wed, 05 Nov 2014 13:41:32 -0800 (PST) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.67.69 with HTTP; Wed, 5 Nov 2014 13:41:12 -0800 (PST) X-Originating-IP: [2620:0:1003:1007:40f9:7510:7feb:6981] In-Reply-To: References: From: Julio Merino Date: Wed, 5 Nov 2014 16:41:12 -0500 X-Google-Sender-Auth: YPhXWe8Y34rB9oah8bUz2Xe6UhY Message-ID: Subject: Re: Kyua/ATF as a test framework discussion To: Garrett Cooper Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 21:41:39 -0000 On Tue, Nov 4, 2014 at 1:38 PM, Garrett Cooper wrot= e: > (developers BCCed; please continue discussion on -testing@) > Hi all, > > A few questions came up yesterday at the BSD Vendor Summit in the= Q&A session after rodrigc=E2=80=99s talk about Kyua/ATF related to testing= on FreeBSD, as related to the testing framework (there are a handful of ot= her questions I=E2=80=99ll bring up in other threads ;)..). In particular, = some of the questions that stood out in my mind were: > > 1. What type of testing does Kyua do? Can it be used in benchmark= testing, stress testing, etc? > 2. Is there a structured way of checking for feature before r= unning my tests? > 3. Can I run tests with custom arguments, e.g. point something at= a filesystem path, etc? How do I do that? > > I=E2=80=99ll try to answer these questions based on experience an= d input I=E2=80=99ve received from several people both internal and externa= l to EMC/Isilon. > > Answer to 1. > > - Kyua/ATF excels as a unit testing framework and only does non-distribut= ed testing. > - Kyua/ATF should not be used in benchmark testing or stress testing. This is a fair description. > There is a key component missing from kyua that allows it to run = functional tests (both on FreeBSD and in general). In particular it does no= t have per test program setup/cleanup hooks that other frameworks do (cmake= , JUnit, LTP, python unittest), and it does not have a per-testcase setup h= ook. Well, this is specifically ATF's fault. The more I look at ATF and other testing libraries, the more I'm convinced that ATF's approach of forcing every test case be in a completely separate process is suboptimal because it prevents having fixtures altogether. If you write your own test program without ATF, then you can easily implement your own global setup and teardown routines. (In fact, the alternative library I'm playing with in https://github.com/jmmv/shtk/tree/unittest actually supports per-test setup/teardown and it'd be trivial to add per-program setup/teardown as well.) Of course, "write your own test program" is poor advice given that doing so from scratch, without support libraries, sucks. The problem is that I do not believe that the ATF interface is fixable to support these cases; we'd need to write a different library, and at that point... is it worth doing? What else is there? You should be able to easily plug existing, more advanced libraries like pyUnit or googletest into Kyua -- and if there is a problem with doing such integration, we should fix it. > Answer to 2. > > There isn=E2=80=99t currently a structured API for asking questio= ns like =E2=80=9Cis this driver loaded?", =E2=80=9Cis this sysctl set?=E2= =80=9D, =E2=80=9Cam I running on UFS/ZFS?=E2=80=9D, etc. In general test ca= ses from tools/regression, etc have invented homegrown logic for handling t= his, so this isn=E2=80=99t a new issue. Having such functionality would be great, but IMHO the majority of it is so tied to FreeBSD that it has to live in the FreeBSD source tree. > Answer to 3. > > Yes and no(-ish). A major goal when designing ATF back then was precisely to avoid variability in test execution: all test programs should behave the same way so that their execution can be scripted and so that the caller (user) needn't know how to run the test program (usually poorly documented). Aren't configuration variables passed through the environment sufficient? Kyua 0.11 should do that. (Note that you say above that #84 wasn't completed, but I'm not sure why.)