From owner-freebsd-testing@FreeBSD.ORG Thu Nov 6 02:02:20 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B94C1AA4; Thu, 6 Nov 2014 02:02:20 +0000 (UTC) Received: from mail-lb0-x232.google.com (mail-lb0-x232.google.com [IPv6:2a00:1450:4010:c04::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 236C7C45; Thu, 6 Nov 2014 02:02:20 +0000 (UTC) Received: by mail-lb0-f178.google.com with SMTP id f15so137488lbj.9 for ; Wed, 05 Nov 2014 18:02:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=pr2LSFw5kFPy4XeuuzWa988o/eESHFJ0K3rmnw1HQ4U=; b=VQ468+/fUO1bHjNn49Kvq4mAhnf85fuC2FohcpejtBETo+c/QhSOs24//o80wAU5c8 ehGqsuJ28vmYb5KCVZAhtq318+f77nvLzwBdSRXJkH1Q7L30p4290adcDTDy8LWPesgd kitWObEdOWEP028V2hJVHFDJqOxGSoLltx9Q/GHFVezbV608ToeXSk3zY+nIoYpQFAtJ sAp20L2MS0HX1FQu0BXY6htpWFzbiVC1OR8pv0PnWYGDVYRyToMCAZnC3iocgHYi+TVy rqtncOrFbKH66LirQl4VzuxT8o1gZdaQuKCcrumlm5Oim8kK0TWj/78fxDLsclR137zO Le8Q== MIME-Version: 1.0 X-Received: by 10.152.198.199 with SMTP id je7mr1137245lac.25.1415239338210; Wed, 05 Nov 2014 18:02:18 -0800 (PST) Sender: crodr001@gmail.com Received: by 10.112.130.168 with HTTP; Wed, 5 Nov 2014 18:02:18 -0800 (PST) In-Reply-To: References: Date: Wed, 5 Nov 2014 18:02:18 -0800 X-Google-Sender-Auth: Zk_HFx5AWtiKgqSyMkYP1bLwtfU Message-ID: Subject: Re: Kyua/ATF as a test framework discussion From: Craig Rodrigues To: Julio Merino Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Nov 2014 02:02:20 -0000 On Wed, Nov 5, 2014 at 3:45 PM, Julio Merino wrote: > See my other mail in this thread. Basically, I believe the current ATF > interface is not worth fixing to support these cases and we should > look at other existing, unit-testing libraries to see if they fit our > needs. Kyua should be the glue to bring them together. In fact, you > should be able to use all the libraries you mentioned with Kyua > already (or at least that's the goal); and, if not, that's a bug that > we should fix. > I'm confused by your e-mail. Are you not interested in fixing/enhancing ATF, and want to move to "something else"? Is "something else" an incremental improvement on ATF, or something completely different and incompatible? After Garrett's recent work to import NetBSD tests into FreeBSD, we now have over 3000 tests in FreeBSD, many of which are ATF tests. I hope to see people writing more tests using ATF, but they will not be inclined to do so if it is perceived as a dead end. Also, keep in mind, we have a few companies building BSD-based products who have started using ATF and have some interest in it. kyua as it exists right now is mostly OK, and the "kyua + Jenkins" integration has been working out quite well, at least for jenkins.freebsd.org. At the recent FreeBSD Devsummit, I presented what I did: https://wiki.freebsd.org/201411DevAndVendorSummit?action=AttachFile&do=view&target=kyua_jenkins.pdf There was a lot of interest in that work. There are a few companies that want to look at that and use it as a model for testing their own products which are based on BSD. Integrating non-ATF test frameworks in kyua is not seamless right now. If I treat each non-ATF test framework as a "plain_test_program" in kyua, that works mostly OK. However, with "plain_test_program", I don't get the ability to have multiple distinct test cases in a single program, like I do with ATF, which are visible via "kyua list, kyua debug", etc. Improving things in this area is doable, for example taking Alan's Python unittest backend to kyua ( https://lists.freebsd.org/pipermail/freebsd-testing/2013-November/000104.html ) and polishing it up. Also, with "plain_test_program", if I want custom setup/teardown, I need to implement that setup/teardown inside the test program. I don't always want that, and being able to maybe specify the custom setup/teardown routines in the Kyuafile would actually be useful. ATF and Kyua are not perfect, but they are actually very, very good tools, and are taking BSD in a very good direction with respect to testing. Fixing some of the issues in ATF, and making Kyua integrate better with other frameworks will only make it better. I hope you are still interested in doing some of these things. -- Craig