From owner-freebsd-testing@FreeBSD.ORG Wed Nov 5 23:46:11 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 15C3F18A for ; Wed, 5 Nov 2014 23:46:11 +0000 (UTC) Received: from mail-qc0-f172.google.com (mail-qc0-f172.google.com [209.85.216.172]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C5C13B99 for ; Wed, 5 Nov 2014 23:46:10 +0000 (UTC) Received: by mail-qc0-f172.google.com with SMTP id i17so1548722qcy.3 for ; Wed, 05 Nov 2014 15:46:09 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type; bh=UnbYiIlfF31xVHINULivBoyfoNmA8gz/CrJ6f5BcVtE=; b=e5+i7OTt8hrqrloCuRJXLEQA4xueCkrRTxgEl7yuCf3x5vyB6fCVd5eG1iYZwWhUWb X6E0ETcQ5j9jshjb2qPiHOvff6Vhew6I9lCgLQav+sup+38aWl31G3T5/godN4Gku2Ol RPm3XRRU0Db2KE+l/oB0E9kIR/+sRQJK6Kb0oYAkV6XQK2yzsQm5lSAKoYRBlRsls32K DcgaTi98ZDMbys8o1+xBLVEaKo5aaoO8jnnMWbaFfQMnVay+cndGC5FIGSryMuIHqDJt UwzVAA5iGPBDnyg2jKjbDdoVzUNtEdYNfktFRfF82JoUmFk4h8Ptv94t3CpUijZtRvl6 5WOA== X-Gm-Message-State: ALoCoQnQ06Xn+72/RH5A+4bN3E62SetY93p/JDf17XnkZL+1laA5CYTi0T7icJ41qaiuLLBdMouz X-Received: by 10.224.167.132 with SMTP id q4mr1117718qay.48.1415231169160; Wed, 05 Nov 2014 15:46:09 -0800 (PST) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.67.69 with HTTP; Wed, 5 Nov 2014 15:45:48 -0800 (PST) X-Originating-IP: [184.153.117.174] In-Reply-To: References: From: Julio Merino Date: Wed, 5 Nov 2014 18:45:48 -0500 X-Google-Sender-Auth: 4yOOYgbfbBxVv0Rx_CWrWMKXsIw Message-ID: Subject: Re: Kyua/ATF as a test framework discussion To: Craig Rodrigues Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 23:46:11 -0000 On Wed, Nov 5, 2014 at 5:02 PM, Craig Rodrigues wrote: > > > On Wed, Nov 5, 2014 at 1:41 PM, Julio Merino wrote: >> >> On Tue, Nov 4, 2014 at 2:40 PM, Craig Rodrigues >> wrote: >> > Having hooks in the test framework to do setup/teardown like in other >> > unit >> > test frameworks would be useful. >> >> Please tell me of a test framework that has setup/teardown >> functionality outside of the test program. I would like to see how >> they handle it. > > http://nose.readthedocs.org/en/latest/writing_tests.html#test-functions > https://docs.python.org/3/library/unittest.html#unittest.TestCase.setUp > http://cunit.sourceforge.net/doc/managing_tests.html > http://junit.sourceforge.net/junit3.8.1/javadoc/junit/framework/TestCase.html#method_summary All of these examples point at *in-program* setup/teardown methods, not what you were describing a while ago with hooks in the Kyuafile. Adding per-test case setup/teardown methods to ATF would be easy, but mostly pointless except for maybe in atf-c++ (but I don't think anyone cares about atf-c++). Adding per-test program setup/teardown methods to ATF is interesting, but difficult due to the semantics of the interface of the test programs. Also, the resulting API would be quite hard to handle and understand from a programmer's perspective... See my other mail in this thread. Basically, I believe the current ATF interface is not worth fixing to support these cases and we should look at other existing, unit-testing libraries to see if they fit our needs. Kyua should be the glue to bring them together. In fact, you should be able to use all the libraries you mentioned with Kyua already (or at least that's the goal); and, if not, that's a bug that we should fix.