From owner-freebsd-testing@FreeBSD.ORG Tue Nov 4 19:40:37 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0BA58BD for ; Tue, 4 Nov 2014 19:40:37 +0000 (UTC) Received: from mail-la0-x232.google.com (mail-la0-x232.google.com [IPv6:2a00:1450:4010:c03::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7F5292F2 for ; Tue, 4 Nov 2014 19:40:36 +0000 (UTC) Received: by mail-la0-f50.google.com with SMTP id hz20so1527905lab.9 for ; Tue, 04 Nov 2014 11:40:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=ySzfTUKqcYf2RF7keFx+sigWu60kj2vb2XGkfzu1xMg=; b=pCpcBDLPr4ubjNpuIwpc6+O+T1TTrLarCOeusR/t1RGeRXqfFQJauinK6tp+Tr50sF lPmix+pECutrMqacvp7bhfzsBbnY3x4E50Kmb9i/Q35HYl7kvIrfRiLztUWGh4M32ku4 FU1xxmdJ2rbh3rzRngggY3xvvgqrMkV9fIPDnd1twgjtOM4gmfCJe2Ynh+yJA3RruCmY CISdbeEHZVzuDXbMGJnalB0DImn5ua9U3+sAjqqOnfRwHrx63PaGIT4b/oVpDA2AuPit veTUOUsl2eZJLvXUA0ngp6HOOTvz+3NUl7/xhdWP082epY9zPMXnxEDY6GrGuKRHgagL Tjdg== MIME-Version: 1.0 X-Received: by 10.152.207.71 with SMTP id lu7mr19594781lac.81.1415130034629; Tue, 04 Nov 2014 11:40:34 -0800 (PST) Sender: crodr001@gmail.com Received: by 10.112.130.168 with HTTP; Tue, 4 Nov 2014 11:40:34 -0800 (PST) In-Reply-To: References: Date: Tue, 4 Nov 2014 11:40:34 -0800 X-Google-Sender-Auth: GChM0YcXm6dKCzsmF02g9NLKO4A Message-ID: Subject: Re: Kyua/ATF as a test framework discussion From: Craig Rodrigues To: "freebsd-testing@freebsd.org" Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Nov 2014 19:40:37 -0000 On Tue, Nov 4, 2014 at 10:38 AM, Garrett Cooper wrote: > There is a key component missing from kyua that allows it to run > functional tests (both on FreeBSD and in general). In particular it does > not have per test program setup/cleanup hooks that other frameworks do > (cmake, JUnit, LTP, python unittest), and it does not have a per-testcase > setup hook. > This is something that I called out several years ago when I first looked > at importing ATF into the base tree (see the "Known Issues" section in > https://wiki.freebsd.org/TestingFreeBSD ). Not having setup/cleanup hooks > has made some testing at EMC/Isilon painful because of how some testers > have written tests, in particular there are testcases that were added to > functionally test out mountd, mount_nfs, syslogd, etc that I had to disable > because of the unnecessary complexity involved in trying to manage state in > setting up/tearing down daemons, running syslogd, etc, and because of the > external state involved the testcases would work sometimes if run > end-to-end, not other times if executed one-by-one, and not other times if > run end-to-end. > > I've raised this issue on kyua-discuss and Julio is not in favor of it: https://groups.google.com/forum/#!msg/kyua-discuss/6DmYvtLN66I/MuG528I1V1AJ I agree with you and disagree with Julio. In some cases, having the hooks in the test framework to do setup/cleanup outside of the test itself is useful. My experience is that people who write tests don't always write perfect code, and do not always do proper setup/teardown inside the testcase. In FreeBSD and NetBSD, we can enforce this and force people to fix the tests. In large products based on FreeBSD, my experience is that this is not often possible in real-world QA environments. Having hooks in the test framework to do setup/teardown like in other unit test frameworks would be useful. -- Craig