From owner-freebsd-testing@FreeBSD.ORG Sun Nov 2 13:56:25 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C0FBB9EE for ; Sun, 2 Nov 2014 13:56:25 +0000 (UTC) Received: from zhost.uxsglobal.com (zhost.xinuos.com [144.202.236.63]) by mx1.freebsd.org (Postfix) with ESMTP id 94B528C0 for ; Sun, 2 Nov 2014 13:56:24 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by zhost.uxsglobal.com (Postfix) with ESMTP id 9B1108C0134; Sun, 2 Nov 2014 08:56:17 -0500 (EST) Received: from zhost.uxsglobal.com ([127.0.0.1]) by localhost (zhost.uxsglobalhost.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kLz2Gj6h6m8t; Sun, 2 Nov 2014 08:56:17 -0500 (EST) Received: from zhost.uxsglobal.com (localhost [127.0.0.1]) by zhost.uxsglobal.com (Postfix) with ESMTP id 50B588C01A8; Sun, 2 Nov 2014 08:56:17 -0500 (EST) Received: from zhost.uxsglobalhost.com (localhost [127.0.0.1]) by zhost.uxsglobal.com (Postfix) with ESMTP id 43CFC8C0134; Sun, 2 Nov 2014 08:56:17 -0500 (EST) Date: Sun, 2 Nov 2014 08:56:17 -0500 (EST) From: Susan Stanziano To: freebsd-testing@freebsd.org Message-ID: <33368816.1215676.1414936577136.JavaMail.zimbra@xinuos.com> In-Reply-To: <1719328169.1212776.1414875717703.JavaMail.zimbra@xinuos.com> Subject: testing with FreeBSD and Kyua MIME-Version: 1.0 X-Originating-IP: [71.251.33.37] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - GC38 (Win)/8.0.6_GA_5922) Thread-Topic: testing with FreeBSD and Kyua Thread-Index: Gh6XpMwJwuXqK3/EaGp84t3c1CqZuA== Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: nextgen X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Nov 2014 13:56:25 -0000 Hi Craig, et al., I'm just finding my way through the FreeBSD testing environment so let me know if this is not the right forum for these questions/observations: (1) I have 10.1-RC3 installed and have run vmrun.sh a number of times. I notice the following failure on subsequent invocations: vm_open: Invalid argument We've seen this error here before in running our UnixWare714+ under bhyve and the development engineer here has addressed this issue by adding a one second sleep after destroying a vm instance. (2) If I attempt to run two VM's under Bhyve at the same time, I see a vm_run error and the first VM is destroyed: vm_run error -1, errno 6 + bhyve_exit=1 + [ 1 -ne 0 ] + break + /usr/sbin/bhyvectl --vm=freebsd-test --destroy + exit 1 I know we are allowing multiple VM's in our UnixWare714+ under bhyve environment and was wondering if this is supposed to be allowed in FreeBSD. (3) After running the kyua suite ("kyua test" command), I have output indicating 4 failures. I have gone to the https://bugs.freebsd.org/bugzilla/ site and searched for the failures I've seen to see if they have been previously reported. I've seen: -- bin/kill/grep-j_test:main - Bug 191019 already reported -- l couldn't find bug reports for these: -- local/atf/atf-c/macros_test:use -> failed: Build of macros_h_test.c failed; some macros in atf-c/macros.h are broken [0.022s] -- local/kyua/testers/run_test:fork_wait__unprivileged_group -> failed: testers/run_test.c:600: Subprocess is still running with GID set to root [0.010s] -- usr.bin/yacc/legacy_test:main -> failed: 15 tests of 15 failed [0.150s] Is there some place that I can see more details on the actual failures from Kyua? I ran "kyua report" and redirected that into a file and that is all the information I have. Thanks for any advice/pointers/corrections, Susan From owner-freebsd-testing@FreeBSD.ORG Sun Nov 2 22:01:16 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 956F2C8D for ; Sun, 2 Nov 2014 22:01:16 +0000 (UTC) Received: from mail-la0-x235.google.com (mail-la0-x235.google.com [IPv6:2a00:1450:4010:c03::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1B7C1AE1 for ; Sun, 2 Nov 2014 22:01:15 +0000 (UTC) Received: by mail-la0-f53.google.com with SMTP id mc6so8453177lab.40 for ; Sun, 02 Nov 2014 14:01:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=Ei+mKmy8m93q+oOB74gm03hy+hmmpdh8EE3Wpp/Um3U=; b=Be5S8yq6RCDePSLZCAcUypl6rOQlNeFNryIpST0a/DmihX24k2aPAihAHWTIyd1GgX 6M+0SwfU3lIINxMPnMiJzzg9UICQD9LctlVpbzN1ceXPiwQw9rmWf9W2KF/9ePOSMxmD ccFzZlqHcYQQrvC4aRGzp85XyXZ5biI3SwSo0L9OsJVVJ0MbqCeYlPof2PBufXrzhs7o OlfBjJ4q+5KwDu5CaphyrG8oYUFLmkZwPGyxTl1J8NwL1DWgd45EJlQrPrSKixBtikRz B9ipJs3WyP4XlskfKu7UVOTZDVv0jyNfWrLlXnZFTeOm8LomSeHcvfvPArWbHKyitMaE pRqg== MIME-Version: 1.0 X-Received: by 10.152.120.199 with SMTP id le7mr23297440lab.67.1414965673976; Sun, 02 Nov 2014 14:01:13 -0800 (PST) Sender: crodr001@gmail.com Received: by 10.112.84.197 with HTTP; Sun, 2 Nov 2014 14:01:13 -0800 (PST) In-Reply-To: <33368816.1215676.1414936577136.JavaMail.zimbra@xinuos.com> References: <1719328169.1212776.1414875717703.JavaMail.zimbra@xinuos.com> <33368816.1215676.1414936577136.JavaMail.zimbra@xinuos.com> Date: Sun, 2 Nov 2014 14:01:13 -0800 X-Google-Sender-Auth: T35SFoT_JvTUZadf0umj7-ujOS0 Message-ID: Subject: Re: testing with FreeBSD and Kyua From: Craig Rodrigues To: Susan Stanziano Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-testing@freebsd.org" , nextgen X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Nov 2014 22:01:16 -0000 On Sun, Nov 2, 2014 at 5:56 AM, Susan Stanziano wrote: > (1) I have 10.1-RC3 installed and have run vmrun.sh a number of times. I > notice the following failure on subsequent invocations: > > vm_open: Invalid argument > What is the full command that you are using to invoke vmrun.sh? (2) If I attempt to run two VM's under Bhyve at the same time, I see a > vm_run error and the first VM is destroyed: > It is possible to run two bhyve VM's at the same time but you must use a different for each invocation of the vmrun.sh script, so you would have to use freebsd-test1, freebsd-test2, etc. > -- local/atf/atf-c/macros_test:use -> failed: Build of macros_h_test.c > failed; some macros in atf-c/macros.h are broken [0.022s] > I think this might be fixed by: https://lists.freebsd.org/pipermail/svn-src-stable/2014-September/025633.html You may need a newer 10.1-RC version to get that fix. > > > > > > > -- local/kyua/testers/run_test:fork_wait__unprivileged_group -> failed: > testers/run_test.c:600: Subprocess is still running with GID set to root > [0.010s] > I think that might be fixed by: https://lists.freebsd.org/pipermail/svn-src-stable/2014-October/025688.html You may need a newer 10.1-RC version to get that fix. > > > > > > -- usr.bin/yacc/legacy_test:main -> failed: 15 tests of 15 failed [0.150s] > The yacc problems were reported here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193499 The fix is not yet in 10.1. > > Is there some place that I can see more details on the actual failures > from Kyua? I ran "kyua report" and redirected that into a file and that is > all the information I have. > You can try using the "kyua report-junit" or "kyua report-html" commands which produce more detailed reports. kyua 0.11 (which just was released last week) has kyua report --verbose, but the "--verbose" flag is not in kyua 0.10 which is what you are using. -- Craig From owner-freebsd-testing@FreeBSD.ORG Mon Nov 3 07:59:12 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A71994BA; Mon, 3 Nov 2014 07:59:12 +0000 (UTC) Received: from mail-pa0-x22b.google.com (mail-pa0-x22b.google.com [IPv6:2607:f8b0:400e:c03::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 69DF58B3; Mon, 3 Nov 2014 07:59:12 +0000 (UTC) Received: by mail-pa0-f43.google.com with SMTP id eu11so11717571pac.2 for ; Sun, 02 Nov 2014 23:59:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=1Eeqs9VgvMYGGnTp3Ko9PwvsolP+m+ImwHkfhHRExf4=; b=uizNjDQBaaCNLiFVNOiWurJbdn/A7jHYNembiLFunEWYgquG3R8UgXsGOm4awEV4i1 36FATTZIb4m/GQexdaUIZE0hKAzQ363VCA/La+wNGKO/Q0gx7HcdyH/ACSZQUyuGr49N 8eykDggpYGTVSo8tzx29megkxeBNdXWuHcu8dTQ9YvneC7aMavhnIVmJWlpWPcZqmzHG usbA6XSZTUG1+fKpHExc5iuhqRiB3ksvliNp9SmMCeimtz+LxqIiDhnqxEx+HrV+PyVZ bRfeXBH7rfT1twHvCStqZiPtCBHRrNBK19DljbBP9hDKAz27hySzgdhZpMmYP+8E6anQ LLcA== X-Received: by 10.66.182.130 with SMTP id ee2mr41814355pac.22.1415001551916; Sun, 02 Nov 2014 23:59:11 -0800 (PST) Received: from [172.16.0.156] (50-206-19-250-static.hfc.comcastbusiness.net. [50.206.19.250]) by mx.google.com with ESMTPSA id z4sm16410872pda.23.2014.11.02.23.59.11 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 02 Nov 2014 23:59:11 -0800 (PST) Content-Type: multipart/signed; boundary="Apple-Mail=_9EABB7B8-9CF2-4321-9652-AFD944299521"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Help with Bhyve VM install From: Garrett Cooper In-Reply-To: Date: Sun, 2 Nov 2014 23:59:11 -0800 Message-Id: References: <1332647925.1184837.1414700255905.JavaMail.zimbra@xinuos.com> <926516146.1185082.1414701141476.JavaMail.zimbra@xinuos.com> To: Craig Rodrigues X-Mailer: Apple Mail (2.1878.6) Cc: "freebsd-testing@freebsd.org" , Susan Stanziano , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Nov 2014 07:59:12 -0000 --Apple-Mail=_9EABB7B8-9CF2-4321-9652-AFD944299521 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Hi Craig, On Oct 30, 2014, at 19:39, Craig Rodrigues wrote: > On Thu, Oct 30, 2014 at 1:32 PM, Susan Stanziano = > wrote: >=20 >>=20 >> I have a system installed with releng10.1 (with tests configured) and = am >> trying to create a VM under Bhyve. >>=20 >> I want to know which image to use for the VM install, specifically = will >> the : >>=20 >> FreeBSD-10.1-RC3-amd64-bootonly.iso >>=20 >> install with the /usr/tests directory created? Or, should I use >> FreeBSD-10.1-RC3-amd64-disc1.iso ? >>=20 >=20 > Hi Susan, >=20 > I'm not sure why Garrett felt it was necessary to redirect you > to the freebsd-virtualization list. Although you are using bhyve, > there is nothing specific to bhyve in your questions. No harm done, > though, since I'm on both lists, and will answer your questions. Thank you for CCing -virtualization. The reason why I requested that Susan email that list directly is that = this email (and the new email thread that followed it, to a lesser = degree) are asking BHyVE specific questions, so I thought that it would = be best for Susan to email the appropriate mailing list, so the right = parties are involved in helping out with her support questions, to make = it easy to search for solutions to this issue (or questions similar to = this) in the future, and to make sure that folks on -testing aren=92t = emailed with -virtualization questions (I use VMware, not BHyVE, so the = questions/answers are less pertinent to me at least). > FreeBSD 10.1 does not have /usr/tests populated by default. =85 Could you please document the steps you provided in the reply on the = wiki? Thank you! -Garrett --Apple-Mail=_9EABB7B8-9CF2-4321-9652-AFD944299521 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUVzXPAAoJEMZr5QU6S73evtMH/2YrMN3paLmdNnBk3YecTk6f AK/aRjFca3HrEwNq3w7FtW76sCXUgT8IEje3tDpX6L0j5uebwH/6TR+rkTR464lJ CNi7INjoeRv2DIHYSL0QjWvYNeELgGMQJWTcDPq0uJBR08kuJt1jFMZogKu+4L7+ aoC792pK+THTfEkGbf3tZqlN6MMYf+7K6oQvbLxVol0gRWCjXmLGG5YCi0ryHB6e iDhU9iBwICqnUFHGLcLygR2IeOdFTp5itnoYYo7gTtn/y9Z788eeNwtnOdeBWzGL qdzM2SkIEJ/Z3CbjESrk5avBqshvqtaG3gaYi/Q6gUT2QhX9xqSgAGfxwOWFlig= =cwdS -----END PGP SIGNATURE----- --Apple-Mail=_9EABB7B8-9CF2-4321-9652-AFD944299521-- From owner-freebsd-testing@FreeBSD.ORG Mon Nov 3 19:25:20 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1F7446C7; Mon, 3 Nov 2014 19:25:20 +0000 (UTC) Received: from zhost.uxsglobal.com (zhost.xinuos.com [144.202.236.63]) by mx1.freebsd.org (Postfix) with ESMTP id E80A3DB8; Mon, 3 Nov 2014 19:25:19 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by zhost.uxsglobal.com (Postfix) with ESMTP id DAA638C02C0; Mon, 3 Nov 2014 14:25:11 -0500 (EST) Received: from zhost.uxsglobal.com ([127.0.0.1]) by localhost (zhost.uxsglobalhost.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fg-eNfl0YnFZ; Mon, 3 Nov 2014 14:25:11 -0500 (EST) Received: from zhost.uxsglobal.com (localhost [127.0.0.1]) by zhost.uxsglobal.com (Postfix) with ESMTP id BA6178C0355; Mon, 3 Nov 2014 14:25:11 -0500 (EST) Received: from zhost.uxsglobalhost.com (localhost [127.0.0.1]) by zhost.uxsglobal.com (Postfix) with ESMTP id B5C908C02C0; Mon, 3 Nov 2014 14:25:11 -0500 (EST) Date: Mon, 3 Nov 2014 14:25:11 -0500 (EST) From: Susan Stanziano To: rodrigc@freebsd.org Message-ID: <471635903.1234347.1415042711588.JavaMail.zimbra@xinuos.com> In-Reply-To: <1353133227.1229761.1415032347171.JavaMail.zimbra@xinuos.com> Subject: Bhyve VM install follow-up MIME-Version: 1.0 X-Originating-IP: [96.234.37.238] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - GC38 (Win)/8.0.6_GA_5922) Thread-Topic: Bhyve VM install follow-up Thread-Index: L7rte7lm3+I0ZXsQqckHH1sOFSeAnQ== Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-testing@freebsd.org X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Nov 2014 19:25:20 -0000 Hi Craig, Regarding the vm_open issue, vm_open: Invalid argument You asked: What is the full command that you are using to invoke vmrun.sh? I used simply "vmrun.sh " as root Here is a portion of the scrolled output from above command: + BOOTDISK=./diskdev + installer_opt='' + /usr/sbin/bhyveload -c stdio -m 512M -d ./diskdev freebsd-test vm_open: Invalid argument + bhyve_exit=1 + [ 1 -ne 0 ] + break + /usr/sbin/bhyvectl --vm=freebsd-test --destroy + exit 1 As regards the multiple VM's, I have tried this again successfully so your comment about invocations using different names solved the problem - my error. One more question, regarding the other failures for which there are now fixes - How do I match a fix to an RC version? Thanks for your time, Susan From owner-freebsd-testing@FreeBSD.ORG Mon Nov 3 22:49:05 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2648BCD6 for ; Mon, 3 Nov 2014 22:49:05 +0000 (UTC) Received: from mail-lb0-x230.google.com (mail-lb0-x230.google.com [IPv6:2a00:1450:4010:c04::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A002E794 for ; Mon, 3 Nov 2014 22:49:04 +0000 (UTC) Received: by mail-lb0-f176.google.com with SMTP id z11so7995212lbi.7 for ; Mon, 03 Nov 2014 14:49:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=Hs9OltX4/6/n4i4ni00KK2gWXVJ/ROZpUjj5Uia/F7Y=; b=iX90QzWhbbij5YTyqVAx/TjB+eEQsoXCgcPMqKXCrtHwG66YwW5yDLyme+MyklOZns n46B9ZXa5HSqqRG2W2ggVAC+GiPmbSOOlBE0ZWnK5R1V+zlbGCJIxVsiDd6rbO9xXKRJ HnHNdA06+9HVPIXHIc2G/NiIzXLob2IkECZHjtWzTtg3IywQ6eXUAbOn+S/tNJ3KRP/h JiULyshv6cDdJAN+2PQYfz7kiPgUhYAvU0COM+FnksWUe4fRzYLEvQ+d+qBcXKUf8q/K mKsCgz3lFhEnW2RKzdjaL166foRcvjr5q/wUWyAWGG2abP2xLY6OxUmliMu04hh+Hb3w sBKg== MIME-Version: 1.0 X-Received: by 10.152.22.135 with SMTP id d7mr54234030laf.46.1415054942471; Mon, 03 Nov 2014 14:49:02 -0800 (PST) Sender: crodr001@gmail.com Received: by 10.112.130.168 with HTTP; Mon, 3 Nov 2014 14:49:02 -0800 (PST) In-Reply-To: <471635903.1234347.1415042711588.JavaMail.zimbra@xinuos.com> References: <1353133227.1229761.1415032347171.JavaMail.zimbra@xinuos.com> <471635903.1234347.1415042711588.JavaMail.zimbra@xinuos.com> Date: Mon, 3 Nov 2014 14:49:02 -0800 X-Google-Sender-Auth: NWh0lHbb7LeWAv_XkNk8jVe4bWw Message-ID: Subject: Re: Bhyve VM install follow-up From: Craig Rodrigues To: Susan Stanziano Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Nov 2014 22:49:05 -0000 On Mon, Nov 3, 2014 at 11:25 AM, Susan Stanziano wrote: > Hi Craig, > > Regarding the vm_open issue, > > vm_open: Invalid argument > > You asked: > > What is the full command that you are using to invoke > vmrun.sh? > > I used simply "vmrun.sh " as root > > Here is a portion of the scrolled output from above command: > > + BOOTDISK=./diskdev > + installer_opt='' > + /usr/sbin/bhyveload -c stdio -m 512M -d ./diskdev freebsd-test > vm_open: Invalid argument > + bhyve_exit=1 > + [ 1 -ne 0 ] > + break > + /usr/sbin/bhyvectl --vm=freebsd-test --destroy > + exit 1 > > > OK, glad you got it to work. What I see right away is that if you do not specify the options: (1) diskdev is the default image used. If you have multiple VM's running, it probably is not safe to use the same disk image across multiple VM's, so you might want to have different disk images, and specify the -d flag. > > As regards the multiple VM's, I have tried this again successfully so > your comment about invocations using different names solved the problem - > my error. > > > One more question, regarding the other failures for which there are now > fixes - How do I match a fix to an RC version? > > That's a very good question. So basically what I do is, first I look at: https://www.freebsd.org/doc/en/articles/committers-guide/subversion-primer.html#svn-getting-started-base-layout to understand the layout of the different branches in FreeBSD. Then I go to: https://svnweb.freebsd.org to look at the commits. The commits targeted for the 10.1 release are going into the /base/releng/10.1 branch. If you are familiar with Subversion, you can also checkout the code from subversion and do "svn log" to look the commits, as an alternative to the web interface. -- Craig From owner-freebsd-testing@FreeBSD.ORG Tue Nov 4 18:38:07 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1C3E2C80; Tue, 4 Nov 2014 18:38:07 +0000 (UTC) Received: from mail-pa0-x22a.google.com (mail-pa0-x22a.google.com [IPv6:2607:f8b0:400e:c03::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DD709B22; Tue, 4 Nov 2014 18:38:06 +0000 (UTC) Received: by mail-pa0-f42.google.com with SMTP id bj1so14967018pad.15 for ; Tue, 04 Nov 2014 10:38:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:date:subject:to:message-id:mime-version; bh=DqarS2Vjwo1bOdflPQZFp3JNKBxa5WGtb45nvF+eyS0=; b=gTg1C5TNRS2qJKUiJbzAIXDfU0rXG8dpIkdLCGEs1iKdOEil14WvQ1Z0RQUoA1CBPX HdqJ8qSXNV32ecxeE9tU+py+sJWWs0q4UvtQuSRaK8Ck+UusMS1IO94dMR9Vy2CZmkwk TIcUZbqT7PB3lQbhYhv1ZJeDcziDwofRw6CvCUnOVEiBv1mT00US4l5IsgSu+aDs1uue tDsPqKvIoAhuk8mXRGUKpuEF9ZK1vlQNAAPPEn2FwevtEEagEXGlUd5B4JS6Lt/nfbUV Hfootb2a2lGNNGoJOTpe0pqLzAiSdUP+XlKDQyCrq9RDQKLZZEwSweoZguLYMzDZHBym iwRA== X-Received: by 10.68.100.69 with SMTP id ew5mr52409019pbb.84.1415126286494; Tue, 04 Nov 2014 10:38:06 -0800 (PST) Received: from [172.16.0.156] (50-206-19-250-static.hfc.comcastbusiness.net. [50.206.19.250]) by mx.google.com with ESMTPSA id wa3sm1073504pac.18.2014.11.04.10.38.05 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 04 Nov 2014 10:38:05 -0800 (PST) From: Garrett Cooper Content-Type: multipart/signed; boundary="Apple-Mail=_5C25921D-35F7-462F-B10A-B36F2089D9EC"; protocol="application/pgp-signature"; micalg=pgp-sha512 Date: Tue, 4 Nov 2014 10:38:03 -0800 Subject: Kyua/ATF as a test framework discussion To: "freebsd-testing@freebsd.org" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) X-Mailer: Apple Mail (2.1878.6) X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Nov 2014 18:38:07 -0000 --Apple-Mail=_5C25921D-35F7-462F-B10A-B36F2089D9EC Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 (developers BCCed; please continue discussion on -testing@) Hi all, A few questions came up yesterday at the BSD Vendor Summit in = the Q&A session after rodrigc=92s talk about Kyua/ATF related to testing = on FreeBSD, as related to the testing framework (there are a handful of = other questions I=92ll bring up in other threads ;)..). In particular, = some of the questions that stood out in my mind were: 1. What type of testing does Kyua do? Can it be used in = benchmark testing, stress testing, etc? 2. Is there a structured way of checking for feature before = running my tests? 3. Can I run tests with custom arguments, e.g. point something = at a filesystem path, etc? How do I do that? I=92ll try to answer these questions based on experience and = input I=92ve received from several people both internal and external to = EMC/Isilon. Answer to 1. - Kyua/ATF excels as a unit testing framework and only does = non-distributed testing. - Kyua/ATF should not be used in benchmark testing or stress testing. There is a key component missing from kyua that allows it to run = functional tests (both on FreeBSD and in general). In particular it does = not have per test program setup/cleanup hooks that other frameworks do = (cmake, JUnit, LTP, python unittest), and it does not have a = per-testcase setup hook. This is something that I called out several = years ago when I first looked at importing ATF into the base tree (see = the =93Known Issues=94 section in = https://wiki.freebsd.org/TestingFreeBSD ). Not having setup/cleanup = hooks has made some testing at EMC/Isilon painful because of how some = testers have written tests, in particular there are testcases that were = added to functionally test out mountd, mount_nfs, syslogd, etc that I = had to disable because of the unnecessary complexity involved in trying = to manage state in setting up/tearing down daemons, running syslogd, = etc, and because of the external state involved the testcases would work = sometimes if run end-to-end, not other times if executed one-by-one, and = not other times if run end-to-end. Adding these hooks isn=92t impossible and should be the next = step in improving the test Kyua/ATF infrastructure (I ran out of time = before trying to implement this, and I=92m not a C++ whiz so I got lost = trying to figure out how to make everything work). Answer to 2. There isn=92t currently a structured API for asking questions = like =93is this driver loaded?", =93is this sysctl set?=94, =93am I = running on UFS/ZFS?=94, etc. In general test cases from = tools/regression, etc have invented homegrown logic for handling this, = so this isn=92t a new issue. I=92ve done some initial prototyping of a C/shell library for = testing for feature requirements on FreeBSD, but the work isn=92t = complete/in the tree yet (please see the mailing list threads below for = more details): - = https://lists.freebsd.org/pipermail/freebsd-testing/2014-June/000439.html - = https://lists.freebsd.org/pipermail/freebsd-testing/2014-July/000467.html I want to complete this sometime before/around the start of the = 2015, in particular because this blocks several of the testcases at = EMC/Isilon from being upstreamed in a good, working state. Answer to 3. Yes and no(-ish). For ATF tests: You can pass through custom arguments on the command line like = so: % kyua test -v mykey=3Dmyvalue This information is that passed through the atf APIs so you can = query the value with either atf-config or atf_tc_config (unfortunately = the mechanism I=92m used to differs enough from what is currently in = place, so I=92ll need to go look up what the current mechanism/structure = is for this, but basically it involves specifying variables kyua.conf). For non-ATF tests: You have to resort to using environment variables today, like I = did with pjdfstest: = https://svnweb.freebsd.org/base/head/tests/sys/pjdfstest/tests/conf?revisi= on=3D274016&view=3Dmarkup#l5 , as I described in the README file: = https://svnweb.freebsd.org/base/head/share/doc/pjdfstest/README?revision=3D= 274016&view=3Dmarkup . I=92d _really_ like to avoid doing this though = because then we=92ll have 40 some different ways to do things around the = tree instead of a single, supported way to do things. There was an enhancement I opened on GitHub to add = =93configuration integration=94 for the other test types (Plain, TAP) = [via environment variables or command querying], but that wasn=92t = completed: https://github.com/jmmv/kyua/issues/84 . I=92ll reopen the = discussion. I=92ll try to add/migrate relevant links for things noted in = this email on the wiki. Thank you! --Apple-Mail=_5C25921D-35F7-462F-B10A-B36F2089D9EC Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUWR0LAAoJEMZr5QU6S73eUiEH/1xeeonM4LbOCYkAWujkhZwA rgib/2fzwJnvxOi92lLLGupvRzXJE7Dy3Dru7+FsDXs9KQ1WIUyKdjbR7wi/mXAY kNSzsG8138Oe6oAc/ow2rQxW0mn6PZCA9GAmmZZ/qJ/5EDlGAqLJmiRDUPqL8AG1 m5HtLqFgHlozJyK5mpxrBCtjSOISCsF9oip6HSFT1AFFuNgZSHseUzVFnbpUrcJG NUvkrkQdOn1CeftpmFu41PtAD+tcDcfhlUpYDN8eRcne9Pnr2Ed+3p6z73qs0rMB fFVD5rBy8gd5WkpUvVaOifr238kanenA9UFaXYtIvgzoRbYgoWf4dSOKl2BLHRY= =POKQ -----END PGP SIGNATURE----- --Apple-Mail=_5C25921D-35F7-462F-B10A-B36F2089D9EC-- From owner-freebsd-testing@FreeBSD.ORG Tue Nov 4 19:16:34 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5420DB45 for ; Tue, 4 Nov 2014 19:16:34 +0000 (UTC) Received: from mail-wg0-x235.google.com (mail-wg0-x235.google.com [IPv6:2a00:1450:400c:c00::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D7027F4D for ; Tue, 4 Nov 2014 19:16:33 +0000 (UTC) Received: by mail-wg0-f53.google.com with SMTP id b13so14351679wgh.12 for ; Tue, 04 Nov 2014 11:16:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=jwgy1HQ96NpUOGb3MuV5iIJGChj0fPrIqJdSpA+do2g=; b=bvQw3FsJhFaopswVCzWaeF0Cj+YL8CQZaDuY48SIwCMRKNfgtK64vjMbAINrkYoHOH SNoER28Vg7Zgdx1dy5TM3taQUWEAuVDAnx6bBsPWIItmDV3UrqWo23fXhIBUAwN1+0Kl kQsvQvbj5vE/Q1yyRDNy1+eoh+jIjzJ3ke+IBhSIF9twJSHhLDS9wosNvQaQ1mXpNgy1 18Swn+mNqLO6mQkgF+gRlT6ewojHvV9YXtoAJQnKqCOQT4bUKTnfJ+XurOvch5sGk/yw Ztb21UlEsGgnePtqev/xPj0/p6Rx970fe3oQBnICmxF3pSrAiEX1gKO3v5MYdJX+hA1h TEBw== MIME-Version: 1.0 X-Received: by 10.194.87.131 with SMTP id ay3mr35429714wjb.66.1415128591974; Tue, 04 Nov 2014 11:16:31 -0800 (PST) Sender: asomers@gmail.com Received: by 10.194.220.227 with HTTP; Tue, 4 Nov 2014 11:16:31 -0800 (PST) In-Reply-To: References: Date: Tue, 4 Nov 2014 12:16:31 -0700 X-Google-Sender-Auth: lzy0Wi9oZ-BBe9kdvpRLZ_bW16I Message-ID: Subject: Re: Kyua/ATF as a test framework discussion From: Alan Somers To: Garrett Cooper Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Nov 2014 19:16:34 -0000 On Tue, Nov 4, 2014 at 11:38 AM, Garrett Cooper wro= te: > (developers BCCed; please continue discussion on -testing@) > Hi all, > > A few questions came up yesterday at the BSD Vendor Summit in the= Q&A session after rodrigc=E2=80=99s talk about Kyua/ATF related to testing= on FreeBSD, as related to the testing framework (there are a handful of ot= her questions I=E2=80=99ll bring up in other threads ;)..). In particular, = some of the questions that stood out in my mind were: > > 1. What type of testing does Kyua do? Can it be used in benchmark= testing, stress testing, etc? > 2. Is there a structured way of checking for feature before r= unning my tests? > 3. Can I run tests with custom arguments, e.g. point something at= a filesystem path, etc? How do I do that? > > I=E2=80=99ll try to answer these questions based on experience an= d input I=E2=80=99ve received from several people both internal and externa= l to EMC/Isilon. > > Answer to 1. > > - Kyua/ATF excels as a unit testing framework and only does non-distribut= ed testing. > - Kyua/ATF should not be used in benchmark testing or stress testing. > > There is a key component missing from kyua that allows it to run = functional tests (both on FreeBSD and in general). In particular it does no= t have per test program setup/cleanup hooks that other frameworks do (cmake= , JUnit, LTP, python unittest), and it does not have a per-testcase setup h= ook. This is something that I called out several years ago when I first loo= ked at importing ATF into the base tree (see the =E2=80=9CKnown Issues=E2= =80=9D section in https://wiki.freebsd.org/TestingFreeBSD ). Not having set= up/cleanup hooks has made some testing at EMC/Isilon painful because of how= some testers have written tests, in particular there are testcases that we= re added to functionally test out mountd, mount_nfs, syslogd, etc that I ha= d to disable because of the unnecessary complexity involved in trying to ma= nage state in setting up/tearing down daemons, running syslogd, etc, and be= cause of the external state involved the testcases would work sometimes if = run end-to-end, not other times if executed one-by-one, and not other times= if run end-to-end. > > Adding these hooks isn=E2=80=99t impossible and should be the nex= t step in improving the test Kyua/ATF infrastructure (I ran out of time bef= ore trying to implement this, and I=E2=80=99m not a C++ whiz so I got lost = trying to figure out how to make everything work). I partially disagree with this answer. While it's true that per-test program setup and teardown steps would be convenient, they are by no means required. Spectra Logic has many functional and stress tests that work just fine. Code duplication due to lack of per-test program setup and teardown steps is minimal. For example, see sys/netinet/fibs_test on a CURRENT system. The ZFS test suite ( base/projects/zfsd/head/tests/sys/cddl/zfs ) was adapted from STF, which uses per-test program setup and cleanups. But it turned out that it didn't really need them, and was using them wrongly as often as not. For example, many test cases could only be run in a certain order because of interactions in their side effects. I converted all of the setup and teardown steps to per-testcase. OTOH, I do agree that Kyua should not be used for benchmarking. Its reports are much too coarse. For one thing, a Kyua test can only pass or fail (or skip, expected fail, etc), whereas a benchmark normally returns one or more real numbers. For another, in order to compare benchmarks from different systems, it is necessary to collect a great deal of information about the configuration of those systems. How much RAM? What model of CPU? What were the network card's sysctl settings? Kyua doesn't try to do that, and I don't think that it ever should. > > Answer to 2. > > There isn=E2=80=99t currently a structured API for asking questio= ns like =E2=80=9Cis this driver loaded?", =E2=80=9Cis this sysctl set?=E2= =80=9D, =E2=80=9Cam I running on UFS/ZFS?=E2=80=9D, etc. In general test ca= ses from tools/regression, etc have invented homegrown logic for handling t= his, so this isn=E2=80=99t a new issue. > > I=E2=80=99ve done some initial prototyping of a C/shell library f= or testing for feature requirements on FreeBSD, but the work isn=E2=80=99t = complete/in the tree yet (please see the mailing list threads below for mor= e details): > > - https://lists.freebsd.org/pipermail/freebsd-testing/2014-June/000439.ht= ml > - https://lists.freebsd.org/pipermail/freebsd-testing/2014-July/000467.ht= ml > > I want to complete this sometime before/around the start of the 2= 015, in particular because this blocks several of the testcases at EMC/Isil= on from being upstreamed in a good, working state. > > Answer to 3. > > Yes and no(-ish). > > For ATF tests: > > You can pass through custom arguments on the command line like so= : > > % kyua test -v mykey=3Dmyvalue > > This information is that passed through the atf APIs so you can q= uery the value with either atf-config or atf_tc_config (unfortunately the m= echanism I=E2=80=99m used to differs enough from what is currently in place= , so I=E2=80=99ll need to go look up what the current mechanism/structure i= s for this, but basically it involves specifying variables kyua.conf). Also, you can put those settings in /usr/local/etc/kyua/kyua.conf instead of on the kyua command line. > > For non-ATF tests: > > You have to resort to using environment variables today, like I d= id with pjdfstest: https://svnweb.freebsd.org/base/head/tests/sys/pjdfstest= /tests/conf?revision=3D274016&view=3Dmarkup#l5 , as I described in the READ= ME file: https://svnweb.freebsd.org/base/head/share/doc/pjdfstest/README?re= vision=3D274016&view=3Dmarkup . I=E2=80=99d _really_ like to avoid doing th= is though because then we=E2=80=99ll have 40 some different ways to do thin= gs around the tree instead of a single, supported way to do things. > > There was an enhancement I opened on GitHub to add =E2=80=9Cconfi= guration integration=E2=80=9D for the other test types (Plain, TAP) [via en= vironment variables or command querying], but that wasn=E2=80=99t completed= : https://github.com/jmmv/kyua/issues/84 . I=E2=80=99ll reopen the discussi= on. > > I=E2=80=99ll try to add/migrate relevant links for things noted i= n this email on the wiki. > > Thank you! -Alan From owner-freebsd-testing@FreeBSD.ORG Tue Nov 4 19:40:37 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0BA58BD for ; Tue, 4 Nov 2014 19:40:37 +0000 (UTC) Received: from mail-la0-x232.google.com (mail-la0-x232.google.com [IPv6:2a00:1450:4010:c03::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7F5292F2 for ; Tue, 4 Nov 2014 19:40:36 +0000 (UTC) Received: by mail-la0-f50.google.com with SMTP id hz20so1527905lab.9 for ; Tue, 04 Nov 2014 11:40:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=ySzfTUKqcYf2RF7keFx+sigWu60kj2vb2XGkfzu1xMg=; b=pCpcBDLPr4ubjNpuIwpc6+O+T1TTrLarCOeusR/t1RGeRXqfFQJauinK6tp+Tr50sF lPmix+pECutrMqacvp7bhfzsBbnY3x4E50Kmb9i/Q35HYl7kvIrfRiLztUWGh4M32ku4 FU1xxmdJ2rbh3rzRngggY3xvvgqrMkV9fIPDnd1twgjtOM4gmfCJe2Ynh+yJA3RruCmY CISdbeEHZVzuDXbMGJnalB0DImn5ua9U3+sAjqqOnfRwHrx63PaGIT4b/oVpDA2AuPit veTUOUsl2eZJLvXUA0ngp6HOOTvz+3NUl7/xhdWP082epY9zPMXnxEDY6GrGuKRHgagL Tjdg== MIME-Version: 1.0 X-Received: by 10.152.207.71 with SMTP id lu7mr19594781lac.81.1415130034629; Tue, 04 Nov 2014 11:40:34 -0800 (PST) Sender: crodr001@gmail.com Received: by 10.112.130.168 with HTTP; Tue, 4 Nov 2014 11:40:34 -0800 (PST) In-Reply-To: References: Date: Tue, 4 Nov 2014 11:40:34 -0800 X-Google-Sender-Auth: GChM0YcXm6dKCzsmF02g9NLKO4A Message-ID: Subject: Re: Kyua/ATF as a test framework discussion From: Craig Rodrigues To: "freebsd-testing@freebsd.org" Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Nov 2014 19:40:37 -0000 On Tue, Nov 4, 2014 at 10:38 AM, Garrett Cooper wrote: > There is a key component missing from kyua that allows it to run > functional tests (both on FreeBSD and in general). In particular it does > not have per test program setup/cleanup hooks that other frameworks do > (cmake, JUnit, LTP, python unittest), and it does not have a per-testcase > setup hook. > This is something that I called out several years ago when I first looked > at importing ATF into the base tree (see the "Known Issues" section in > https://wiki.freebsd.org/TestingFreeBSD ). Not having setup/cleanup hooks > has made some testing at EMC/Isilon painful because of how some testers > have written tests, in particular there are testcases that were added to > functionally test out mountd, mount_nfs, syslogd, etc that I had to disable > because of the unnecessary complexity involved in trying to manage state in > setting up/tearing down daemons, running syslogd, etc, and because of the > external state involved the testcases would work sometimes if run > end-to-end, not other times if executed one-by-one, and not other times if > run end-to-end. > > I've raised this issue on kyua-discuss and Julio is not in favor of it: https://groups.google.com/forum/#!msg/kyua-discuss/6DmYvtLN66I/MuG528I1V1AJ I agree with you and disagree with Julio. In some cases, having the hooks in the test framework to do setup/cleanup outside of the test itself is useful. My experience is that people who write tests don't always write perfect code, and do not always do proper setup/teardown inside the testcase. In FreeBSD and NetBSD, we can enforce this and force people to fix the tests. In large products based on FreeBSD, my experience is that this is not often possible in real-world QA environments. Having hooks in the test framework to do setup/teardown like in other unit test frameworks would be useful. -- Craig From owner-freebsd-testing@FreeBSD.ORG Tue Nov 4 20:42:48 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2C03A992; Tue, 4 Nov 2014 20:42:48 +0000 (UTC) Received: from mail-pa0-x22f.google.com (mail-pa0-x22f.google.com [IPv6:2607:f8b0:400e:c03::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E50D2BDD; Tue, 4 Nov 2014 20:42:47 +0000 (UTC) Received: by mail-pa0-f47.google.com with SMTP id kx10so15041814pab.6 for ; Tue, 04 Nov 2014 12:42:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=O0VTyylUwJ1jlbggNqXmdWgY9QNeBBChjhz+Mm4xJ3A=; b=NqMYHcusxntMR/wA0H5vwtYAX1cajL6WEPx/cRBPgGmMST2iXGAEpARQeB1DZMXB30 A9awJdjPEmwfbtSSdGibgbmq7xfiP3JuVedTPYdbWalVZpA8/G0TQA1pt/1iVIEpIy9w AxCabI5d6t7QEwxqAKBx4Fm3QswVmShtlBLrZljzicalt1j/y+tsLj/rgsrbDQcjy2oc jABtlvOSsfx5aSiy99wkQ1mZK7z1i8CuiunkIxslxMACVcWNGI5WX/zVSAGEKb5gEEzV EKwjWgiEYM/8QdiYKSdb8cNG0xOrx9Hyh2Fvzq3Z9meT9QKihpnzb62adznfaf8OBOJe RoVQ== X-Received: by 10.68.255.195 with SMTP id as3mr53425035pbd.38.1415133767027; Tue, 04 Nov 2014 12:42:47 -0800 (PST) Received: from [192.168.192.112] ([129.253.54.225]) by mx.google.com with ESMTPSA id dl1sm1220225pbc.16.2014.11.04.12.42.45 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 04 Nov 2014 12:42:46 -0800 (PST) Content-Type: multipart/signed; boundary="Apple-Mail=_A3DBDBAC-4EC7-4C3E-8FBC-5FE6428C2EAE"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Build failed in Jenkins: FreeBSD_HEAD-tests2 #187 From: Garrett Cooper In-Reply-To: <931111222.11.1415131748662.JavaMail.jenkins@jenkins-9.freebsd.org> Date: Tue, 4 Nov 2014 12:42:43 -0800 Message-Id: References: <1762572422.10.1415110079367.JavaMail.jenkins@jenkins-9.freebsd.org> <931111222.11.1415131748662.JavaMail.jenkins@jenkins-9.freebsd.org> To: jenkins-admin@freebsd.org X-Mailer: Apple Mail (2.1878.6) Cc: "freebsd-testing@freebsd.org" , freebsd-current Current X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Nov 2014 20:42:48 -0000 --Apple-Mail=_A3DBDBAC-4EC7-4C3E-8FBC-5FE6428C2EAE Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Nov 4, 2014, at 12:09, jenkins-admin@freebsd.org wrote: > See ... Hi Craig/Jenkins admins, I opened a pull request to increase the timeout from 1 to 2 = hours when running "kyua test=94: = https://github.com/freebsd/freebsd-ci/pull/1/files . Thank you! --Apple-Mail=_A3DBDBAC-4EC7-4C3E-8FBC-5FE6428C2EAE Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUWTpDAAoJEMZr5QU6S73e6d4IAJ1jfzmgdmy26iPHRkLN/5tf 2/yhDKqv/QuK9lsVmiy2kwZbtsOdWyDNe/mrka1EhboU48ymUqYnev9n6jbtefZf boG78KsnOsFE0JKl4/6cYZstTWq7nSv1nvTOaY6Wmnfg9+AliDR1FgXUH5lqiqjN zDi3tSeAut63kqJ4FXnUI/I6nIMpSNApPHl2G/z3ryLzTZhONxTxqNTUQNYM6xsT 7byURojZO6bYmMwZpI5TSTQtQw3nOzz2XbVOeQKTukU4IJAJKUw4fOIGIEKHXNN7 aerx0vqDDRqOFwLODZHg2HjXUZp2DIrI2D6Z/x1IaGiQixRc5/dp1ccU4/D+h1w= =VOto -----END PGP SIGNATURE----- --Apple-Mail=_A3DBDBAC-4EC7-4C3E-8FBC-5FE6428C2EAE-- From owner-freebsd-testing@FreeBSD.ORG Tue Nov 4 23:49:38 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DFDF4324; Tue, 4 Nov 2014 23:49:38 +0000 (UTC) Received: from mail-pd0-x22c.google.com (mail-pd0-x22c.google.com [IPv6:2607:f8b0:400e:c02::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A45AD390; Tue, 4 Nov 2014 23:49:38 +0000 (UTC) Received: by mail-pd0-f172.google.com with SMTP id r10so14627532pdi.17 for ; Tue, 04 Nov 2014 15:49:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=VgHiZgoYvCkXC/vLeyOv1rDAuQvg6FXkdPY1GtqEkWY=; b=1JtoCW9j+VnZd5WKOXCXCSwarrQTuEOLFqDLXv08JYwsP4wILYGZ0GtChz4jrR+cIs 2vFDDw6+pjjR7eazqQOjUSVXOepY50tIVDqeppmmaW1hS0d1CxjXLWRTI+79dthNZPml 7uu4WU5x26x10tDPUkd5RIwobCxJCM2/7XjfmNRa1073JMW/HLXP6EcPjKJcLX97YmCz MLzqrXsIvV13suPlnz3f1yI0of6l+2/7KXyDLLvlny2iMKcnI551fFk2tDo74TylbDnU Gz31nL50PlYPkGbGbblwI7WSfGs3mxZcuWn9bb7qiPb4vuT3Hq8ntobMrn58JcXF4iwO utYw== X-Received: by 10.66.237.145 with SMTP id vc17mr53507399pac.34.1415144978259; Tue, 04 Nov 2014 15:49:38 -0800 (PST) Received: from [192.168.192.112] ([129.253.54.225]) by mx.google.com with ESMTPSA id wa3sm1426281pac.18.2014.11.04.15.49.36 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 04 Nov 2014 15:49:37 -0800 (PST) Content-Type: multipart/signed; boundary="Apple-Mail=_12D40C76-719F-471B-A47B-C2BC5019C6B8"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Jenkins build is unstable: FreeBSD_HEAD-tests2 #188 From: Garrett Cooper In-Reply-To: <528653889.12.1415142652660.JavaMail.jenkins@jenkins-9.freebsd.org> Date: Tue, 4 Nov 2014 15:49:35 -0800 Message-Id: <31975C97-98BA-40C5-B5EB-AD48718CC188@gmail.com> References: <931111222.11.1415131748662.JavaMail.jenkins@jenkins-9.freebsd.org> <528653889.12.1415142652660.JavaMail.jenkins@jenkins-9.freebsd.org> To: jenkins-admin@freebsd.org X-Mailer: Apple Mail (2.1878.6) Cc: "freebsd-testing@freebsd.org" , freebsd-current Current X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Nov 2014 23:49:39 -0000 --Apple-Mail=_12D40C76-719F-471B-A47B-C2BC5019C6B8 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Nov 4, 2014, at 15:10, jenkins-admin@freebsd.org wrote: > See Hi! I=92ve filed the following bugs to track these failures. I=92ll = closely monitor the tests for another couple hours, then open up reviews = to fix the issues noted: - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D194828 - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D194829 Thank you! --Apple-Mail=_12D40C76-719F-471B-A47B-C2BC5019C6B8 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUWWYPAAoJEMZr5QU6S73eJDsH+wdQuXVCxL4ymd4fSKh39XsS 0jOrWEhIanQg+2t0lvoHkz1Jb1Hfl8c0kPxdbyljkOew7jTbxeP3Mel3p6FvurSp tByfGtlYDM/VE5ensLAc3GIyL6PvvpYRe3T8FpoU8uW/Sw53Sbj6vcC4qfnaxGpj J6+9PfeozQVyBuCW7HXOMzgrOlLFZD8DbXEpdO9+6RMcrNrpfRTaCvU8dhyEVqs3 VWVMPms3kqfH6n2Zua2JNOXFY2KalOTrspv4BXsqGYpuCCsG82Lx9yT9wjr3Nyem y1Zgl0GfL2ZyTZDQqgWMqPeFefzcXaBrhfJsp6WJ3fbqNEtz6FUQy1MLaOG+tvE= =Ywu9 -----END PGP SIGNATURE----- --Apple-Mail=_12D40C76-719F-471B-A47B-C2BC5019C6B8-- From owner-freebsd-testing@FreeBSD.ORG Wed Nov 5 05:52:26 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 682CADC8 for ; Wed, 5 Nov 2014 05:52:26 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4F1AF195 for ; Wed, 5 Nov 2014 05:52:26 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id sA55qQIu088294 for ; Wed, 5 Nov 2014 05:52:26 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-testing@freebsd.org Subject: [Bug 194829] [test] lib.libc.sys.mincore_test:mincore_resid fails in jenkins; not on FreeBSD/i386 with CURRENT Date: Wed, 05 Nov 2014 05:52:26 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: Needs Triage X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: ngie@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc assigned_to short_desc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 05:52:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194829 Garrett Cooper changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |freebsd-testing@freebsd.org Assignee|freebsd-bugs@FreeBSD.org |ngie@FreeBSD.org Summary|lib.libc.sys.mincore_test:m |[test] |incore_resid fails in |lib.libc.sys.mincore_test:m |jenkins; not on |incore_resid fails in |FreeBSD/i386 with CURRENT |jenkins; not on | |FreeBSD/i386 with CURRENT --- Comment #2 from Garrett Cooper --- Strange. Haven't run into any failures on i386. -- You are receiving this mail because: You are on the CC list for the bug. From owner-freebsd-testing@FreeBSD.ORG Wed Nov 5 06:06:09 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9C68FD1 for ; Wed, 5 Nov 2014 06:06:09 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 833142BD for ; Wed, 5 Nov 2014 06:06:09 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id sA5669B7075140 for ; Wed, 5 Nov 2014 06:06:09 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-testing@freebsd.org Subject: [Bug 194828] [test] lib.libc.sys.getcontext_test.setcontext_link fails in jenkins; not on FreeBSD/i386 with CURRENT Date: Wed, 05 Nov 2014 06:06:09 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: ngie@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status cc rep_platform bug_severity Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 06:06:09 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194828 Garrett Cooper changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Triage |Open CC| |freebsd-testing@freebsd.org Hardware|Any |amd64 Severity|Affects Only Me |Affects Some People --- Comment #1 from Garrett Cooper --- This seems to work out of the box on i386, so the reason why it might have been failing before could have been an artifact of my test environment -- either kernel/world version (I was testing on a kernel from 10/15, with a world potentially from 10/31, or thereabouts) or the fact that I was running things out of a chroot part of the time to accelerate testing. I'm testing out a fix on CURRENT with amd64/i386 with a kernel/world built and installed today, in the following config permutations: - amd64/i386 - 11.0-CURRENT @ r273153 - 100 times in a tight loop as root with the following commands... -- kyua test lib/libc -- kyua test lib/libc/sys -- kyua test lib/libc/sys/mincore_test -- You are receiving this mail because: You are on the CC list for the bug. From owner-freebsd-testing@FreeBSD.ORG Wed Nov 5 07:01:19 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2B46CAC3 for ; Wed, 5 Nov 2014 07:01:19 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 12F78A3B for ; Wed, 5 Nov 2014 07:01:19 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id sA571Isp031581 for ; Wed, 5 Nov 2014 07:01:18 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-testing@freebsd.org Subject: [Bug 194828] [test] lib.libc.sys.getcontext_test.setcontext_link fails in jenkins; not on FreeBSD/i386 with CURRENT Date: Wed, 05 Nov 2014 07:01:19 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: ngie@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 07:01:19 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194828 --- Comment #2 from Garrett Cooper --- (In reply to Garrett Cooper from comment #1) > This seems to work out of the box on i386, so the reason why it might have > been failing before could have been an artifact of my test environment -- > either kernel/world version (I was testing on a kernel from 10/15, with a > world potentially from 10/31, or thereabouts) or the fact that I was running > things out of a chroot part of the time to accelerate testing. > > I'm testing out a fix on CURRENT with amd64/i386 with a kernel/world built > and installed today, in the following config permutations: > > - amd64/i386 > - 11.0-CURRENT @ r273153 > - 100 times in a tight loop as root with the following commands... > -- kyua test lib/libc > -- kyua test lib/libc/sys > -- kyua test lib/libc/sys/mincore_test This comment should have been added to bug 194829. -- You are receiving this mail because: You are on the CC list for the bug. From owner-freebsd-testing@FreeBSD.ORG Wed Nov 5 07:01:28 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B829EAD8 for ; Wed, 5 Nov 2014 07:01:28 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A03A9A3D for ; Wed, 5 Nov 2014 07:01:28 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id sA571SWi034326 for ; Wed, 5 Nov 2014 07:01:28 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-testing@freebsd.org Subject: [Bug 194829] [test] lib.libc.sys.mincore_test:mincore_resid fails on recent CURRENT Date: Wed, 05 Nov 2014 07:01:28 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: Needs Triage X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: ngie@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: short_desc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 07:01:28 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194829 Garrett Cooper changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|[test] |[test] |lib.libc.sys.mincore_test:m |lib.libc.sys.mincore_test:m |incore_resid fails in |incore_resid fails on |jenkins; not on |recent CURRENT |FreeBSD/i386 with CURRENT | --- Comment #3 from Garrett Cooper --- (In reply to Garrett Cooper from comment #2) > Strange. Haven't run into any failures on i386. This comment should have been added to bug 194828. -- You are receiving this mail because: You are on the CC list for the bug. From owner-freebsd-testing@FreeBSD.ORG Wed Nov 5 07:10:14 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3EF2DC12 for ; Wed, 5 Nov 2014 07:10:14 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 26AB5AAF for ; Wed, 5 Nov 2014 07:10:14 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id sA57AEqb056227 for ; Wed, 5 Nov 2014 07:10:14 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-testing@freebsd.org Subject: [Bug 194829] [test] lib.libc.sys.mincore_test:mincore_resid fails on recent CURRENT Date: Wed, 05 Nov 2014 07:10:14 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: ngie@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 07:10:14 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194829 Garrett Cooper changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Triage |Open --- Comment #4 from Garrett Cooper --- (In reply to Garrett Cooper from comment #3) > (In reply to Garrett Cooper from comment #2) > > Strange. Haven't run into any failures on i386. > > This comment should have been added to bug 194828. Ah, looking back at the pull request I was using to track the deltas for Isilon's modification to my GitHub branch ( https://github.com/yaneurabeya/freebsd/pull/1 ) this was probably a regression that was introduced between 07/08/2014 and 07/21/2014. -- You are receiving this mail because: You are on the CC list for the bug. From owner-freebsd-testing@FreeBSD.ORG Wed Nov 5 07:10:57 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7AE85C3C for ; Wed, 5 Nov 2014 07:10:57 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 62A35ABD for ; Wed, 5 Nov 2014 07:10:57 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id sA57AvKF057041 for ; Wed, 5 Nov 2014 07:10:57 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-testing@freebsd.org Subject: [Bug 194828] [test] lib.libc.sys.getcontext_test.setcontext_link fails in jenkins; not on FreeBSD/i386 with CURRENT Date: Wed, 05 Nov 2014 07:10:57 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: ngie@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 07:10:57 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194828 --- Comment #3 from Garrett Cooper --- Confirmed that this issue happens on a recent copy of amd64: $ sudo kyua test getcontext_test getcontext_test:getcontext_err -> passed [0.006s] getcontext_test:setcontext_err -> passed [0.006s] getcontext_test:setcontext_link -> failed: /usr/src/contrib/netbsd-tests/lib/libc/sys/t_getcontext.c:118: setcontext(&uc[DEPTH-1]) != 0 [0.005s] Results file id is usr_tests_lib_libc_sys.20141105-070522-007241 Results saved to /root/.kyua/store/results.usr_tests_lib_libc_sys.20141105-070522-007241.db 2/3 passed (1 failed) $ uname -a FreeBSD fuji-current-amd64.local 11.0-CURRENT FreeBSD 11.0-CURRENT #1 r274088+75af1e3(isilon-atf)-dirty: Tue Nov 4 22:27:19 PST 2014 root@fuji-current-amd64.local:/usr/obj/usr/src/sys/FUJI amd64 Rebuilding the kernel again on i386 to see whether or not this occurs there. If it does, this is a potentially new bug. -- You are receiving this mail because: You are on the CC list for the bug. From owner-freebsd-testing@FreeBSD.ORG Wed Nov 5 08:22:11 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 06AA8947 for ; Wed, 5 Nov 2014 08:22:11 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DC64E24B for ; Wed, 5 Nov 2014 08:22:10 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id sA58MAmd076715 for ; Wed, 5 Nov 2014 08:22:10 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-testing@freebsd.org Subject: [Bug 194828] [test] lib.libc.sys.getcontext_test.setcontext_link fails in jenkins; not on FreeBSD/i386 with CURRENT Date: Wed, 05 Nov 2014 08:22:11 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: ngie@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 08:22:11 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194828 --- Comment #4 from Garrett Cooper --- (In reply to Garrett Cooper from comment #3) > Confirmed that this issue happens on a recent copy of amd64: > > $ sudo kyua test getcontext_test > getcontext_test:getcontext_err -> passed [0.006s] > getcontext_test:setcontext_err -> passed [0.006s] > getcontext_test:setcontext_link -> failed: > /usr/src/contrib/netbsd-tests/lib/libc/sys/t_getcontext.c:118: > setcontext(&uc[DEPTH-1]) != 0 [0.005s] > > Results file id is usr_tests_lib_libc_sys.20141105-070522-007241 > Results saved to > /root/.kyua/store/results.usr_tests_lib_libc_sys.20141105-070522-007241.db > > 2/3 passed (1 failed) > $ uname -a > FreeBSD fuji-current-amd64.local 11.0-CURRENT FreeBSD 11.0-CURRENT #1 > r274088+75af1e3(isilon-atf)-dirty: Tue Nov 4 22:27:19 PST 2014 > root@fuji-current-amd64.local:/usr/obj/usr/src/sys/FUJI amd64 > > Rebuilding the kernel again on i386 to see whether or not this occurs there. > If it does, this is a potentially new bug. - The symptom for the failure with the testcase is setcontext(2) is failing with rc == -1, errno == EINVAL on amd64. This isn't documented in setcontext(2), so it's potentially a side effect with the system call. There's an undocumented requirement check in sys_setcontext for uap->ucp == NULL, which will return EINVAL, however. - setcontext still isn't failing on i386. Digging into sys/kern/kern_context.c, there're a few issues that have been revealed with this bug: 1. The testcase might not be setting up required state for setcontext to succeed on FreeBSD. I need to dig into whether or not a fix/workaround was provided for setcontext internally at Isilon. 2. The documentation doesn't note that setcontext can fail with: -- EINVAL in sys_setcontext(2) -- EINVAL in set_mcontext(2) on amd64/i386. -- Whatever copyin would fail with when copying in certain fields from userland when dealing with savecontext(2). Sidenote, there's an additional check in amd64 that's not handled in i386 which could potentially feed back an EINVAL (the "-" context is amd64, the "+" context is i386): - if (mcp->mc_len != sizeof(*mcp) || - (mcp->mc_flags & ~_MC_FLAG_MASK) != 0) + if (mcp->mc_len != sizeof(*mcp)) return (EINVAL); jhb made a recent change to i386, but not amd64 in this syscall. -- You are receiving this mail because: You are on the CC list for the bug. From owner-freebsd-testing@FreeBSD.ORG Wed Nov 5 08:23:34 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9030A97E for ; Wed, 5 Nov 2014 08:23:34 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 77C7725C for ; Wed, 5 Nov 2014 08:23:34 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id sA58NY7Q077225 for ; Wed, 5 Nov 2014 08:23:34 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-testing@freebsd.org Subject: [Bug 194828] [test] lib.libc.sys.getcontext_test.setcontext_link fails on amd64, not i386 Date: Wed, 05 Nov 2014 08:23:34 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: ngie@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: short_desc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 08:23:34 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194828 Garrett Cooper changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|[test] |[test] |lib.libc.sys.getcontext_tes |lib.libc.sys.getcontext_tes |t.setcontext_link fails in |t.setcontext_link fails on |jenkins; not on |amd64, not i386 |FreeBSD/i386 with CURRENT | -- You are receiving this mail because: You are on the CC list for the bug. From owner-freebsd-testing@FreeBSD.ORG Wed Nov 5 16:30:36 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BC979D76 for ; Wed, 5 Nov 2014 16:30:36 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A38F0390 for ; Wed, 5 Nov 2014 16:30:36 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id sA5GUabG040860 for ; Wed, 5 Nov 2014 16:30:36 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-testing@freebsd.org Subject: [Bug 194829] [test] lib.libc.sys.mincore_test:mincore_resid fails on recent CURRENT Date: Wed, 05 Nov 2014 16:30:36 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: Issue Resolved X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: ngie@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 16:30:36 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194829 Garrett Cooper changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Open |Issue Resolved Resolution|--- |FIXED -- You are receiving this mail because: You are on the CC list for the bug. From owner-freebsd-testing@FreeBSD.ORG Wed Nov 5 16:30:45 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7076FD8D for ; Wed, 5 Nov 2014 16:30:45 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 57E92393 for ; Wed, 5 Nov 2014 16:30:45 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id sA5GUj56041133 for ; Wed, 5 Nov 2014 16:30:45 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-testing@freebsd.org Subject: [Bug 194829] [test] lib.libc.sys.mincore_test:mincore_resid fails on recent CURRENT Date: Wed, 05 Nov 2014 16:30:45 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: Issue Resolved X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: ngie@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 16:30:45 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194829 --- Comment #5 from commit-hook@freebsd.org --- A commit references this bug: Author: ngie Date: Wed Nov 5 16:30:02 UTC 2014 New revision: 274142 URL: https://svnweb.freebsd.org/changeset/base/274142 Log: Remove expected failure from lib.libc.sys.t_mincore:mincore_resid The failure was added based on observation seen on 11.0-CURRENT @ r273153, not based on internal testing at EMC/Isilon PR: 194829 Tested with the following configuration: - amd64/i386 - 11.0-CURRENT @ r273153 - 100 times in a tight loop as root with the following commands... -- kyua test lib/libc -- kyua test lib/libc/sys -- kyua test lib/libc/sys/mincore_test Changes: head/contrib/netbsd-tests/lib/libc/sys/t_mincore.c -- You are receiving this mail because: You are on the CC list for the bug. From owner-freebsd-testing@FreeBSD.ORG Wed Nov 5 16:39:47 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C6E071B0 for ; Wed, 5 Nov 2014 16:39:47 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AE51B62A for ; Wed, 5 Nov 2014 16:39:47 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id sA5Gdl2p078092 for ; Wed, 5 Nov 2014 16:39:47 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-testing@freebsd.org Subject: [Bug 194828] [test] lib.libc.sys.getcontext_test.setcontext_link fails on amd64, not i386 Date: Wed, 05 Nov 2014 16:39:47 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: Open X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: ngie@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 16:39:47 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194828 --- Comment #5 from commit-hook@freebsd.org --- A commit references this bug: Author: ngie Date: Wed Nov 5 16:38:57 UTC 2014 New revision: 274143 URL: https://svnweb.freebsd.org/changeset/base/274143 Log: Expect lib.libc.sys.getcontext_test.setcontext_link to fail on amd64; add additional debugging to make the underlying problem more visible Calling setcontext(2) on amd64 as shown in the test program is failing on amd64, not i386, with a return code of -1 and an errno of EINVAL Further investigation is being done in the PR to determine the root cause for the failure PR: 194828 Tested with the following configuration: - amd64/i386 - 11.0-CURRENT @ r273153 - 100 times in a tight loop as root with the following commands... -- kyua test lib/libc -- kyua test lib/libc/sys -- kyua test lib/libc/sys/getcontext_test Changes: head/contrib/netbsd-tests/lib/libc/sys/t_getcontext.c -- You are receiving this mail because: You are on the CC list for the bug. From owner-freebsd-testing@FreeBSD.ORG Wed Nov 5 16:52:17 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 383257A2; Wed, 5 Nov 2014 16:52:17 +0000 (UTC) Received: from mail-pa0-x22a.google.com (mail-pa0-x22a.google.com [IPv6:2607:f8b0:400e:c03::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EF0FE89D; Wed, 5 Nov 2014 16:52:16 +0000 (UTC) Received: by mail-pa0-f42.google.com with SMTP id bj1so1121318pad.29 for ; Wed, 05 Nov 2014 08:52:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=klB2SIqlcPJ/kIPdMA6GYnw99A6tVHsTdrVwGbOW5UA=; b=ZrPQWxYnh4ItsvCGTwKUrtjaKKfHRhRg8nOPggGv0K7iCwK/saEiWS5mBnWTFiSoGZ AVSbRbqcGXKtsInd4RjKfiVks86Hu+7GPT5f3qJ6FRGar1R1gzslwx8rNBz1li7kXYVY MN0MYqPcbyZbfS9pduqvRKaUnYtSV5LhNPvQWJGS5cUVBl/5f2yGAVlDxOammwOR4bZr D1BT6/iFeVNbuJHnE+WZSTJJKbbpIQ9zdWA4z9qFeY6q7i4vh34ijKf47lf5Adl1O7Xs iQWjcDEGcmR+KrDJNQJLQbecgCpsFSoS/uL43ZkfzoccphK/hY4K0JXhgc0P4G7G/gQ0 uDcw== X-Received: by 10.70.88.139 with SMTP id bg11mr2763761pdb.167.1415206336575; Wed, 05 Nov 2014 08:52:16 -0800 (PST) Received: from ?IPv6:2601:8:ab80:7d6:a5f2:d04a:984a:91? ([2601:8:ab80:7d6:a5f2:d04a:984a:91]) by mx.google.com with ESMTPSA id qh4sm3665008pbb.35.2014.11.05.08.52.15 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 05 Nov 2014 08:52:15 -0800 (PST) Content-Type: multipart/signed; boundary="Apple-Mail=_0E9839E2-6C37-44FD-8822-1A0036A69387"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Build failed in Jenkins: FreeBSD_HEAD-tests2 #187 From: Garrett Cooper In-Reply-To: Date: Wed, 5 Nov 2014 08:52:14 -0800 Message-Id: <0E6A20F9-A705-4BDC-9198-E7B57C9B94A1@gmail.com> References: <1762572422.10.1415110079367.JavaMail.jenkins@jenkins-9.freebsd.org> <931111222.11.1415131748662.JavaMail.jenkins@jenkins-9.freebsd.org> To: jenkins-admin@freebsd.org X-Mailer: Apple Mail (2.1878.6) Cc: "freebsd-testing@freebsd.org" , freebsd-current Current X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 16:52:17 -0000 --Apple-Mail=_0E9839E2-6C37-44FD-8822-1A0036A69387 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Nov 4, 2014, at 12:42, Garrett Cooper wrote: > On Nov 4, 2014, at 12:09, jenkins-admin@freebsd.org wrote: >=20 >> See = >=20 > ... >=20 > Hi Craig/Jenkins admins, > I opened a pull request to increase the timeout from 1 to 2 = hours when running "kyua test=94: = https://github.com/freebsd/freebsd-ci/pull/1/files . > Thank you! Hi again, I looked at the output from the latest failed run, and the real = problem is that it=92s assuming that =93# =93 is a sufficiently unique = string to find a prompt. Long story short is it isn=92t. Unique prompt = handling is done better in pxssh with a bit more complex algorithms. = I=92ll provide better unique prompt handling and redo the pull request. Thanks! --Apple-Mail=_0E9839E2-6C37-44FD-8822-1A0036A69387 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUWlW+AAoJEMZr5QU6S73eLxAIAI2Nc4GyXy4MxYkhHKa+9dwv VgtF6N6pbwU3l415KNdHfbs+gw0sWHnH9zx4lBtvWNV8tXoG9uvd1dkwyjvMM6+M TYaBYXKIlqHc3I5Bi/2hfmdELdro1mHGlqTh6W8LL1YN5QBOAR53v9lKGPqU9pcq X08Kiv1aAIwZqb0usPJdqzM7W+nj3bmsJ/+sCI8oBs76FnVbfNJFnRJPpcFG5qpv PlJKAaxhUuyiYpV5JfSgYRgQcVEORE2cmRKZPQNy93eovWgcUaRKAJuCuD9iU69u 3i8hwzcWGE3D298R0vpbjSu7wLcPBO4gMXBMquv06vZc4H2m/9bw0ZQJg/JSLeA= =V9pk -----END PGP SIGNATURE----- --Apple-Mail=_0E9839E2-6C37-44FD-8822-1A0036A69387-- From owner-freebsd-testing@FreeBSD.ORG Wed Nov 5 21:41:39 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 47CC764C for ; Wed, 5 Nov 2014 21:41:39 +0000 (UTC) Received: from mail-qg0-f53.google.com (mail-qg0-f53.google.com [209.85.192.53]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 062E8B95 for ; Wed, 5 Nov 2014 21:41:38 +0000 (UTC) Received: by mail-qg0-f53.google.com with SMTP id z107so22146qgd.40 for ; Wed, 05 Nov 2014 13:41:32 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=pl4ClTB4I6q7PIF/N3IK/YFphKplo5J4r2TevsyDMqs=; b=lLWanwkbXB4KetgmfW4RJSynBo0GvrLX5YOU5OVDkQfzLRc0YxNcbXj/5E6dfMDnUe agpUoV6W71kml6MeU9Wi7Z54B14bFzw/ZsFg9YVe/0rcYNvu8JDZlMcB01nSosbjIPlE +D0IQ3Ej5jOUHW1pewugwDPGSATiwk5LFhniLmnF1HsyM9G1Lek5SuA2wZ4MbCjHj6r6 eo8kKpE29ATPN3DSMP4LOB9W41uSJW8CDFvkyw4FDaMuopkb1EQlrIVfBT2Vh+x8zceW kWpmPvgXvKfgZ0VcVHd3jtA4E3iwhInuwXa7JyG18sUUcFfE9Fvy14RmQTd5OQ6s4oRD tpVg== X-Gm-Message-State: ALoCoQkW16jCGCIzi9QwnfzMJBA7OMDo69GsLXZVA1B8P4Q+yrGsMXOfA7IIC/QXy2c3EmdZvvB4 X-Received: by 10.140.108.182 with SMTP id j51mr41345qgf.27.1415223692304; Wed, 05 Nov 2014 13:41:32 -0800 (PST) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.67.69 with HTTP; Wed, 5 Nov 2014 13:41:12 -0800 (PST) X-Originating-IP: [2620:0:1003:1007:40f9:7510:7feb:6981] In-Reply-To: References: From: Julio Merino Date: Wed, 5 Nov 2014 16:41:12 -0500 X-Google-Sender-Auth: YPhXWe8Y34rB9oah8bUz2Xe6UhY Message-ID: Subject: Re: Kyua/ATF as a test framework discussion To: Garrett Cooper Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 21:41:39 -0000 On Tue, Nov 4, 2014 at 1:38 PM, Garrett Cooper wrot= e: > (developers BCCed; please continue discussion on -testing@) > Hi all, > > A few questions came up yesterday at the BSD Vendor Summit in the= Q&A session after rodrigc=E2=80=99s talk about Kyua/ATF related to testing= on FreeBSD, as related to the testing framework (there are a handful of ot= her questions I=E2=80=99ll bring up in other threads ;)..). In particular, = some of the questions that stood out in my mind were: > > 1. What type of testing does Kyua do? Can it be used in benchmark= testing, stress testing, etc? > 2. Is there a structured way of checking for feature before r= unning my tests? > 3. Can I run tests with custom arguments, e.g. point something at= a filesystem path, etc? How do I do that? > > I=E2=80=99ll try to answer these questions based on experience an= d input I=E2=80=99ve received from several people both internal and externa= l to EMC/Isilon. > > Answer to 1. > > - Kyua/ATF excels as a unit testing framework and only does non-distribut= ed testing. > - Kyua/ATF should not be used in benchmark testing or stress testing. This is a fair description. > There is a key component missing from kyua that allows it to run = functional tests (both on FreeBSD and in general). In particular it does no= t have per test program setup/cleanup hooks that other frameworks do (cmake= , JUnit, LTP, python unittest), and it does not have a per-testcase setup h= ook. Well, this is specifically ATF's fault. The more I look at ATF and other testing libraries, the more I'm convinced that ATF's approach of forcing every test case be in a completely separate process is suboptimal because it prevents having fixtures altogether. If you write your own test program without ATF, then you can easily implement your own global setup and teardown routines. (In fact, the alternative library I'm playing with in https://github.com/jmmv/shtk/tree/unittest actually supports per-test setup/teardown and it'd be trivial to add per-program setup/teardown as well.) Of course, "write your own test program" is poor advice given that doing so from scratch, without support libraries, sucks. The problem is that I do not believe that the ATF interface is fixable to support these cases; we'd need to write a different library, and at that point... is it worth doing? What else is there? You should be able to easily plug existing, more advanced libraries like pyUnit or googletest into Kyua -- and if there is a problem with doing such integration, we should fix it. > Answer to 2. > > There isn=E2=80=99t currently a structured API for asking questio= ns like =E2=80=9Cis this driver loaded?", =E2=80=9Cis this sysctl set?=E2= =80=9D, =E2=80=9Cam I running on UFS/ZFS?=E2=80=9D, etc. In general test ca= ses from tools/regression, etc have invented homegrown logic for handling t= his, so this isn=E2=80=99t a new issue. Having such functionality would be great, but IMHO the majority of it is so tied to FreeBSD that it has to live in the FreeBSD source tree. > Answer to 3. > > Yes and no(-ish). A major goal when designing ATF back then was precisely to avoid variability in test execution: all test programs should behave the same way so that their execution can be scripted and so that the caller (user) needn't know how to run the test program (usually poorly documented). Aren't configuration variables passed through the environment sufficient? Kyua 0.11 should do that. (Note that you say above that #84 wasn't completed, but I'm not sure why.) From owner-freebsd-testing@FreeBSD.ORG Wed Nov 5 21:41:49 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B8CB7669 for ; Wed, 5 Nov 2014 21:41:49 +0000 (UTC) Received: from mail-qa0-f46.google.com (mail-qa0-f46.google.com [209.85.216.46]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 77E67B98 for ; Wed, 5 Nov 2014 21:41:49 +0000 (UTC) Received: by mail-qa0-f46.google.com with SMTP id n8so1189000qaq.33 for ; Wed, 05 Nov 2014 13:41:42 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type; bh=uca9MDkaA4uIG0mc7X4eJzTYa7JmSbHVDnOe645F6H0=; b=ZYqXV2xgPdkhK6SKT0mgqcypUmMM5nA6e3ltPv4cb3eEvLHSy4IFPATrt/3e9bhgE0 eih8U99ERgTPKZz3BtvwWxABy2HbwFbHC/neY6EiXOfKsuvfvIBundfNqMGDb3Mub75v p4VOZ0CdOs1LrdzTSLvVhipt74PzkfPLTBq4udtButVESNbjgFJfnLQ4XHAAAAAU/RrN 7q24BiWvFfhLt+1rd0nB/7LLxS7uqDY9Zu61vjhcsvIUuZXo8orbhEFnAnuSIYSaMh+W 2NWNend1xsqTN80Wk1tw+BTiuh+K3OQ7YGnRP4OCsAXcMh/XysmzCShY7OUGQMKbXn6J 4ppA== X-Gm-Message-State: ALoCoQmHFqFVQ+1ZzKL8uuhr/TjI5sGzdB4639G21a6GNz05gsIitbtkFLJSOZq2v3NLhVBaS2x5 X-Received: by 10.140.19.9 with SMTP id 9mr122595qgg.98.1415223702049; Wed, 05 Nov 2014 13:41:42 -0800 (PST) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.67.69 with HTTP; Wed, 5 Nov 2014 13:41:21 -0800 (PST) X-Originating-IP: [2620:0:1003:1007:40f9:7510:7feb:6981] In-Reply-To: References: From: Julio Merino Date: Wed, 5 Nov 2014 16:41:21 -0500 X-Google-Sender-Auth: IbK0rMRd2Gm-8vcXzYRqpJuojSg Message-ID: Subject: Re: Kyua/ATF as a test framework discussion To: Craig Rodrigues Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 21:41:49 -0000 On Tue, Nov 4, 2014 at 2:40 PM, Craig Rodrigues wrote: > Having hooks in the test framework to do setup/teardown like in other unit > test frameworks would be useful. Please tell me of a test framework that has setup/teardown functionality outside of the test program. I would like to see how they handle it. From owner-freebsd-testing@FreeBSD.ORG Wed Nov 5 22:02:50 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 21EB7BA0; Wed, 5 Nov 2014 22:02:50 +0000 (UTC) Received: from mail-la0-x235.google.com (mail-la0-x235.google.com [IPv6:2a00:1450:4010:c03::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9302FE46; Wed, 5 Nov 2014 22:02:49 +0000 (UTC) Received: by mail-la0-f53.google.com with SMTP id mc6so1564239lab.12 for ; Wed, 05 Nov 2014 14:02:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=HEObTxpz1NhdoYBFMfZ9MY/pa3d6ggbeS81ADN0fsws=; b=zUF/bILAqASRukHfixW3iIEAb0M8AjWe5ysPzgVxxyTawuHKGdmuotXA+i5Q9WwTwe RjnF3l9i1zDa5dRmbM/HyhNpKR1FEFCZZLtyGZYWT6Ualh2QwkDpTvKE+KDQfokqXEn9 MH0he4oX/pFGw/wkc6ZTnF8Ebjh/Sgn22j+Quif2p+6NLdty/w3cnwBtEFp+6hTkiVHS dSdCpUhxzB9Txt9xHT78b2k8pfuLZTvvW16e3IQPLDVjeiDsQKRgbODIeNPPS4dBzQ4T 8fwF2nxMeJYdEcbLi/4o5iykhesBBubFfhQT07SrqeeKW3jPNsPNS5GbIHx4FAnIwiZx mn5g== MIME-Version: 1.0 X-Received: by 10.112.137.39 with SMTP id qf7mr112832lbb.47.1415224967604; Wed, 05 Nov 2014 14:02:47 -0800 (PST) Sender: crodr001@gmail.com Received: by 10.112.130.168 with HTTP; Wed, 5 Nov 2014 14:02:47 -0800 (PST) In-Reply-To: References: Date: Wed, 5 Nov 2014 14:02:47 -0800 X-Google-Sender-Auth: 7eeTZZUiqPdd-YtxdE7ts9MUTu8 Message-ID: Subject: Re: Kyua/ATF as a test framework discussion From: Craig Rodrigues To: Julio Merino Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 22:02:50 -0000 On Wed, Nov 5, 2014 at 1:41 PM, Julio Merino wrote: > > On Tue, Nov 4, 2014 at 2:40 PM, Craig Rodrigues wrote: > > Having hooks in the test framework to do setup/teardown like in other unit > > test frameworks would be useful. > > Please tell me of a test framework that has setup/teardown > functionality outside of the test program. I would like to see how > they handle it. http://nose.readthedocs.org/en/latest/writing_tests.html#test-functions https://docs.python.org/3/library/unittest.html#unittest.TestCase.setUp http://cunit.sourceforge.net/doc/managing_tests.html http://junit.sourceforge.net/junit3.8.1/javadoc/junit/framework/TestCase.html#method_summary -- Craig From owner-freebsd-testing@FreeBSD.ORG Wed Nov 5 23:46:11 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 15C3F18A for ; Wed, 5 Nov 2014 23:46:11 +0000 (UTC) Received: from mail-qc0-f172.google.com (mail-qc0-f172.google.com [209.85.216.172]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C5C13B99 for ; Wed, 5 Nov 2014 23:46:10 +0000 (UTC) Received: by mail-qc0-f172.google.com with SMTP id i17so1548722qcy.3 for ; Wed, 05 Nov 2014 15:46:09 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type; bh=UnbYiIlfF31xVHINULivBoyfoNmA8gz/CrJ6f5BcVtE=; b=e5+i7OTt8hrqrloCuRJXLEQA4xueCkrRTxgEl7yuCf3x5vyB6fCVd5eG1iYZwWhUWb X6E0ETcQ5j9jshjb2qPiHOvff6Vhew6I9lCgLQav+sup+38aWl31G3T5/godN4Gku2Ol RPm3XRRU0Db2KE+l/oB0E9kIR/+sRQJK6Kb0oYAkV6XQK2yzsQm5lSAKoYRBlRsls32K DcgaTi98ZDMbys8o1+xBLVEaKo5aaoO8jnnMWbaFfQMnVay+cndGC5FIGSryMuIHqDJt UwzVAA5iGPBDnyg2jKjbDdoVzUNtEdYNfktFRfF82JoUmFk4h8Ptv94t3CpUijZtRvl6 5WOA== X-Gm-Message-State: ALoCoQnQ06Xn+72/RH5A+4bN3E62SetY93p/JDf17XnkZL+1laA5CYTi0T7icJ41qaiuLLBdMouz X-Received: by 10.224.167.132 with SMTP id q4mr1117718qay.48.1415231169160; Wed, 05 Nov 2014 15:46:09 -0800 (PST) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.67.69 with HTTP; Wed, 5 Nov 2014 15:45:48 -0800 (PST) X-Originating-IP: [184.153.117.174] In-Reply-To: References: From: Julio Merino Date: Wed, 5 Nov 2014 18:45:48 -0500 X-Google-Sender-Auth: 4yOOYgbfbBxVv0Rx_CWrWMKXsIw Message-ID: Subject: Re: Kyua/ATF as a test framework discussion To: Craig Rodrigues Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 23:46:11 -0000 On Wed, Nov 5, 2014 at 5:02 PM, Craig Rodrigues wrote: > > > On Wed, Nov 5, 2014 at 1:41 PM, Julio Merino wrote: >> >> On Tue, Nov 4, 2014 at 2:40 PM, Craig Rodrigues >> wrote: >> > Having hooks in the test framework to do setup/teardown like in other >> > unit >> > test frameworks would be useful. >> >> Please tell me of a test framework that has setup/teardown >> functionality outside of the test program. I would like to see how >> they handle it. > > http://nose.readthedocs.org/en/latest/writing_tests.html#test-functions > https://docs.python.org/3/library/unittest.html#unittest.TestCase.setUp > http://cunit.sourceforge.net/doc/managing_tests.html > http://junit.sourceforge.net/junit3.8.1/javadoc/junit/framework/TestCase.html#method_summary All of these examples point at *in-program* setup/teardown methods, not what you were describing a while ago with hooks in the Kyuafile. Adding per-test case setup/teardown methods to ATF would be easy, but mostly pointless except for maybe in atf-c++ (but I don't think anyone cares about atf-c++). Adding per-test program setup/teardown methods to ATF is interesting, but difficult due to the semantics of the interface of the test programs. Also, the resulting API would be quite hard to handle and understand from a programmer's perspective... See my other mail in this thread. Basically, I believe the current ATF interface is not worth fixing to support these cases and we should look at other existing, unit-testing libraries to see if they fit our needs. Kyua should be the glue to bring them together. In fact, you should be able to use all the libraries you mentioned with Kyua already (or at least that's the goal); and, if not, that's a bug that we should fix. From owner-freebsd-testing@FreeBSD.ORG Thu Nov 6 02:02:20 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B94C1AA4; Thu, 6 Nov 2014 02:02:20 +0000 (UTC) Received: from mail-lb0-x232.google.com (mail-lb0-x232.google.com [IPv6:2a00:1450:4010:c04::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 236C7C45; Thu, 6 Nov 2014 02:02:20 +0000 (UTC) Received: by mail-lb0-f178.google.com with SMTP id f15so137488lbj.9 for ; Wed, 05 Nov 2014 18:02:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=pr2LSFw5kFPy4XeuuzWa988o/eESHFJ0K3rmnw1HQ4U=; b=VQ468+/fUO1bHjNn49Kvq4mAhnf85fuC2FohcpejtBETo+c/QhSOs24//o80wAU5c8 ehGqsuJ28vmYb5KCVZAhtq318+f77nvLzwBdSRXJkH1Q7L30p4290adcDTDy8LWPesgd kitWObEdOWEP028V2hJVHFDJqOxGSoLltx9Q/GHFVezbV608ToeXSk3zY+nIoYpQFAtJ sAp20L2MS0HX1FQu0BXY6htpWFzbiVC1OR8pv0PnWYGDVYRyToMCAZnC3iocgHYi+TVy rqtncOrFbKH66LirQl4VzuxT8o1gZdaQuKCcrumlm5Oim8kK0TWj/78fxDLsclR137zO Le8Q== MIME-Version: 1.0 X-Received: by 10.152.198.199 with SMTP id je7mr1137245lac.25.1415239338210; Wed, 05 Nov 2014 18:02:18 -0800 (PST) Sender: crodr001@gmail.com Received: by 10.112.130.168 with HTTP; Wed, 5 Nov 2014 18:02:18 -0800 (PST) In-Reply-To: References: Date: Wed, 5 Nov 2014 18:02:18 -0800 X-Google-Sender-Auth: Zk_HFx5AWtiKgqSyMkYP1bLwtfU Message-ID: Subject: Re: Kyua/ATF as a test framework discussion From: Craig Rodrigues To: Julio Merino Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Nov 2014 02:02:20 -0000 On Wed, Nov 5, 2014 at 3:45 PM, Julio Merino wrote: > See my other mail in this thread. Basically, I believe the current ATF > interface is not worth fixing to support these cases and we should > look at other existing, unit-testing libraries to see if they fit our > needs. Kyua should be the glue to bring them together. In fact, you > should be able to use all the libraries you mentioned with Kyua > already (or at least that's the goal); and, if not, that's a bug that > we should fix. > I'm confused by your e-mail. Are you not interested in fixing/enhancing ATF, and want to move to "something else"? Is "something else" an incremental improvement on ATF, or something completely different and incompatible? After Garrett's recent work to import NetBSD tests into FreeBSD, we now have over 3000 tests in FreeBSD, many of which are ATF tests. I hope to see people writing more tests using ATF, but they will not be inclined to do so if it is perceived as a dead end. Also, keep in mind, we have a few companies building BSD-based products who have started using ATF and have some interest in it. kyua as it exists right now is mostly OK, and the "kyua + Jenkins" integration has been working out quite well, at least for jenkins.freebsd.org. At the recent FreeBSD Devsummit, I presented what I did: https://wiki.freebsd.org/201411DevAndVendorSummit?action=AttachFile&do=view&target=kyua_jenkins.pdf There was a lot of interest in that work. There are a few companies that want to look at that and use it as a model for testing their own products which are based on BSD. Integrating non-ATF test frameworks in kyua is not seamless right now. If I treat each non-ATF test framework as a "plain_test_program" in kyua, that works mostly OK. However, with "plain_test_program", I don't get the ability to have multiple distinct test cases in a single program, like I do with ATF, which are visible via "kyua list, kyua debug", etc. Improving things in this area is doable, for example taking Alan's Python unittest backend to kyua ( https://lists.freebsd.org/pipermail/freebsd-testing/2013-November/000104.html ) and polishing it up. Also, with "plain_test_program", if I want custom setup/teardown, I need to implement that setup/teardown inside the test program. I don't always want that, and being able to maybe specify the custom setup/teardown routines in the Kyuafile would actually be useful. ATF and Kyua are not perfect, but they are actually very, very good tools, and are taking BSD in a very good direction with respect to testing. Fixing some of the issues in ATF, and making Kyua integrate better with other frameworks will only make it better. I hope you are still interested in doing some of these things. -- Craig