From owner-freebsd-current@FreeBSD.ORG Mon Feb 11 11:43:22 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 7719021D for ; Mon, 11 Feb 2013 11:43:22 +0000 (UTC) (envelope-from erik@cederstrand.dk) Received: from csmtp3.one.com (csmtp3.one.com [91.198.169.23]) by mx1.freebsd.org (Postfix) with ESMTP id 413D4F0 for ; Mon, 11 Feb 2013 11:43:22 +0000 (UTC) Received: from [192.168.1.11] (unknown [217.157.7.221]) by csmtp3.one.com (Postfix) with ESMTPA id 18BBC240B1D8; Mon, 11 Feb 2013 11:43:18 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: 7+ days of dogfood From: Erik Cederstrand In-Reply-To: <20130211063858.0375a6ed@X220.ovitrap.com> Date: Mon, 11 Feb 2013 12:43:17 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <88F312FE-E783-4C24-8964-BBDE4DE6653E@cederstrand.dk> References: <20130210000723.GA73630@troutmask.apl.washington.edu> <20130210083010.7df5b997@X220.ovitrap.com> <256C7A7B-DFA7-4C0F-B389-AB10E0DA42D0@cederstrand.dk> <20130211063858.0375a6ed@X220.ovitrap.com> To: Erich Dollansky X-Mailer: Apple Mail (2.1499) Cc: freebsd-current@freebsd.org, Steve Kargl X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Feb 2013 11:43:22 -0000 Erich, Den 11/02/2013 kl. 00.38 skrev Erich Dollansky = : > On Sun, 10 Feb 2013 15:57:01 +0100 > Erik Cederstrand wrote: >=20 >> And as long as there is no automatic can taster doing quality >> assurance of the produced cans, then foul cans will go unnoticed >> until a dog pukes all over the carpet :-) >>=20 > Isn't this the idea of HEAD? It's certainly not the idea of HEAD that everyone should experience the = same bugs, compile errors, runtime errors and even have old bugs pop up = again repeatedly. It may be the consequence of running HEAD, but = certainly not the idea. >> For this to change, we really need to catch up on years of neglect in >> e.g. src/tools/regression/. I really applaud the people doing the >> thankless job of changing this. >>=20 > I do not believe that this all can be automated. I'm not saying that testing is all-or-nothing. OS testing is not easy, = and many tests are impractical or expensive if they require real = hardware in complicated setups. How do you reliably test an IEEE 802.11s = mesh implementation? Or scheduling on huge servers that are too = expensive to purchase? I think that is one of the reasons that FreeBSD = has not caught up on automated testing and continuous integration. But = regression tests are useful even though they don't give 100% code = coverage. Currently, you can't even "make test" in src/tools/regression/ = and run the tests that are there. Apart from the compile-tests done by = the tinderboxes, I'm not aware of any coordinated effort to = systematically do runtime or even performance testing of FreeBSD. Thanks, Erik=