From owner-freebsd-net@FreeBSD.ORG  Tue Mar 29 18:42:42 2005
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP
	id 2FCCE16A4CE; Tue, 29 Mar 2005 18:42:42 +0000 (GMT)
Received: from mta1.cl.cam.ac.uk (mta1.cl.cam.ac.uk [128.232.0.15])
	by mx1.FreeBSD.org (Postfix) with ESMTP
	id E1E1543D31; Tue, 29 Mar 2005 18:42:40 +0000 (GMT)
	(envelope-from peter.sewell@cl.cam.ac.uk)
Received: from stem.cl.cam.ac.uk ([128.232.9.62] helo=cl.cam.ac.uk)
	by mta1.cl.cam.ac.uk with esmtp (Exim 3.092 #1)
	id 1DGLfi-0001Xt-00; Tue, 29 Mar 2005 19:42:38 +0100
To: Brooks Davis <brooks@one-eyed-alien.net>
In-Reply-To: Message from Brooks Davis <brooks@one-eyed-alien.net> 
	<20050329172338.GA5995@odin.ac.hmc.edu> 
Date: Tue, 29 Mar 2005 19:42:37 +0100
From: Peter Sewell <Peter.Sewell@cl.cam.ac.uk>
Message-Id: <E1DGLfi-0001Xt-00@mta1.cl.cam.ac.uk>
cc: gnn@freebsd.org
cc: Peter.Sewell@cl.cam.ac.uk
cc: freebsd-net@freebsd.org
Subject: Re: Rigorous specification for TCP, UDP, and Sockets
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.1
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 29 Mar 2005 18:42:42 -0000


>> >Are you plannning to share the tools as
>> >well?  That's what I'd be most interested in seeing, basically the
>> >ability to turn your conformance tests into regression tests.
>> 
>> It's unclear at the moment - we'd certainly like to make that
>> regression testing more routine, but the current tools are
>> >non-trivial
>> to drive.  We'll be looking to see how much interest there is, and
>> also how accessible the spec is, before going further.
>
>In what way are they hard to drive?  Are they difficult to set up, or
>difficult to run?  

both, I'm afraid - there's quite a complex infrastructure, both for
generating tests and for running the checker over them (in parallel on
a server farm), then interpreting the results of the checker is
non-trivial.  One needs to iterate the test
generation/checking/spec-fixing loop many times. 

>If they are difficult to set up, you might take a
>look at using EmuLab (www.emulab.net) to build images with configured
>tests so OSes just need to be updated to test a change.

Interesting - I didn't know about that (though I don't think it helps right now).


>> What kind of regression testing is in use now?
>
>None for the most part.  People test some limited things, but I don't
>think anyone does the kind if rigorous testing we should be doing.

ah :-(

cheers,
Peter