From owner-freebsd-hackers@FreeBSD.ORG Wed Jan 8 20:19:25 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 14904362; Wed, 8 Jan 2014 20:19:25 +0000 (UTC) Received: from csmtp11.one.com (csmtp11.one.com [195.47.247.117]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id C4D631E1D; Wed, 8 Jan 2014 20:19:24 +0000 (UTC) Received: from bigmac.router9fbd7c.com (unknown [176.222.238.90]) by csmtp11.one.com (Postfix) with ESMTPA id 8FB23C03B555A; Wed, 8 Jan 2014 20:14:00 +0000 (UTC) Received: from bigmac.router9fbd7c.com ([UNAVAILABLE]. [176.222.238.90]) (using TLSv1 with cipher AES128-SHA) by 0.0.0.0:2500 (trex/4.8.87); Wed, 08 Jan 2014 20:14:00 GMT Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.1 \(1827\)) Subject: Re: Continual benchmarking / regression testing? From: Erik Cederstrand In-Reply-To: Date: Wed, 8 Jan 2014 21:13:59 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <513D4C78-D6FC-45D8-8B1F-CFD2C96E872F@cederstrand.dk> References: To: Alan Somers X-Mailer: Apple Mail (2.1827) Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jan 2014 20:19:25 -0000 Den 08/01/2014 kl. 17.38 skrev Alan Somers : >=20 > I like that you stored test results in a SQL database. My SQL-foo is > poor, so ATM my framework is using CSV. It also looks like you've got > code to generate a website. Do you have any example output? Yes, there=92s a website to filter results, generate graphs, see commit = messages between two data points and show the hardware and software = configuration of the client running the benchmark. A continuous = benchmarking framework is only useful if it can assist you in analyzing = the data, finding regressions and their cause. > The PXE stuff, however, does not belong in the > benchmark framework, IMHO. I think that the benchmark framework > should just include the benchmarking and system profiling aspect, not > system configuration. Provisioning and configuring systems can be > done in a separate utility, one that can be shared, for example, with > the continuous Kyua tester. System configuration affects benchmark results, so that needs to be = recorded along with the benchmark results. My work was intended as a = complete continuous benchmarking system with a build machine that = produces OS installation images and tells clients what to install and = what to run. But I agree that a benchmark framework contains many = self-contained parts that could be shared among projects. Erik=