Date: Wed, 8 Jan 2014 21:13:59 +0100 From: Erik Cederstrand <erik+lists@cederstrand.dk> To: Alan Somers <asomers@FreeBSD.org> Cc: "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org> Subject: Re: Continual benchmarking / regression testing? Message-ID: <513D4C78-D6FC-45D8-8B1F-CFD2C96E872F@cederstrand.dk> In-Reply-To: <CAOtMX2hiMAnZ5=-FC1TW07eML4p2s_f6iG%2B6KPofq9zxpbauNg@mail.gmail.com> References: <lah8s3$8ur$1@ger.gmane.org> <CDBEEA8C-90FE-4E4B-B16E-8A5EF7685F51@cederstrand.dk> <CAOtMX2hiMAnZ5=-FC1TW07eML4p2s_f6iG%2B6KPofq9zxpbauNg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Den 08/01/2014 kl. 17.38 skrev Alan Somers <asomers@FreeBSD.org>: >=20 > I like that you stored test results in a SQL database. My SQL-foo is > poor, so ATM my framework is using CSV. It also looks like you've got > code to generate a website. Do you have any example output? Yes, there=92s a website to filter results, generate graphs, see commit = messages between two data points and show the hardware and software = configuration of the client running the benchmark. A continuous = benchmarking framework is only useful if it can assist you in analyzing = the data, finding regressions and their cause. > The PXE stuff, however, does not belong in the > benchmark framework, IMHO. I think that the benchmark framework > should just include the benchmarking and system profiling aspect, not > system configuration. Provisioning and configuring systems can be > done in a separate utility, one that can be shared, for example, with > the continuous Kyua tester. System configuration affects benchmark results, so that needs to be = recorded along with the benchmark results. My work was intended as a = complete continuous benchmarking system with a build machine that = produces OS installation images and tells clients what to install and = what to run. But I agree that a benchmark framework contains many = self-contained parts that could be shared among projects. Erik=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?513D4C78-D6FC-45D8-8B1F-CFD2C96E872F>