From owner-freebsd-performance@FreeBSD.ORG Mon Sep 22 21:22:11 2014 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 66A4E31E for ; Mon, 22 Sep 2014 21:22:11 +0000 (UTC) Received: from albert.catwhisker.org (mx.catwhisker.org [198.144.209.73]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 33C8BABB for ; Mon, 22 Sep 2014 21:22:10 +0000 (UTC) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.14.9/8.14.9) with ESMTP id s8MLM9QV010215 for ; Mon, 22 Sep 2014 14:22:09 -0700 (PDT) (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.14.9/8.14.9/Submit) id s8MLM9GO010214 for freebsd-performance@freebsd.org; Mon, 22 Sep 2014 14:22:09 -0700 (PDT) (envelope-from david) Date: Mon, 22 Sep 2014 14:22:09 -0700 From: David Wolfskill To: freebsd-performance@freebsd.org Subject: I like iostat, but... Message-ID: <20140922212209.GA9619@albert.catwhisker.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="FL5UXtIhxfXey3p5" Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Sep 2014 21:22:11 -0000 --FL5UXtIhxfXey3p5 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =2E.. I rather wish I could get the same information via sysctl. (Well, something seems to be available via the "opaque" kern.devstat.all sysctl(8) variable, but sysctl(8) doesn't display all of it, and parsing it seems as if that would require knowledge about the internals of the system where the data were acquired.) If iostat(8) could be taught to (optionally) provide a timestamp, that might suffice. The problem I'm trying to solve is this: I need to be able to acquire various resource counters (along with timestamps), so I can post-process the acquired data (generally, on a system other than the one where the data were gathered) in order to be able to see how the resource-consumption changes over the duration of a moderately long-running task (typically. 0.5 - 8 hrs.). I have (with some success) cobbled up a shell script to do much of this. (And yes, I've measured the behavior of some typical workloads in this environment vs. merely running the workload under "/usr/bin/time -lpo", with 5 test iterations for each, there was no statistically significant difference with a 95% confidence interval.) The script: * Parses its arguments, and from that information constructs a command line (which invokes sysctl(8), and (optionally) netstat(1), and pipes that output through awk(1)). * Determines the current time-of-day ("date +%s") * Enters a loop, which: + "eval"s the constructed command line (causing a timestamped line of information to be spat out standard output); the timestamp is from the most-recently-acquired time-of-day. + Determines the current time-of-day ("date +%s"). + Based on the desired sampling interval, calculates the number of seconds to sleep. (If I could get strftime to format fractional seconds, that could be handy.) + Sleeps for the calculated interval. + Determines the current time-of-day ("date +%s"). And for most things I care about, it works fairly well. Further, I do this as a shell script precisely so I don't need to build a new version of the tool for a new target system: the script purposely only requires tools that are in base FreeBSD, and requires no special privilege to use. For some sample graphs I have generated from this kind of data, please see . I believe that having an ability to correlate the "iostat -x" information with the CPU, load average, and memory utilization would be useful -- but I don't see a reasonable way to go about it. If anyone has suggestions, I'm listening. :-} Peace, david --=20 David H. Wolfskill david@catwhisker.org Taliban: Evil cowards with guns afraid of truth from a 14-year old girl. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --FL5UXtIhxfXey3p5 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJUIJL+XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4RThEMDY4QTIxMjc1MDZFRDIzODYzRTc4 QTY3RjlDOERFRjQxOTNCAAoJEIpn+cje9Bk7ha0P/jLtEbOUlGC7te61kHAIAeaf T80IeCwkwZiSokelMP2/w/zTl7aSdlk67Mds+CdrONhIokbjD8Itga2kAsgAR+eW 3FNkx6Y9K89TeA+xFX0uqzvEHUXGplM1EF50VlNpgG2U5RP5cghCs1sigc0frtQ3 a9ykO0Mersbo04S0s51Z7iYCY7hQVDHfER0emHtUh1U1FDC7BRvJbqxErvlgN7ik BWa9ApHVGV3so1PF9GVP6CPchLjpJqScuupzISq6/vV4sGFOv/181ZdBo8er1QuO e5ABffLNg3dMp9RTHzIWffRwe5fH++woy4Pwacpgv8z8oGvB6ZiHE6B/rpS32erg rAiaORE8dqZRuSsYJujH/QASciED9P1iHlrt4dC9CdTzCcmR69Qp3nY5wejaZ7lc tdqVIMNaprai/U3qKdf5vbisNLtZgwx+ptuiDAl13f+AjwuM1ULgrkzFFSbcIobW Ck6HXRInUzNUguVgHIeg2YTyxO3sLt/IJnWbtEKqjF2CPy9l92XfJqfZ7dzMY9PH 7sPqFbB+65fO5SiL/mRfii1r7JaiqzXhBwk4kcWa+VazdPXa6TB2p2SJA7yWW5b8 I2Tj356YsaQ7czHivolCB7/KzuIztomTGhdKyNTAXir9yDzuLyrpAxd1WxNeHsBf vWnW9y9ug1d7gTnJafSU =wi7z -----END PGP SIGNATURE----- --FL5UXtIhxfXey3p5-- From owner-freebsd-performance@FreeBSD.ORG Mon Sep 22 21:30:52 2014 Return-Path: Delivered-To: freebsd-performance@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 218FB51D for ; Mon, 22 Sep 2014 21:30:52 +0000 (UTC) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 53EB4BC9 for ; Mon, 22 Sep 2014 21:30:48 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id AAA05780; Tue, 23 Sep 2014 00:30:44 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1XWBC0-0006ZQ-38; Tue, 23 Sep 2014 00:30:44 +0300 Message-ID: <542094B2.5040302@FreeBSD.org> Date: Tue, 23 Sep 2014 00:29:22 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.1.0 MIME-Version: 1.0 To: David Wolfskill , freebsd-performance@FreeBSD.org Subject: Re: I like iostat, but... References: <20140922212209.GA9619@albert.catwhisker.org> In-Reply-To: <20140922212209.GA9619@albert.catwhisker.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 22 Sep 2014 23:03:59 +0000 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Sep 2014 21:30:52 -0000 On 23/09/2014 00:22, David Wolfskill wrote: > ... I rather wish I could get the same information via sysctl. (Well, > something seems to be available via the "opaque" kern.devstat.all > sysctl(8) variable, but sysctl(8) doesn't display all of it, and parsing > it seems as if that would require knowledge about the internals of the > system where the data were acquired.) Perhaps sysutils/devstat could be of help? > If iostat(8) could be taught to (optionally) provide a timestamp, that > might suffice. > > The problem I'm trying to solve is this: I need to be able to acquire > various resource counters (along with timestamps), so I can post-process > the acquired data (generally, on a system other than the one where the > data were gathered) in order to be able to see how the > resource-consumption changes over the duration of a moderately > long-running task (typically. 0.5 - 8 hrs.). > > I have (with some success) cobbled up a shell script to do much of > this. (And yes, I've measured the behavior of some typical workloads > in this environment vs. merely running the workload under "/usr/bin/time > -lpo", with 5 test iterations for each, there was no statistically > significant difference with a 95% confidence interval.) > > The script: > > * Parses its arguments, and from that information constructs a command > line (which invokes sysctl(8), and (optionally) netstat(1), and > pipes that output through awk(1)). > > * Determines the current time-of-day ("date +%s") > > * Enters a loop, which: > + "eval"s the constructed command line (causing a timestamped line of > information to be spat out standard output); the timestamp is from > the most-recently-acquired time-of-day. > + Determines the current time-of-day ("date +%s"). > + Based on the desired sampling interval, calculates the number of > seconds to sleep. (If I could get strftime to format fractional > seconds, that could be handy.) > + Sleeps for the calculated interval. > + Determines the current time-of-day ("date +%s"). > > And for most things I care about, it works fairly well. > > Further, I do this as a shell script precisely so I don't need to build > a new version of the tool for a new target system: the script purposely > only requires tools that are in base FreeBSD, and requires no special > privilege to use. > > For some sample graphs I have generated from this kind of data, please > see . > > I believe that having an ability to correlate the "iostat -x" > information with the CPU, load average, and memory utilization would be > useful -- but I don't see a reasonable way to go about it. If anyone > has suggestions, I'm listening. :-} -- Andriy Gapon From owner-freebsd-performance@FreeBSD.ORG Tue Sep 23 06:57:03 2014 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B9C3EBAC for ; Tue, 23 Sep 2014 06:57:03 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 74E36829 for ; Tue, 23 Sep 2014 06:57:02 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id 574459DD1D2; Tue, 23 Sep 2014 08:56:58 +0200 (CEST) Subject: Re: I like iostat, but... Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Borja Marcos In-Reply-To: <20140922212209.GA9619@albert.catwhisker.org> Date: Tue, 23 Sep 2014 08:56:54 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <35F9FD7B-6847-4209-A2F9-B1F0ABD01E4C@sarenet.es> References: <20140922212209.GA9619@albert.catwhisker.org> To: David Wolfskill X-Mailer: Apple Mail (2.1283) Cc: freebsd-performance@freebsd.org X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Sep 2014 06:57:03 -0000 On Sep 22, 2014, at 11:22 PM, David Wolfskill wrote: > ... I rather wish I could get the same information via sysctl. (Well, > something seems to be available via the "opaque" kern.devstat.all > sysctl(8) variable, but sysctl(8) doesn't display all of it, and = parsing > it seems as if that would require knowledge about the internals of the > system where the data were acquired.) Reading sysctl from a small C program is not hard at all. I did it for = devilator (a data recollector for Orca). And there's a lot of data available. An advantage is, you avoid launching = several processes. Borja. From owner-freebsd-performance@FreeBSD.ORG Tue Sep 23 08:41:56 2014 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 20DEB431 for ; Tue, 23 Sep 2014 08:41:56 +0000 (UTC) Received: from systemdatarecorder.org (mail.systemdatarecorder.org [54.246.96.61]) (using TLSv1.1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "localhost", Issuer "localhost" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id A0C8F2DD for ; Tue, 23 Sep 2014 08:41:55 +0000 (UTC) Received: from nereid (188-127-209-196.cust.suomicom.net [188.127.209.196]) (authenticated bits=0) by systemdatarecorder.org (8.14.4/8.14.4/Debian-2ubuntu2.1) with ESMTP id s8N8ZS9n014275 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 23 Sep 2014 08:35:29 GMT Date: Tue, 23 Sep 2014 11:38:44 +0300 From: Stefan Parvu To: freebsd-performance@freebsd.org Subject: Re: I like iostat, but... Message-Id: <20140923113844.6f9e9584965dfd401f6943af@systemdatarecorder.org> In-Reply-To: <20140922212209.GA9619@albert.catwhisker.org> References: <20140922212209.GA9619@albert.catwhisker.org> Organization: systemdatarecorder.org X-Mailer: Sylpheed 3.4.2 (GTK+ 2.24.22; amd64-portbld-freebsd11.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Sep 2014 08:41:56 -0000 > ... I rather wish I could get the same information via sysctl. (Well, > something seems to be available via the "opaque" kern.devstat.all > sysctl(8) variable, but sysctl(8) doesn't display all of it, and parsing > it seems as if that would require knowledge about the internals of the > system where the data were acquired.) I gave up parsing sysctl via Perl for disks and network devices. It would be nice to have devstat properly working via sysctl for disk devices. Similar way kern.cp_times does. Currently there is no simple way to extract per disk stats from sysctl as a Perl or Sh consumer, unless we build a C module to do that. > If iostat(8) could be taught to (optionally) provide a timestamp, that > might suffice. In fact all performance userland tools should be able to nicely produce timestamp CSV records which can be used for capacity planning and feed them to an analytic product. Something like: 1411445589:4.01:16.03:383.97:1.93:0.29:1.68:0.11:95.99:0.00:0.00:0.00:0.00:123.00:229516.00:570992.00 or something like iostat: 1411445733:ada0:0.00:0.00:0.00:0.00:0.00:0.00:0.00 where the first field would be always the timestmp, unix time. It is not that complicated but it does not exist. > The problem I'm trying to solve is this: I need to be able to acquire > various resource counters (along with timestamps), so I can post-process > the acquired data (generally, on a system other than the one where the > data were gathered) in order to be able to see how the > resource-consumption changes over the duration of a moderately > long-running task (typically. 0.5 - 8 hrs.). My idea of having standard data recorders: sysrec, cpurec, nicrec diskrec which can extract: overall system consumption, per device statistics. http://www.systemdatarecorder.org/recording/agents.html First place to start with will be sysrec, the main recorder which will report overall system consumption: * cpu utilization across all CPUs * memory utilization * disk IO across all disk devices * network IO across all NICs * LA http://www.systemdatarecorder.org/recording/sysrec_freebsd.html > I believe that having an ability to correlate the "iostat -x" > information with the CPU, load average, and memory utilization would be > useful -- but I don't see a reasonable way to go about it. If anyone > has suggestions, I'm listening. :-} How about sysrec, like describe above. I dont like this version because it is using iostat, netstat for disks and NICs but I dont have a better solution right now. There is another way for NIC stats to use libstatgrab and a Perl module for it. I got some troubles in using it under FreeBSD 10 and having some considerable memory usage over time, and I did not have time to test it further. But I will a bit later. There are some limitations about libstatgrab: does not know per CPU stats for example. http://www.i-scream.org/libstatgrab/ -- Stefan Parvu From owner-freebsd-performance@FreeBSD.ORG Tue Sep 23 08:45:21 2014 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 04FAB4B9 for ; Tue, 23 Sep 2014 08:45:21 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B37A32FA for ; Tue, 23 Sep 2014 08:45:19 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id 6D8359DCF83; Tue, 23 Sep 2014 10:45:16 +0200 (CEST) Subject: Re: I like iostat, but... Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Borja Marcos In-Reply-To: <20140923113844.6f9e9584965dfd401f6943af@systemdatarecorder.org> Date: Tue, 23 Sep 2014 10:45:14 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <659899B2-1816-41FA-9DED-57416928A1EE@sarenet.es> References: <20140922212209.GA9619@albert.catwhisker.org> <20140923113844.6f9e9584965dfd401f6943af@systemdatarecorder.org> To: Stefan Parvu X-Mailer: Apple Mail (2.1283) Cc: freebsd-performance@freebsd.org X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Sep 2014 08:45:21 -0000 On Sep 23, 2014, at 10:38 AM, Stefan Parvu wrote: >=20 >> ... I rather wish I could get the same information via sysctl. = (Well, >> something seems to be available via the "opaque" kern.devstat.all >> sysctl(8) variable, but sysctl(8) doesn't display all of it, and = parsing >> it seems as if that would require knowledge about the internals of = the >> system where the data were acquired.) >=20 > I gave up parsing sysctl via Perl for disks and network devices. It = would be > nice to have devstat properly working via sysctl for disk devices. = Similar way > kern.cp_times does. Currently there is no simple way to extract per = disk stats from > sysctl as a Perl or Sh consumer, unless we build a C module to do = that.=20 Anyway, for disk stats GEOM offers a nice API. You can get delays per = GEOM provider, bandwidths, etc. Borja. From owner-freebsd-performance@FreeBSD.ORG Tue Sep 23 08:55:04 2014 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4D07F661 for ; Tue, 23 Sep 2014 08:55:04 +0000 (UTC) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 157FF625 for ; Tue, 23 Sep 2014 08:55:03 +0000 (UTC) Received: from Julian-MBP3.local (ppp121-45-249-73.lns20.per2.internode.on.net [121.45.249.73]) (authenticated bits=0) by vps1.elischer.org (8.14.9/8.14.9) with ESMTP id s8N8sxgS035052 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Tue, 23 Sep 2014 01:55:02 -0700 (PDT) (envelope-from julian@freebsd.org) Message-ID: <5421355D.1020305@freebsd.org> Date: Tue, 23 Sep 2014 16:54:53 +0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Stefan Parvu , freebsd-performance@freebsd.org Subject: Re: I like iostat, but... References: <20140922212209.GA9619@albert.catwhisker.org> <20140923113844.6f9e9584965dfd401f6943af@systemdatarecorder.org> In-Reply-To: <20140923113844.6f9e9584965dfd401f6943af@systemdatarecorder.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Sep 2014 08:55:04 -0000 On 9/23/14, 4:38 PM, Stefan Parvu wrote: >> ... I rather wish I could get the same information via sysctl. (Well, >> something seems to be available via the "opaque" kern.devstat.all >> sysctl(8) variable, but sysctl(8) doesn't display all of it, and parsing >> it seems as if that would require knowledge about the internals of the >> system where the data were acquired.) > I gave up parsing sysctl via Perl for disks and network devices. It would be > nice to have devstat properly working via sysctl for disk devices. Similar way > kern.cp_times does. Currently there is no simple way to extract per disk stats from > sysctl as a Perl or Sh consumer, unless we build a C module to do that. > >> If iostat(8) could be taught to (optionally) provide a timestamp, that >> might suffice. > In fact all performance userland tools should be able to nicely produce timestamp > CSV records which can be used for capacity planning and feed them to an > analytic product. Something like: > > 1411445589:4.01:16.03:383.97:1.93:0.29:1.68:0.11:95.99:0.00:0.00:0.00:0.00:123.00:229516.00:570992.00 > > or something like iostat: > > 1411445733:ada0:0.00:0.00:0.00:0.00:0.00:0.00:0.00 > > where the first field would be always the timestmp, unix time. It is not that complicated > but it does not exist. > >> The problem I'm trying to solve is this: I need to be able to acquire >> various resource counters (along with timestamps), so I can post-process >> the acquired data (generally, on a system other than the one where the >> data were gathered) in order to be able to see how the >> resource-consumption changes over the duration of a moderately >> long-running task (typically. 0.5 - 8 hrs.). > My idea of having standard data recorders: sysrec, cpurec, nicrec diskrec > which can extract: overall system consumption, per device statistics. > http://www.systemdatarecorder.org/recording/agents.html > > First place to start with will be sysrec, the main recorder which will report overall > system consumption: > > * cpu utilization across all CPUs > * memory utilization > * disk IO across all disk devices > * network IO across all NICs > * LA > http://www.systemdatarecorder.org/recording/sysrec_freebsd.html > > >> I believe that having an ability to correlate the "iostat -x" >> information with the CPU, load average, and memory utilization would be >> useful -- but I don't see a reasonable way to go about it. If anyone >> has suggestions, I'm listening. :-} > How about sysrec, like describe above. I dont like this version because it is using > iostat, netstat for disks and NICs but I dont have a better solution right now. > > There is another way for NIC stats to use libstatgrab and a Perl module for it. I got some > troubles in using it under FreeBSD 10 and having some considerable memory > usage over time, and I did not have time to test it further. But I will a bit later. > There are some limitations about libstatgrab: does not know per CPU stats for > example. > > > http://www.i-scream.org/libstatgrab/ > try this: http://lists.freebsd.org/pipermail/freebsd-current/2006-August/065003.html From owner-freebsd-performance@FreeBSD.ORG Tue Sep 23 09:19:54 2014 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 546F1F96 for ; Tue, 23 Sep 2014 09:19:54 +0000 (UTC) Received: from systemdatarecorder.org (mail.systemdatarecorder.org [54.246.96.61]) (using TLSv1.1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "localhost", Issuer "localhost" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E26E38E2 for ; Tue, 23 Sep 2014 09:19:53 +0000 (UTC) Received: from nereid (188-127-209-196.cust.suomicom.net [188.127.209.196]) (authenticated bits=0) by systemdatarecorder.org (8.14.4/8.14.4/Debian-2ubuntu2.1) with ESMTP id s8N9GRTc014424 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 23 Sep 2014 09:16:27 GMT Date: Tue, 23 Sep 2014 12:19:45 +0300 From: Stefan Parvu To: freebsd-performance@freebsd.org Subject: Re: I like iostat, but... Message-Id: <20140923121945.8975311d308a1ff088dd90b0@systemdatarecorder.org> In-Reply-To: <659899B2-1816-41FA-9DED-57416928A1EE@sarenet.es> References: <20140922212209.GA9619@albert.catwhisker.org> <20140923113844.6f9e9584965dfd401f6943af@systemdatarecorder.org> <659899B2-1816-41FA-9DED-57416928A1EE@sarenet.es> Organization: systemdatarecorder.org X-Mailer: Sylpheed 3.4.2 (GTK+ 2.24.22; amd64-portbld-freebsd11.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Sep 2014 09:19:54 -0000 > Anyway, for disk stats GEOM offers a nice API. You can get delays per GEOM provider, bandwidths, etc. Are you talking about C consumers or I can do that using Perl, Sh ? Is there any way to consume the metrics via Perl, for example ? I could not find any decent documentation how to do that except some emails: http://lists.freebsd.org/pipermail/freebsd-questions/2009-March/194081.html Any ideas ? Thanks, -- Stefan Parvu From owner-freebsd-performance@FreeBSD.ORG Tue Sep 23 09:43:08 2014 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B684ACAB for ; Tue, 23 Sep 2014 09:43:08 +0000 (UTC) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6EBBEC0F for ; Tue, 23 Sep 2014 09:43:07 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop04.sare.net (Postfix) with ESMTPSA id 8B1A69DCCBA; Tue, 23 Sep 2014 11:42:58 +0200 (CEST) Subject: Re: I like iostat, but... Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Borja Marcos In-Reply-To: <20140923121945.8975311d308a1ff088dd90b0@systemdatarecorder.org> Date: Tue, 23 Sep 2014 11:42:57 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20140922212209.GA9619@albert.catwhisker.org> <20140923113844.6f9e9584965dfd401f6943af@systemdatarecorder.org> <659899B2-1816-41FA-9DED-57416928A1EE@sarenet.es> <20140923121945.8975311d308a1ff088dd90b0@systemdatarecorder.org> To: Stefan Parvu X-Mailer: Apple Mail (2.1283) Cc: freebsd-performance@freebsd.org X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Sep 2014 09:43:08 -0000 On Sep 23, 2014, at 11:19 AM, Stefan Parvu wrote: >=20 >> Anyway, for disk stats GEOM offers a nice API. You can get delays per = GEOM provider, bandwidths, etc. >=20 > Are you talking about C consumers or I can do that using Perl, Sh ? >=20 > Is there any way to consume the metrics via Perl, for example ? I = could not find > any decent documentation how to do that except some emails: > = http://lists.freebsd.org/pipermail/freebsd-questions/2009-March/194081.htm= l C consumers, although of course you can write a Perl module in C in case = you prefer Perl. If the shock of reading atrocious code won't make you pull your eyes out = of your sockets I can send you the latest version of devilator a FreeBSD specific data = collector for Orca (https://www.orcaware.com/orca/). It fetches several system stats (CPU usage, memory usage, etc) and = creates text files with a timestamp. Orca generates RRD databases and graphs consuming that = data.=20 The code is ugly as hell (it's always been an evolving hack) but it = works, I've been using it for years. Maybe it will help to do your own version if you wish. Borja. From owner-freebsd-performance@FreeBSD.ORG Tue Sep 23 11:07:38 2014 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C2297ACB for ; Tue, 23 Sep 2014 11:07:38 +0000 (UTC) Received: from systemdatarecorder.org (mail.systemdatarecorder.org [54.246.96.61]) (using TLSv1.1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "localhost", Issuer "localhost" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 5A24A8FA for ; Tue, 23 Sep 2014 11:07:37 +0000 (UTC) Received: from nereid (188-127-209-196.cust.suomicom.net [188.127.209.196]) (authenticated bits=0) by systemdatarecorder.org (8.14.4/8.14.4/Debian-2ubuntu2.1) with ESMTP id s8NB4BZ2014864 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 23 Sep 2014 11:04:11 GMT Date: Tue, 23 Sep 2014 14:07:29 +0300 From: Stefan Parvu To: freebsd-performance@freebsd.org Subject: Re: I like iostat, but... Message-Id: <20140923140729.e0befde4661a35e76c1a4209@systemdatarecorder.org> In-Reply-To: References: <20140922212209.GA9619@albert.catwhisker.org> <20140923113844.6f9e9584965dfd401f6943af@systemdatarecorder.org> <659899B2-1816-41FA-9DED-57416928A1EE@sarenet.es> <20140923121945.8975311d308a1ff088dd90b0@systemdatarecorder.org> Organization: systemdatarecorder.org X-Mailer: Sylpheed 3.4.2 (GTK+ 2.24.22; amd64-portbld-freebsd11.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Sep 2014 11:07:38 -0000 > If the shock of reading atrocious code won't make you pull your eyes out of your sockets I > can send you the latest version of devilator a FreeBSD specific data collector for Orca > (https://www.orcaware.com/orca/). sure. Im still thinking how to improve things on my side. Im not C fluent but I can handle somehow. > It fetches several system stats (CPU usage, memory usage, etc) and creates text files > with a timestamp. Orca generates RRD databases and graphs consuming that data. Sweet. I need to check this. Many thanks indeed. -- Stefan Parvu From owner-freebsd-performance@FreeBSD.ORG Wed Sep 24 15:09:24 2014 Return-Path: Delivered-To: freebsd-performance@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 09896A0D for ; Wed, 24 Sep 2014 15:09:24 +0000 (UTC) Received: from albert.catwhisker.org (mx.catwhisker.org [198.144.209.73]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BBAF8A34 for ; Wed, 24 Sep 2014 15:09:23 +0000 (UTC) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.14.9/8.14.9) with ESMTP id s8OF9FZD031741 for ; Wed, 24 Sep 2014 08:09:15 -0700 (PDT) (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.14.9/8.14.9/Submit) id s8OF9Frm031740 for freebsd-performance@FreeBSD.org; Wed, 24 Sep 2014 08:09:15 -0700 (PDT) (envelope-from david) Date: Wed, 24 Sep 2014 08:09:15 -0700 From: David Wolfskill To: freebsd-performance@FreeBSD.org Subject: Re: I like iostat, but... Message-ID: <20140924150915.GC1221@albert.catwhisker.org> Reply-To: freebsd-performance@FreeBSD.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="b0bHyb0BmsSIM7d2" Content-Disposition: inline In-Reply-To: <5421355D.1020305@freebsd.org> <20140923121945.8975311d308a1ff088dd90b0@systemdatarecorder.org> <659899B2-1816-41FA-9DED-57416928A1EE@sarenet.es> <20140923113844.6f9e9584965dfd401f6943af@systemdatarecorder.org> <35F9FD7B-6847-4209-A2F9-B1F0ABD01E4C@sarenet.es> <542094B2.5040302@FreeBSD.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 15:09:24 -0000 --b0bHyb0BmsSIM7d2 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Sep 23, 2014 at 12:29:22AM +0300, Andriy Gapon wrote: > On 23/09/2014 00:22, David Wolfskill wrote: > > ... I rather wish I could get the same information via sysctl. (Well, > > something seems to be available via the "opaque" kern.devstat.all > > sysctl(8) variable, but sysctl(8) doesn't display all of it, and parsing > > it seems as if that would require knowledge about the internals of the > > system where the data were acquired.) >=20 > Perhaps sysutils/devstat could be of help? > ... On Tue, Sep 23, 2014 at 08:56:54AM +0200, Borja Marcos wrote: > ...=20 > Reading sysctl from a small C program is not hard at all. I did it for de= vilator (a data recollector for Orca). And > there's a lot of data available. An advantage is, you avoid launching sev= eral processes. > ... On Tue, Sep 23, 2014 at 10:45:14AM +0200, Borja Marcos wrote: > ...=20 > Anyway, for disk stats GEOM offers a nice API. You can get delays per GEO= M provider, bandwidths, etc. On Tue, Sep 23, 2014 at 11:38:44AM +0300, Stefan Parvu wrote: > ...=20 > I gave up parsing sysctl via Perl for disks and network devices. It would= be > nice to have devstat properly working via sysctl for disk devices. Simila= r way > kern.cp_times does. Currently there is no simple way to extract per disk = stats from > sysctl as a Perl or Sh consumer, unless we build a C module to do that.= =20 Folks, I appreciate the suggestions, but they address problems other than the one I am trying to solve. In particular: * I require that the tool must only depend on components of base FreeBSD; thus, I don't need to perturb the system I want to measure by installing otherwise unneeded software on it. * I am using a shell script (which uses date, sysctl, netstat, and awk) so I don't need to recompile my data acquisition tool. * The tasks I am trying to measure are software builds similar to a stable/10 "make universe" -- but about 2 - 3 times the duration (and reading and writing a significantly larger amount of data). Thus, the number of additional processes caused by my probe firing even as often as once/second is lost in the noise. > ... > > If iostat(8) could be taught to (optionally) provide a timestamp, that > > might suffice. >=20 > In fact all performance userland tools should be able to nicely produce t= imestamp > CSV records which can be used for capacity planning and feed them to an > analytic product. Something like: >=20 > 1411445589:4.01:16.03:383.97:1.93:0.29:1.68:0.11:95.99:0.00:0.00:0.00:0.0= 0:123.00:229516.00:570992.00 >=20 > or something like iostat: >=20 > 1411445733:ada0:0.00:0.00:0.00:0.00:0.00:0.00:0.00 >=20 > where the first field would be always the timestmp, unix time. It is not = that complicated > but it does not exist. The output of my shell script may be described as: # ::=3D newline # ::=3D | tab # ::=3D : # ::=3D [_0-9a-zA-Z][-_.0-9a-zA-Z]+ # ::=3D [^\t\n]* # Each record will have a field with a tag called "time"; the # associated value will be the number of seconds since the Epoch, # but will be coerced as necessary to ensure that it is monotonically # increasing. # Note that the colon (":") is a valid character in the value part # of a field (but not in the tag). Further, there is no whitespace # on either side of that delimiting colon: everything to the left of the # colon (up to, but not including, the previous tab, if any) is part of the # tag; everything to the right of the colon (up to, but not including, # the next tab or newline) is part of the value. A prior version of the script output CSV; for this version, I chose to use the above format because I had situations where some fields showed up on some lines, but not on others. That tends to make the use of CSV a bit problematic. (On a machine where I post-process the collected data, I have some Perl that reads the above format, creates new field names as necessary to cope with multivariate data (e.g., kern.cp_time or vm.loadavg), and generates CSV from the result.) > ... > My idea of having standard data recorders: sysrec, cpurec, nicrec diskrec= =20 > which can extract: overall system consumption, per device statistics. > http://www.systemdatarecorder.org/recording/agents.html=20 > ... That looks interesting and useful for a broad class of similar problems. However, as far as I can tell, it is not suitable for the problem(s) I am trying to solve in this case. Basically, I have something that works "well enough" for things like CPU counters, memory usage (at the rather coarse granularity that top(1) provides, vs. "vmstat -m" output), load avergaes, and NIC counters, and is readily extensible to any univariate (or simple list of multivariate) (non-opaque) sysctl OIDs. I'd like to be able to include information from the I/O subsystem -- in particular, data that is accessible from "iostat -x". Peace, david --=20 David H. Wolfskill david@catwhisker.org Taliban: Evil cowards with guns afraid of truth from a 14-year old girl. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --b0bHyb0BmsSIM7d2 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJUIt6ZXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4RThEMDY4QTIxMjc1MDZFRDIzODYzRTc4 QTY3RjlDOERFRjQxOTNCAAoJEIpn+cje9Bk7cuUP/0abmI3lcrpBNvTfnEyHkake Kxy475LBqjtGYBTq4jiagQC3wrFCCGLarrW2EW5cmNKOOjmyUl2Fa7KCpccFN2LP JeJDA7rTSH5XNKgt5Xe8902AgQz5FBgkXZeN11eQgLzNs9rn3rjyvJibcdWvO5rj +4eaKct4YZ3HF5BSSXkfMQM8GB6quroZPAQ4/2ZywwFraIUVd8k3akHMsvTOeJ31 XVTB+jGo4Jk0Y/wDjIJ0oTo5q+VyXMJMbqt54J7B81n+CVXzjO1WI2Y+hSDMa/qU Oy4bt7FQnSDND00yqJEa+I/IEuyWljLnvGvbi7/7TUjqPXW7jMNE9BtGZjzf2gY7 5QcZ87pbuLvU8Acsz0n4+VlPTczt7khQ7ibCn6V2aqj930Ow49QVo4MUinksJUQd hiPd2FfqVV+7gL/TVVk5uGdYOQNIaPXfuItZIPoIOz42iECcVk+lkeY1pLHmMiYf s8O9kPcHm2F9PzqcfOUEm0+EAp+whh7rzQPAH7D9+4MkSvJ0tmNuVrIjPU+dt3ic aZvGgFU1RZkeBiLGStaKVJjSlV5BWEq/ole8nSMLjaYbMdEZ+15a3trDTX9NSq4x GDXN0rVbS/WrtQwOFGXmBaBg57ma4LK/GBcXM8QBbCGXwPueVIk7gw0RwoIOrRTx +JtK3oM4x7gkaVGfNHQC =snLV -----END PGP SIGNATURE----- --b0bHyb0BmsSIM7d2-- From owner-freebsd-performance@FreeBSD.ORG Wed Sep 24 15:16:22 2014 Return-Path: Delivered-To: freebsd-performance@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ABAEBFB4 for ; Wed, 24 Sep 2014 15:16:22 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 66673B5B for ; Wed, 24 Sep 2014 15:16:22 +0000 (UTC) Received: from [172.16.1.232] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id 8319A9DD043 for ; Wed, 24 Sep 2014 17:16:13 +0200 (CEST) From: Borja Marcos Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (1.0) Subject: Re: I like iostat, but... Message-Id: <5DBB0BDC-92FB-44FF-8869-67CFA2B52C26@sarenet.es> Date: Wed, 24 Sep 2014 17:16:14 +0200 References: <20140924150915.GC1221@albert.catwhisker.org> In-Reply-To: <20140924150915.GC1221@albert.catwhisker.org> To: "freebsd-performance@FreeBSD.org" X-Mailer: iPad Mail (12A365) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 15:16:22 -0000 > On 24/9/2014, at 17:09, David Wolfskill wrote: >=20 >=20 >> On Tue, Sep 23, 2014 at 10:45:14AM +0200, Borja Marcos wrote: >> ...=20 >> Anyway, for disk stats GEOM offers a nice API. You can get delays per GEO= M provider, bandwidths, etc. >=20 >>=20 >=20 >=20 > Folks, I appreciate the suggestions, but they address problems other > than the one I am trying to solve. >=20 > In particular: > * I require that the tool must only depend on components of base FreeBSD; > thus, I don't need to perturb the system I want to measure by > installing otherwise unneeded software on it. Devilator has no dependencies. It reads sysctl and geom. >=20 > Basically, I have something that works "well enough" for things > like CPU counters, memory usage (at the rather coarse granularity > that top(1) provides, vs. "vmstat -m" output), load avergaes, and > NIC counters, and is readily extensible to any univariate (or simple > list of multivariate) (non-opaque) sysctl OIDs. I'd like to be > able to include information from the I/O subsystem -- in particular, > data that is accessible from "iostat -x". Check the diskbw.c module. Actually Most of it is borrowed from gstat(8). Ju= st format the output data as you wish ;) But you don't need Orca. The agent just creates text files.=20 Cheers, Borja. From owner-freebsd-performance@FreeBSD.ORG Thu Sep 25 00:49:54 2014 Return-Path: Delivered-To: freebsd-performance@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C5F5C218 for ; Thu, 25 Sep 2014 00:49:54 +0000 (UTC) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id B283E331 for ; Thu, 25 Sep 2014 00:49:54 +0000 (UTC) Received: from u10-2-16-063.office.norse-data.com (unknown [50.204.88.51]) by elvis.mu.org (Postfix) with ESMTPSA id F2D3C346DE7C for ; Wed, 24 Sep 2014 17:49:47 -0700 (PDT) Message-ID: <542366AC.3030700@freebsd.org> Date: Wed, 24 Sep 2014 17:49:48 -0700 From: Alfred Perlstein Organization: FreeBSD User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.1.1 MIME-Version: 1.0 To: freebsd-performance@FreeBSD.org Subject: Re: I like iostat, but... References: <20140924150915.GC1221@albert.catwhisker.org> In-Reply-To: <20140924150915.GC1221@albert.catwhisker.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Sep 2014 00:49:54 -0000 On 9/24/14 8:09 AM, David Wolfskill wrote: > On Tue, Sep 23, 2014 at 12:29:22AM +0300, Andriy Gapon wrote: >> On 23/09/2014 00:22, David Wolfskill wrote: >>> ... I rather wish I could get the same information via sysctl. (Well, >>> something seems to be available via the "opaque" kern.devstat.all >>> sysctl(8) variable, but sysctl(8) doesn't display all of it, and parsing >>> it seems as if that would require knowledge about the internals of the >>> system where the data were acquired.) >> Perhaps sysutils/devstat could be of help? >> ... > On Tue, Sep 23, 2014 at 08:56:54AM +0200, Borja Marcos wrote: >> ... >> Reading sysctl from a small C program is not hard at all. I did it for devilator (a data recollector for Orca). And >> there's a lot of data available. An advantage is, you avoid launching several processes. >> ... > On Tue, Sep 23, 2014 at 10:45:14AM +0200, Borja Marcos wrote: >> ... >> Anyway, for disk stats GEOM offers a nice API. You can get delays per GEOM provider, bandwidths, etc. > On Tue, Sep 23, 2014 at 11:38:44AM +0300, Stefan Parvu wrote: >> ... >> I gave up parsing sysctl via Perl for disks and network devices. It would be >> nice to have devstat properly working via sysctl for disk devices. Similar way >> kern.cp_times does. Currently there is no simple way to extract per disk stats from >> sysctl as a Perl or Sh consumer, unless we build a C module to do that. > > Folks, I appreciate the suggestions, but they address problems other > than the one I am trying to solve. > > In particular: > * I require that the tool must only depend on components of base FreeBSD; > thus, I don't need to perturb the system I want to measure by > installing otherwise unneeded software on it. > > * I am using a shell script (which uses date, sysctl, netstat, and awk) > so I don't need to recompile my data acquisition tool. > > * The tasks I am trying to measure are software builds similar to > a stable/10 "make universe" -- but about 2 - 3 times the duration > (and reading and writing a significantly larger amount of data). > > Thus, the number of additional processes caused by my probe firing > even as often as once/second is lost in the noise. > > > >> ... >>> If iostat(8) could be taught to (optionally) provide a timestamp, that >>> might suffice. >> In fact all performance userland tools should be able to nicely produce timestamp >> CSV records which can be used for capacity planning and feed them to an >> analytic product. Something like: >> >> 1411445589:4.01:16.03:383.97:1.93:0.29:1.68:0.11:95.99:0.00:0.00:0.00:0.00:123.00:229516.00:570992.00 >> >> or something like iostat: >> >> 1411445733:ada0:0.00:0.00:0.00:0.00:0.00:0.00:0.00 >> >> where the first field would be always the timestmp, unix time. It is not that complicated >> but it does not exist. > > The output of my shell script may be described as: > > # ::= newline > # ::= | tab > # ::= : > # ::= [_0-9a-zA-Z][-_.0-9a-zA-Z]+ > # ::= [^\t\n]* > > # Each record will have a field with a tag called "time"; the > # associated value will be the number of seconds since the Epoch, > # but will be coerced as necessary to ensure that it is monotonically > # increasing. > > # Note that the colon (":") is a valid character in the value part > # of a field (but not in the tag). Further, there is no whitespace > # on either side of that delimiting colon: everything to the left of the > # colon (up to, but not including, the previous tab, if any) is part of the > # tag; everything to the right of the colon (up to, but not including, > # the next tab or newline) is part of the value. > > A prior version of the script output CSV; for this version, I chose to > use the above format because I had situations where some fields showed > up on some lines, but not on others. That tends to make the use of CSV > a bit problematic. (On a machine where I post-process the collected > data, I have some Perl that reads the above format, creates new field > names as necessary to cope with multivariate data (e.g., kern.cp_time or > vm.loadavg), and generates CSV from the result.) > >> ... >> My idea of having standard data recorders: sysrec, cpurec, nicrec diskrec >> which can extract: overall system consumption, per device statistics. >> http://www.systemdatarecorder.org/recording/agents.html >> ... > That looks interesting and useful for a broad class of similar problems. > > However, as far as I can tell, it is not suitable for the problem(s) I > am trying to solve in this case. > > Basically, I have something that works "well enough" for things > like CPU counters, memory usage (at the rather coarse granularity > that top(1) provides, vs. "vmstat -m" output), load avergaes, and > NIC counters, and is readily extensible to any univariate (or simple > list of multivariate) (non-opaque) sysctl OIDs. I'd like to be > able to include information from the I/O subsystem -- in particular, > data that is accessible from "iostat -x". > David: check this out: https://github.com/alfredperlstein/eagleeye it makes time series data out of a number of tools, it's not really up to date, but you can probably pull useful code from it. Also there was the "machine readable output from utilities" GSoC project that I still need to merge, but that is a larger project. -Alfred