From owner-freebsd-dtrace@FreeBSD.ORG Sun Oct 13 02:17:57 2013 Return-Path: Delivered-To: dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 8F42591A for ; Sun, 13 Oct 2013 02:17:57 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from nm34-vm7.bullet.mail.bf1.yahoo.com (nm34-vm7.bullet.mail.bf1.yahoo.com [72.30.239.79]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2F75D2BA4 for ; Sun, 13 Oct 2013 02:17:56 +0000 (UTC) Received: from [98.139.215.140] by nm34.bullet.mail.bf1.yahoo.com with NNFMP; 13 Oct 2013 02:17:49 -0000 Received: from [98.139.211.207] by tm11.bullet.mail.bf1.yahoo.com with NNFMP; 13 Oct 2013 02:17:49 -0000 Received: from [127.0.0.1] by smtp216.mail.bf1.yahoo.com with NNFMP; 13 Oct 2013 02:17:49 -0000 X-Yahoo-Newman-Id: 344694.74294.bm@smtp216.mail.bf1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: thgVkUYVM1m0.jfZ1Ascx5EOjjluJIjEFkuwALKnCBgcCfe 7LmpqluBtyqnxsiW_5g5MlEoOkdZ6e5ADJpLgItpeBTPtaVtoMaOGO8ot9cg UJFSmeQ4QiRK6qCXsHVLV9.0QUtUU16JHNnJO2DY.cHvVFXzrwmRmvADQSap WSkOhMeEptwPyjzHKmP21ZarZI1ZLVL1x.05Iguvc6g5vQTS9PD4p5rCw_fi .aNbUUg_qLFj6xMtZnI3vTIOMp_F9P3q5aGsV6tCg8jKAMGYjvfJtTQhe1Af LiDSRfwpfaKsVssotNZQ0FEMlQUQEq9mXKBkpVXX3q0aKBaE1KpWM7vboV03 MMw14C5ONz3uKzwplzw9aYclec3FdAOVKWew7pwLnpmt_Sm4P2Egc9gnlYpl WulxYKL0NuaQSYjko92PhjwFX3fmjG1MkuPKBK2aRV5qGYi9CtOo9rw13shi t8U28SPJqBkWoR3KD7QuXLwnFoa_OxzvXbVAuylGafBC4FR_pbBlXsEjnw.l Ei4rxlFThrxwWIEwLvYHaor.UziamFuHQCZHpuVscKtXerNon2zKEZHhJEDO N.Ixw.l3hDq0uFmvc79P8UOgLkf6HWk9nbRqrqbaDNuxb8dCoybDZzqYwH_I B3VsNNI8pUUPeI.SDhG_NaMnspqJD0_xKQcU.sVOy8j8R6Mx8KVZH_lxD6ag VU9cMtoJANo63R1NvkTVgiMiqH_Y.FIMeHLqaYFd9wnHaOjgcQLS4MZLvheD 83fR8ilyxa4FTursZgqiSu2NzwIUHahKbe6W6zbdDLAeRtDzVXx9OcwRr9Jk Vk6K0OpWYJZnsArmeQjZgL4dT3Ezjke2Q6vS1vVES714GE1..1DYxjwnaDAc QUfUrrxFH7kBy6k960mO3i0ZLhRinzZFheDpG0XPuPacztNP883cwu537KAz VkauOGQ-- X-Yahoo-SMTP: xcjD0guswBAZaPPIbxpWwLcp9Unf X-Rocket-Received: from [192.168.0.101] (pfg@190.157.126.109 with ) by smtp216.mail.bf1.yahoo.com with SMTP; 12 Oct 2013 19:17:49 -0700 PDT Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\)) Subject: Re: Dtrace providers wanted list From: Pedro Giffuni In-Reply-To: <52596B73.2050702@FreeBSD.org> Date: Sat, 12 Oct 2013 21:17:46 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: <6A4B05EA-D9C7-4014-BF86-4676EF01071C@FreeBSD.org> References: <52546385.2050203@FreeBSD.org> <20131010042544.GB65451@raichu> <52596B73.2050702@FreeBSD.org> To: dtrace@FreeBSD.org X-Mailer: Apple Mail (2.1510) X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Oct 2013 02:17:57 -0000 Il giorno 12/ott/2013, alle ore 10:32, Pedro Giffuni = ha scritto: > On 09.10.2013 23:25, Mark Johnston wrote: >> On Tue, Oct 08, 2013 at 02:56:53PM -0500, Pedro Giffuni wrote: >>> Hello; >>>=20 >>> Sometime ago I looked around the list of Oracle's DTrace providers. >>>=20 >>> https://wikis.oracle.com/display/DTrace/Providers >>>=20 >>> We absolutely want them and although extremely useful as it is, = DTrace >>> won't be complete until we have them all ;). For the time being we >>> should probably focus on getting the really critical ones though. >>>=20 >>> The first in the list that we don't have that I think is critical is >>> mentioned in Brendan Gregg's FreeBSD specific blog post: >>>=20 >>> = http://dtrace.org/blogs/brendan/2013/09/25/the-use-method-freebsd-performa= nce-checklist/ >>>=20 >>> "Tracing paging is tricker until the vminfo provider is added; you = could >>> try tracing from swap_pager_putpages() and swap_pager_getpages(), = but I >>> didn=92t see an easy way to walk back to a thread struct; another = approach >>> may be via vm_fault_hold(). Good luck. ..." >=20 > I started looking at this but it is somewhat more work than expected: > Solaris has a specific vminfo interface in sysinfo.h. It looks very = handy > for providing general vm statistics but our vm is different and I am > not sure we want to go into implementing such interface. It depends > on a VM guru, anyways. >=20 > OTOH, I found the fsinfo provider: >=20 > = https://bitbucket.org/illumos/illumos-gate/commits/5b50c2dcfdf948c4e4f0b88= 0eb76210067e7af7c >=20 > Our VFS does have some similarity so this may be easier/doable, plus > it may be needed by the scsi and ZFS providers. Ugh =85 I looked at this one too, and similarly to the vminfo provider, there is = a vopstats structure in vnode.h that we don't have. Both the vopstats and vminfo structures are used to support Solaris' = kstat(1M) utility. In other words, this is an in kernel system made to report statistics. = While it seems natural to use them for DTrace in Solaris, we would have to = reimplement it in some way if we want to have equivalent providers in FreeBSD. :-( I am afraid that the CPC Provider may have similar issues but I haven't = looked at it. Pedro. From owner-freebsd-dtrace@FreeBSD.ORG Sun Oct 13 16:00:12 2013 Return-Path: Delivered-To: dtrace@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 32F0482E; Sun, 13 Oct 2013 16:00:12 +0000 (UTC) (envelope-from gnn@freebsd.org) Received: from vps.hungerhost.com (vps.hungerhost.com [216.38.53.176]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id F0B5F2931; Sun, 13 Oct 2013 16:00:08 +0000 (UTC) Received: from pool-108-21-114-62.nycmny.east.verizon.net ([108.21.114.62]:58115 helo=[192.168.0.193]) by vps.hungerhost.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.80.1) (envelope-from ) id 1VVO4M-00022W-4E; Sun, 13 Oct 2013 11:59:03 -0400 Content-Type: multipart/signed; boundary="Apple-Mail=_3D78A635-0546-401D-9A60-5092B35A94F9"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\)) Subject: Re: Dtrace providers wanted list From: George Neville-Neil In-Reply-To: <6A4B05EA-D9C7-4014-BF86-4676EF01071C@FreeBSD.org> Date: Sun, 13 Oct 2013 11:59:27 -0400 Message-Id: <9455846D-2301-4055-9AA1-99177F21A3AC@freebsd.org> References: <52546385.2050203@FreeBSD.org> <20131010042544.GB65451@raichu> <52596B73.2050702@FreeBSD.org> <6A4B05EA-D9C7-4014-BF86-4676EF01071C@FreeBSD.org> To: Pedro Giffuni X-Mailer: Apple Mail (2.1510) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - vps.hungerhost.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - freebsd.org X-Get-Message-Sender-Via: vps.hungerhost.com: authenticated_id: gnn@neville-neil.com Cc: dtrace@FreeBSD.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Oct 2013 16:00:12 -0000 --Apple-Mail=_3D78A635-0546-401D-9A60-5092B35A94F9 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Oct 12, 2013, at 22:17 , Pedro Giffuni wrote: >=20 > Il giorno 12/ott/2013, alle ore 10:32, Pedro Giffuni = ha scritto: >=20 >> On 09.10.2013 23:25, Mark Johnston wrote: >>> On Tue, Oct 08, 2013 at 02:56:53PM -0500, Pedro Giffuni wrote: >>>> Hello; >>>>=20 >>>> Sometime ago I looked around the list of Oracle's DTrace providers. >>>>=20 >>>> https://wikis.oracle.com/display/DTrace/Providers >>>>=20 >>>> We absolutely want them and although extremely useful as it is, = DTrace >>>> won't be complete until we have them all ;). For the time being we >>>> should probably focus on getting the really critical ones though. >>>>=20 >>>> The first in the list that we don't have that I think is critical = is >>>> mentioned in Brendan Gregg's FreeBSD specific blog post: >>>>=20 >>>> = http://dtrace.org/blogs/brendan/2013/09/25/the-use-method-freebsd-performa= nce-checklist/ >>>>=20 >>>> "Tracing paging is tricker until the vminfo provider is added; you = could >>>> try tracing from swap_pager_putpages() and swap_pager_getpages(), = but I >>>> didn=92t see an easy way to walk back to a thread struct; another = approach >>>> may be via vm_fault_hold(). Good luck. ..." >>=20 >> I started looking at this but it is somewhat more work than expected: >> Solaris has a specific vminfo interface in sysinfo.h. It looks very = handy >> for providing general vm statistics but our vm is different and I am >> not sure we want to go into implementing such interface. It depends >> on a VM guru, anyways. >>=20 >> OTOH, I found the fsinfo provider: >>=20 >> = https://bitbucket.org/illumos/illumos-gate/commits/5b50c2dcfdf948c4e4f0b88= 0eb76210067e7af7c >>=20 >> Our VFS does have some similarity so this may be easier/doable, plus >> it may be needed by the scsi and ZFS providers. >=20 > Ugh =85 >=20 > I looked at this one too, and similarly to the vminfo provider, there = is a vopstats > structure in vnode.h that we don't have. >=20 > Both the vopstats and vminfo structures are used to support Solaris' = kstat(1M) utility. > In other words, this is an in kernel system made to report statistics. = While it > seems natural to use them for DTrace in Solaris, we would have to = reimplement > it in some way if we want to have equivalent providers in FreeBSD. :-( >=20 > I am afraid that the CPC Provider may have similar issues but I = haven't looked at it. >=20 The CPC provider can be hooked to hwpmc, but one thing I note in the = Illumos provider is that they preface things with PAPI (which is actually its own = package, though the Illumos folks seem to have just snagged the name and don't use the actual = library). Best, George --Apple-Mail=_3D78A635-0546-401D-9A60-5092B35A94F9 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlJaw18ACgkQYdh2wUQKM9L5rwCgigRPXQjoWxJez9URo/jR1EII A/cAn2Y8EP0AqSlgenr+kdpmkDWpgcCO =YfXU -----END PGP SIGNATURE----- --Apple-Mail=_3D78A635-0546-401D-9A60-5092B35A94F9-- From owner-freebsd-dtrace@FreeBSD.ORG Tue Oct 15 22:43:49 2013 Return-Path: Delivered-To: dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 5B7D06FF for ; Tue, 15 Oct 2013 22:43:49 +0000 (UTC) (envelope-from brendan.gregg@joyent.com) Received: from mail-pb0-f53.google.com (mail-pb0-f53.google.com [209.85.160.53]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 357622DA7 for ; Tue, 15 Oct 2013 22:43:48 +0000 (UTC) Received: by mail-pb0-f53.google.com with SMTP id up15so9453861pbc.12 for ; Tue, 15 Oct 2013 15:43:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=8fLi/vcvfRFgHSu+G0G55wvffj37M0kjv0KQQKNnotU=; b=cIfDAafNveY57xwAkalJ4PS7BtJB+gM/C9uZ4GaMkaSfPoCTg1bOKJPrHf1X7j2tyo v2DaEYEfPBUiQO/8jM2FKwLkQpVw1rfXi7hk7BLAYNv0/Ek2H4zFwWy89x02V8msJbLG 4IPV2ju8ILMSVkZIHoVUeErl7bQs7aUCZKVXSPM77iEAYluXBv2WQwbUu/FnPPOmpXoC UG5gGDJdwimftj0qs7Zv7br7FNaHEXwHWIpVRh5k2oVR88SYVF5PHtoHbp2ZQLc5Cqhv ux1x3JoHN34wE9htUwA5jLJQtk5IzOR39+s/KDqfXK5sj3h30cNumUAd+naPxN5pR/sj ga/Q== X-Gm-Message-State: ALoCoQlv/DkWKfpf0750RYw4laa0f/3y4+ACriALvZlh5rbuNYpsq6Q5LSid0lHLYiXqCbZ95PQu MIME-Version: 1.0 X-Received: by 10.68.44.33 with SMTP id b1mr43558474pbm.53.1381877028377; Tue, 15 Oct 2013 15:43:48 -0700 (PDT) Received: by 10.70.55.34 with HTTP; Tue, 15 Oct 2013 15:43:48 -0700 (PDT) In-Reply-To: <20131010042544.GB65451@raichu> References: <52546385.2050203@FreeBSD.org> <20131010042544.GB65451@raichu> Date: Tue, 15 Oct 2013 15:43:48 -0700 Message-ID: Subject: Re: Dtrace providers wanted list From: Brendan Gregg To: Mark Johnston , Robert Mustacchi Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: dtrace@freebsd.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 22:43:49 -0000 On Wed, Oct 9, 2013 at 9:25 PM, Mark Johnston wrote: > On Tue, Oct 08, 2013 at 02:56:53PM -0500, Pedro Giffuni wrote: > [...] > > > > This is closely related to pmc(3) but obviously our implementation is > > completely different from the Solaris one. > > > > > > Well, just though I should share the above links in the hope of > > motivating more DTrace provider porting. At this time our base Dtrace > > port is pretty good but we have the chicken and egg problem where > > developers don't know how useful DTrace really is because there is no > > provider for their code. > > I think it's also important to have more and better documentation of > existing providers. More providers are nice, but they're not as useful > when their existence is not widely known or it's not clear how they > might be used. > > I don't really like having links to an Oracle wiki page at least partly > because none of the FreeBSD ports of those providers are 100% compatible. > I've started writing man pages for the network providers; an example is > here: > > http://people.freebsd.org/~markj/dtrace/dtrace-ip.4.txt > > But I'm open to other forms of documentation as well. > Man pages would be handy. These would be in addition to the other online docs I use: dtrace -lvn, and vi /usr/lib/dtrace/*.d! Robert Mustacchi has been getting an updated version of the original DTrace guide on http://dtrace.org/guide, from https://github.com/rmustacc/illumos-docbooks/tree/master/raw/dtrace. It's currently the "illumos Dynamic Tracing Guide", although we have talked about how to support other OSes in the same place (not sure how). Brendan -- Brendan Gregg, Joyent http://dtrace.org/blogs/brendan From owner-freebsd-dtrace@FreeBSD.ORG Tue Oct 15 23:32:23 2013 Return-Path: Delivered-To: freebsd-dtrace@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id CF53463B for ; Tue, 15 Oct 2013 23:32:23 +0000 (UTC) (envelope-from jmg@h2.funkthat.com) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B236C2053 for ; Tue, 15 Oct 2013 23:32:20 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id r9FNWJcS044587 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 15 Oct 2013 16:32:19 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id r9FNWJWH044586 for freebsd-dtrace@FreeBSD.org; Tue, 15 Oct 2013 16:32:19 -0700 (PDT) (envelope-from jmg) Date: Tue, 15 Oct 2013 16:32:19 -0700 From: John-Mark Gurney To: freebsd-dtrace@FreeBSD.org Subject: dtrace -c doesn't work? Message-ID: <20131015233219.GZ56872@funkthat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Tue, 15 Oct 2013 16:32:19 -0700 (PDT) X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 23:32:23 -0000 Well, I've decided to try to learn dtrace to do some benchmarking, and so I tried to use -c to measure what the command does... except it seems to fail... even the simple: dtrace -n 'syscall:::entry { @num[execname] = count(); }' -c 'echo foo' doesn't work... it gives me: # dtrace -n 'syscall:::entry { @num[execname] = count(); }' -c 'echo foo' foo dtrace: failed to control pid 3766: process exited with status 0 ktrace shows it execing /bin/echo and it running fine, but for some reason dtrace can't handle it... P.S. Why are dtrace and dtraceall seperate? -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-dtrace@FreeBSD.ORG Wed Oct 16 01:50:34 2013 Return-Path: Delivered-To: freebsd-dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 9E98BDCC for ; Wed, 16 Oct 2013 01:50:34 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qe0-x22d.google.com (mail-qe0-x22d.google.com [IPv6:2607:f8b0:400d:c02::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5E1E0263A for ; Wed, 16 Oct 2013 01:50:34 +0000 (UTC) Received: by mail-qe0-f45.google.com with SMTP id 8so80253qea.4 for ; Tue, 15 Oct 2013 18:50:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=TTPqCLK9MwKmdbQfx2a94wOeSgw+RCsx4wnG+4BTyYA=; b=Iz3+R+X8ZxtZq5irAh4xDt7TI6X9revzp/P+OTXFbIyZ2qe3p+ODh6QXcHec5jDHZT kYs7uLKB/fBpNemHEBSFjc1I7jPHwL3nT3UZ9eMCOKjA65UTZiKhDKnSlQFQBApTx3t0 qAZJTe1fn9Z0QOrRkE1VUwdlmn6hR/5O+cTd43f0CSeptHhr0NpKM5cLEtc4X5gt7+7D oAygLdSaY+9UWyj9d9TF5r19boS/ZTvnzAD/0MOK6P1+rAat6BZ85R/JyRrnGbfknhY5 lz/dHDkTH2hDL1OkCdSYDpr9WZuCcS1VV4o65HBTlPbj9IGsjtsBKOeg8DTxkuK2hvEE Cnsg== X-Received: by 10.224.28.65 with SMTP id l1mr1126362qac.47.1381888233534; Tue, 15 Oct 2013 18:50:33 -0700 (PDT) Received: from raichu (24-212-218-13.cable.teksavvy.com. [24.212.218.13]) by mx.google.com with ESMTPSA id h6sm4365124qej.4.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 15 Oct 2013 18:50:32 -0700 (PDT) Sender: Mark Johnston Date: Tue, 15 Oct 2013 21:50:27 -0400 From: Mark Johnston To: Brendan Gregg Subject: Re: [dtrace-discuss] Re: pr_psargs on FreeBSD Message-ID: <20131016015027.GA81435@raichu> References: <20130915221622.GA2981@raichu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-dtrace@freebsd.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Oct 2013 01:50:34 -0000 On Mon, Sep 23, 2013 at 09:04:02AM -0700, Brendan Gregg wrote: > G'Day Mark, > > Looks good, and I can't think of a better way. Cool, thanks! > > In the DIF_SUBR_MEMSTR loop, do you want to add a tunable so that it aborts > after some large number of iterations? Otherwise a typo could put us in a > very long loop in DTrace/interrupts-disabled context. The code could be > like the following from DIF_SUBR_MSGSIZE: > > /* > * We want to prevent against denial-of-service > here, > * so we're only going to search the list for > * dtrace_msgdsize_max mblks. > */ > if (cont++ > dtrace_msgdsize_max) { > *flags |= CPU_DTRACE_ILLOP; > break; > } > > So, create dtrace_memstr_max that's set to 512 or something. I added a memstr_max sysctl and committed the change as r256571: http://svnweb.freebsd.org/base?view=revision&revision=256571 > > Having execsnoop work would be great, along with opensnoop and friends. > Hopefully they can join dtruss in /usr/bin. :) > > Brendan > > > > On Sun, Sep 15, 2013 at 3:16 PM, Mark Johnston wrote: > > > Hi! > > > > One of the problems with FreeBSD's DTrace implementation is that it > > doesn't have a good way of getting the arguments of a given process. > > This is because of the way that they're stored, which is as a single > > buffer with null-terminator between consecutive arguments. In > > particular, there is no way within DTrace to convert such a buffer to a > > single string (say by replacing all but the last terminator with > > spaces). So scripts like execsnoop will only print the first element of > > a process' argv on FreeBSD, which isn't particularly helpful for > > interpreted programs. > > > > You can get a fixed number of arguments with something like > > > > printf("%s %s %s", cupsinfo->pr_psargs, > > curpsinfo->pr_psargs + strlen(curpsinfo->pr_psargs) + 1, > > curpsinfo->pr_psargs + strlen(curpsinfo->pr_psargs) + 1 + > > strlen(curpsinfo->pr_psargs + strlen(curpsinfo->pr_psargs) + 1) + 1); > > > > but this is less than ideal. :) > > > > Apparently OS X has the same problem, but I don't have a machine to test > > with, so I'm not completely sure. > > > > It seems to me that there are two ways to solve this problem: change the > > format that FreeBSD uses to store process arguments, or add a function > > to DTrace which can convert the argument buffer to a string. I'm not too > > keen on the former approach, so here's a proposal for the latter. It'd > > be great to get some feedback on it and find out whether anyone thinks > > it's a terrible idea. :) > > > > It's kind of a strong-armed approach, but the problem's been around for a > > while and I can't think of any clever solutions - I'd love to see an > > alternative solution. > > > > My patch against FreeBSD (pasted below) adds a function called memstr() > > to DTrace. memstr() takes three arguments: addr, c, and len, and returns > > a copy of addr of length len with all null-terminators replaced by c, > > and with the last byte replaced by a null-terminator. In particular, > > memstr() always returns a string of length len - 1, unless len == 0, in > > which case we return the empty string. > > > > With this function, the translator for the pr_psargs fields becomes > > > > translator psinfo_t < struct proc *T > { > > ... > > pr_psargs = memstr(T->p_args->ar_args, ' ', T->p_args->ar_length); > > ... > > }; > > > > and execsnoop works properly. Any thoughts on this function? Have I missed > > a better solution? A patch for FreeBSD is below. > > > > Thanks, > > -Mark > > > > diff --git a/cddl/contrib/opensolaris/lib/libdtrace/common/dt_open.c > > b/cddl/contrib/opensolaris/lib/libdtrace/common/dt_open.c > > index 9c9b2a6..83c81e5 100644 > > --- a/cddl/contrib/opensolaris/lib/libdtrace/common/dt_open.c > > +++ b/cddl/contrib/opensolaris/lib/libdtrace/common/dt_open.c > > @@ -122,8 +122,9 @@ > > #define DT_VERS_1_8_1 DT_VERSION_NUMBER(1, 8, 1) > > #define DT_VERS_1_9 DT_VERSION_NUMBER(1, 9, 0) > > #define DT_VERS_1_9_1 DT_VERSION_NUMBER(1, 9, 1) > > -#define DT_VERS_LATEST DT_VERS_1_9_1 > > -#define DT_VERS_STRING "Sun D 1.9.1" > > +#define DT_VERS_1_9_2 DT_VERSION_NUMBER(1, 9, 2) > > +#define DT_VERS_LATEST DT_VERS_1_9_2 > > +#define DT_VERS_STRING "Sun D 1.9.2" > > > > const dt_version_t _dtrace_versions[] = { > > DT_VERS_1_0, /* D API 1.0.0 (PSARC 2001/466) Solaris 10 FCS */ > > @@ -145,6 +146,7 @@ const dt_version_t _dtrace_versions[] = { > > DT_VERS_1_8_1, /* D API 1.8.1 */ > > DT_VERS_1_9, /* D API 1.9 */ > > DT_VERS_1_9_1, /* D API 1.9.1 */ > > + DT_VERS_1_9_2, /* D API 1.9.2 */ > > 0 > > }; > > > > @@ -311,6 +313,8 @@ static const dt_ident_t _dtrace_globals[] = { > > &dt_idops_func, "void(@)" }, > > { "memref", DT_IDENT_FUNC, 0, DIF_SUBR_MEMREF, DT_ATTR_STABCMN, > > DT_VERS_1_1, > > &dt_idops_func, "uintptr_t *(void *, size_t)" }, > > +{ "memstr", DT_IDENT_FUNC, 0, DIF_SUBR_MEMSTR, DT_ATTR_STABCMN, > > DT_VERS_1_0, > > + &dt_idops_func, "string(void *, char, size_t)" }, > > { "min", DT_IDENT_AGGFUNC, 0, DTRACEAGG_MIN, DT_ATTR_STABCMN, DT_VERS_1_0, > > &dt_idops_func, "void(@)" }, > > { "mod", DT_IDENT_ACTFUNC, 0, DT_ACT_MOD, DT_ATTR_STABCMN, > > diff --git a/cddl/lib/libdtrace/psinfo.d b/cddl/lib/libdtrace/psinfo.d > > index 068e72e..b2a009a 100644 > > --- a/cddl/lib/libdtrace/psinfo.d > > +++ b/cddl/lib/libdtrace/psinfo.d > > @@ -57,7 +57,7 @@ translator psinfo_t < struct proc *T > { > > pr_gid = T->p_ucred->cr_rgid; > > pr_egid = T->p_ucred->cr_groups[0]; > > pr_addr = 0; > > - pr_psargs = stringof(T->p_args->ar_args); > > + pr_psargs = memstr(T->p_args->ar_args, ' ', T->p_args->ar_length); > > pr_arglen = T->p_args->ar_length; > > pr_jailid = T->p_ucred->cr_prison->pr_id; > > }; > > diff --git a/sys/cddl/contrib/opensolaris/uts/common/dtrace/dtrace.c > > b/sys/cddl/contrib/opensolaris/uts/common/dtrace/dtrace.c > > index babc42c4..6d991fb 100644 > > --- a/sys/cddl/contrib/opensolaris/uts/common/dtrace/dtrace.c > > +++ b/sys/cddl/contrib/opensolaris/uts/common/dtrace/dtrace.c > > @@ -4920,6 +4920,38 @@ inetout: regs[rd] = (uintptr_t)end + 1; > > break; > > } > > > > + case DIF_SUBR_MEMSTR: { > > + char *str = (char *)mstate->dtms_scratch_ptr; > > + uintptr_t mem = tupregs[0].dttk_value; > > + char c = tupregs[1].dttk_value; > > + size_t size = tupregs[2].dttk_value; > > + uint8_t n; > > + int i; > > + > > + regs[rd] = 0; > > + > > + if (size == 0) > > + break; > > + > > + if (!dtrace_canload(mem, size - 1, mstate, vstate)) > > + break; > > + > > + if (!DTRACE_INSCRATCH(mstate, size)) { > > + DTRACE_CPUFLAG_SET(CPU_DTRACE_NOSCRATCH); > > + break; > > + } > > + > > + for (i = 0; i < size - 1; i++) { > > + n = dtrace_load8(mem++); > > + str[i] = (n == 0) ? c : n; > > + } > > + str[size - 1] = 0; > > + > > + regs[rd] = (uintptr_t)str; > > + mstate->dtms_scratch_ptr += size; > > + break; > > + } > > + > > case DIF_SUBR_TYPEREF: { > > uintptr_t size = 4 * sizeof(uintptr_t); > > uintptr_t *typeref = (uintptr_t *) > > P2ROUNDUP(mstate->dtms_scratch_ptr, sizeof(uintptr_t)); > > @@ -9102,6 +9134,7 @@ dtrace_difo_validate_helper(dtrace_difo_t *dp) > > subr == DIF_SUBR_NTOHL || > > subr == DIF_SUBR_NTOHLL || > > subr == DIF_SUBR_MEMREF || > > + subr == DIF_SUBR_MEMSTR || > > subr == DIF_SUBR_TYPEREF) > > break; > > > > diff --git a/sys/cddl/contrib/opensolaris/uts/common/sys/dtrace.h > > b/sys/cddl/contrib/opensolaris/uts/common/sys/dtrace.h > > index 8728e30..295457c 100644 > > --- a/sys/cddl/contrib/opensolaris/uts/common/sys/dtrace.h > > +++ b/sys/cddl/contrib/opensolaris/uts/common/sys/dtrace.h > > @@ -311,8 +311,9 @@ typedef enum dtrace_probespec { > > #define DIF_SUBR_SX_SHARED_HELD 48 > > #define DIF_SUBR_SX_EXCLUSIVE_HELD 49 > > #define DIF_SUBR_SX_ISEXCLUSIVE 50 > > +#define DIF_SUBR_MEMSTR 51 > > > > -#define DIF_SUBR_MAX 50 /* max subroutine > > value */ > > +#define DIF_SUBR_MAX 51 /* max subroutine > > value */ > > > > typedef uint32_t dif_instr_t; > > > > _______________________________________________ > > freebsd-dtrace@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-dtrace > > To unsubscribe, send any mail to "freebsd-dtrace-unsubscribe@freebsd.org" > > > > > > -- > Brendan Gregg, Joyent http://dtrace.org/blogs/brendan > > > > ------------------------------------------- > dtrace-discuss > Archives: https://www.listbox.com/member/archive/184261/=now > RSS Feed: https://www.listbox.com/member/archive/rss/184261/24010072-2177618e > Modify Your Subscription: https://www.listbox.com/member/?member_id=24010072&id_secret=24010072-bf518af7 > Powered by Listbox: http://www.listbox.com From owner-freebsd-dtrace@FreeBSD.ORG Thu Oct 17 03:34:07 2013 Return-Path: Delivered-To: freebsd-dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 582232DF for ; Thu, 17 Oct 2013 03:34:07 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-ie0-x22a.google.com (mail-ie0-x22a.google.com [IPv6:2607:f8b0:4001:c03::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 276692E06 for ; Thu, 17 Oct 2013 03:34:07 +0000 (UTC) Received: by mail-ie0-f170.google.com with SMTP id at1so3071247iec.29 for ; Wed, 16 Oct 2013 20:34:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=xwslVZl4H5MpqhO7ynyYmpVt1+HEEBL2R6uaMqViLpg=; b=JlVbdgn2W6bf7lz41jCPR2GRSCT4KKousYGBU/+UZ/7YexGq9l0v3wB822dkFqSwZV n0Pc6ehjljxFvNEaNaMn1e+C1nhrlDa2PWY9xrniPtZKckttvnUtUHIxw16BL2ThF8p4 IVgwdNoCXAqzKjlosWB57nZ/MSVC8pCUTaXnnq/xq7nlRTIqpehU+LIeeXNRXhH0KAcV hRso8ZtE7sTkYdOHPlt4czbp7SvE7Wd78ngUmJaGl0Z/Yo9xhWrhlQHRNb3jHGxHglQV 9LTXvr5AujAmBL+LjTWfRpsZBUd5NeD/tmwL0ahX3Rk2DRHFdd4bZU/HO0Yu9D5s+LoC Su5Q== X-Received: by 10.50.126.74 with SMTP id mw10mr24766860igb.24.1381980845815; Wed, 16 Oct 2013 20:34:05 -0700 (PDT) Received: from charmander (24-212-218-13.cable.teksavvy.com. [24.212.218.13]) by mx.google.com with ESMTPSA id ft2sm6420108igb.5.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 16 Oct 2013 20:34:05 -0700 (PDT) Sender: Mark Johnston Date: Wed, 16 Oct 2013 23:34:00 -0400 From: Mark Johnston To: John-Mark Gurney Subject: Re: dtrace -c doesn't work? Message-ID: <20131017033400.GA31544@charmander> References: <20131015233219.GZ56872@funkthat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20131015233219.GZ56872@funkthat.com> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-dtrace@FreeBSD.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Oct 2013 03:34:07 -0000 On Tue, Oct 15, 2013 at 04:32:19PM -0700, John-Mark Gurney wrote: > Well, I've decided to try to learn dtrace to do some benchmarking, and > so I tried to use -c to measure what the command does... except it seems > to fail... > > even the simple: > dtrace -n 'syscall:::entry { @num[execname] = count(); }' -c 'echo foo' > > doesn't work... it gives me: > # dtrace -n 'syscall:::entry { @num[execname] = count(); }' -c 'echo foo' > foo > dtrace: failed to control pid 3766: process exited with status 0 > > ktrace shows it execing /bin/echo and it running fine, but for some > reason dtrace can't handle it... My guess is that the /bin/echo on your system is stripped. dtrace(1) on FreeBSD will set a breakpoint on main() in the victim process and register the script with the kernel when that breakpoint fires. If libproc can't find the address of main(), the breakpoint won't fire, and your script won't run. If /bin/echo isn't stripped, your example works properly on my laptop. Adding '-x evaltime=postinit' to the dtrace(1) flags should also allow the script to run properly. avg@ tried to fix this a little while ago by changing dtrace to instead put a breakpoint on r_debug_state in rtld (r248644). This works for your example, but breaks USDT since that breakpoint apparently fires before ELF ctors run, and thus before dtrace_dof_init() can run (which is needed for USDT to work). I'm not sure what the best way to fix this is. Perhaps we could add a second breakpoint to rtld for use by dtrace, or maybe there's some way to set a breakpoint on the DOF init code. It would also be nice if libproc could learn to follow .gnu_debuglink sections when performing symbol lookup. I use WITH_DEBUG_FILES on my laptop to keep debug info in separate files, but I still have the same problem as you with the above example. I'll work on fixing this too. > > P.S. Why are dtrace and dtraceall seperate? dtrace.ko contains the "core" DTrace kernel code. Most of the actual providers and support code (e.g. fasttrap, sdt) are implemented in their own modules. dtraceall depends on all the DTrace-related modules, so it's almost always better to kldload dtraceall instead of kldloading dtrace. In fact, dtrace(1) was recently changed to automatically kldload dtraceall if needed, so there's no reason to manually kldload anything now. -Mark From owner-freebsd-dtrace@FreeBSD.ORG Thu Oct 17 07:15:49 2013 Return-Path: Delivered-To: freebsd-dtrace@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 9555DB4A; Thu, 17 Oct 2013 07:15:49 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 813F7286B; Thu, 17 Oct 2013 07:15:47 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id KAA02541; Thu, 17 Oct 2013 10:15:46 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1VWho9-0000HY-S1; Thu, 17 Oct 2013 10:15:45 +0300 Message-ID: <525F8E64.4020105@FreeBSD.org> Date: Thu, 17 Oct 2013 10:14:44 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.0.1 MIME-Version: 1.0 To: Mark Johnston , John-Mark Gurney Subject: Re: dtrace -c doesn't work? References: <20131015233219.GZ56872@funkthat.com> <20131017033400.GA31544@charmander> In-Reply-To: <20131017033400.GA31544@charmander> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Konstantin Belousov , freebsd-dtrace@FreeBSD.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Oct 2013 07:15:49 -0000 on 17/10/2013 06:34 Mark Johnston said the following: > My guess is that the /bin/echo on your system is stripped. dtrace(1) on > FreeBSD will set a breakpoint on main() in the victim process and > register the script with the kernel when that breakpoint fires. If > libproc can't find the address of main(), the breakpoint won't fire, and > your script won't run. If /bin/echo isn't stripped, your example works > properly on my laptop. Adding '-x evaltime=postinit' to the dtrace(1) > flags should also allow the script to run properly. > > avg@ tried to fix this a little while ago by changing dtrace to > instead put a breakpoint on r_debug_state in rtld (r248644). This works > for your example, but breaks USDT since that breakpoint apparently fires > before ELF ctors run, and thus before dtrace_dof_init() can run (which is > needed for USDT to work). That's exactly what happened. My idea of fixing that was to move r_debug_state call to after preinit_main. But Kostik told me that that was a terrible idea. Unfortunately I can't recall right now why. I've been meaning to restore my knowledge and memory of the code and discuss the issue again. > I'm not sure what the best way to fix this is. Perhaps we could add a > second breakpoint to rtld for use by dtrace, or maybe there's some way > to set a breakpoint on the DOF init code. Kostik also suggested this. -- Andriy Gapon From owner-freebsd-dtrace@FreeBSD.ORG Thu Oct 17 15:40:20 2013 Return-Path: Delivered-To: freebsd-dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2594F287; Thu, 17 Oct 2013 15:40:20 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qa0-x22e.google.com (mail-qa0-x22e.google.com [IPv6:2607:f8b0:400d:c00::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B7C8A2B3D; Thu, 17 Oct 2013 15:40:19 +0000 (UTC) Received: by mail-qa0-f46.google.com with SMTP id j15so5623489qaq.19 for ; Thu, 17 Oct 2013 08:40:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=4s+M9XlZ6rB4ZqGkRCqOytn9/l7h7YEhLdsYB59h0O8=; b=LZx5ZJGI80mj/OUMeKbwdnaRjZAUZ+zRWaKwMk7ReRuJTZBjsyVeI9+9Y9sYMZrBb3 k73V9Dfw9YXshCI189eSsCFvnxp5qhEV6x2Tja2zk8O/zhTBcLXBa6gCfIm+q22HHEOc 9EVJkz1lqoP9h+ZBVbH2wxiGwvuoqsgZGB3dF3in6TjcWKBf2/SVM3I8Q6nm4CPMqpZY njRW7mChUEvTFeIRMOR0FX7UYOdQ5D+TkxY0naSQW5YCD8WQtgXuRjAOK1STYEKdO/gq Oh4gYlpFOsCv0f/WQTaUl/Q5IZ32TI68fLxXzDmv1yuNK0XdfB2i95P/pQiF8d95Z5fH nbRA== X-Received: by 10.49.76.6 with SMTP id g6mr4850840qew.41.1382024418866; Thu, 17 Oct 2013 08:40:18 -0700 (PDT) Received: from charmander.sandvine.com ([64.7.137.182]) by mx.google.com with ESMTPSA id n7sm179965300qai.1.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 17 Oct 2013 08:40:18 -0700 (PDT) Sender: Mark Johnston Date: Thu, 17 Oct 2013 11:40:03 -0400 From: Mark Johnston To: Andriy Gapon Subject: Re: dtrace -c doesn't work? Message-ID: <20131017154003.GA2926@charmander.sandvine.com> References: <20131015233219.GZ56872@funkthat.com> <20131017033400.GA31544@charmander> <525F8E64.4020105@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <525F8E64.4020105@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: John-Mark Gurney , Konstantin Belousov , freebsd-dtrace@FreeBSD.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Oct 2013 15:40:20 -0000 On Thu, Oct 17, 2013 at 10:14:44AM +0300, Andriy Gapon wrote: > on 17/10/2013 06:34 Mark Johnston said the following: > > My guess is that the /bin/echo on your system is stripped. dtrace(1) on > > FreeBSD will set a breakpoint on main() in the victim process and > > register the script with the kernel when that breakpoint fires. If > > libproc can't find the address of main(), the breakpoint won't fire, and > > your script won't run. If /bin/echo isn't stripped, your example works > > properly on my laptop. Adding '-x evaltime=postinit' to the dtrace(1) > > flags should also allow the script to run properly. > > > > avg@ tried to fix this a little while ago by changing dtrace to > > instead put a breakpoint on r_debug_state in rtld (r248644). This works > > for your example, but breaks USDT since that breakpoint apparently fires > > before ELF ctors run, and thus before dtrace_dof_init() can run (which is > > needed for USDT to work). > > That's exactly what happened. > My idea of fixing that was to move r_debug_state call to after preinit_main. > But Kostik told me that that was a terrible idea. Unfortunately I can't recall > right now why. I suppose this would make it impossible to use gdb to step through init functions? > I've been meaning to restore my knowledge and memory of the code > and discuss the issue again. > > > I'm not sure what the best way to fix this is. Perhaps we could add a > > second breakpoint to rtld for use by dtrace, or maybe there's some way > > to set a breakpoint on the DOF init code. > > Kostik also suggested this. You mean adding a second breakpoint function to rtld? Perhaps something like r_debug_state_postinit(), called after objlist_call_init(). -Mark From owner-freebsd-dtrace@FreeBSD.ORG Thu Oct 17 17:27:51 2013 Return-Path: Delivered-To: freebsd-dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 59A975D6; Thu, 17 Oct 2013 17:27:51 +0000 (UTC) (envelope-from jmg@h2.funkthat.com) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 3AE9D238F; Thu, 17 Oct 2013 17:27:51 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id r9HHRi13076452 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 17 Oct 2013 10:27:44 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id r9HHRiSf076451; Thu, 17 Oct 2013 10:27:44 -0700 (PDT) (envelope-from jmg) Date: Thu, 17 Oct 2013 10:27:44 -0700 From: John-Mark Gurney To: Mark Johnston Subject: Re: dtrace -c doesn't work? Message-ID: <20131017172744.GF56872@funkthat.com> References: <20131015233219.GZ56872@funkthat.com> <20131017033400.GA31544@charmander> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20131017033400.GA31544@charmander> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Thu, 17 Oct 2013 10:27:45 -0700 (PDT) Cc: freebsd-dtrace@freebsd.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Oct 2013 17:27:51 -0000 Mark Johnston wrote this message on Wed, Oct 16, 2013 at 23:34 -0400: > On Tue, Oct 15, 2013 at 04:32:19PM -0700, John-Mark Gurney wrote: > > Well, I've decided to try to learn dtrace to do some benchmarking, and > > so I tried to use -c to measure what the command does... except it seems > > to fail... > > > > even the simple: > > dtrace -n 'syscall:::entry { @num[execname] = count(); }' -c 'echo foo' > > > > doesn't work... it gives me: > > # dtrace -n 'syscall:::entry { @num[execname] = count(); }' -c 'echo foo' > > foo > > dtrace: failed to control pid 3766: process exited with status 0 > > > > ktrace shows it execing /bin/echo and it running fine, but for some > > reason dtrace can't handle it... > > My guess is that the /bin/echo on your system is stripped. dtrace(1) on > FreeBSD will set a breakpoint on main() in the victim process and > register the script with the kernel when that breakpoint fires. If > libproc can't find the address of main(), the breakpoint won't fire, and > your script won't run. If /bin/echo isn't stripped, your example works > properly on my laptop. Adding '-x evaltime=postinit' to the dtrace(1) > flags should also allow the script to run properly. We might want to document this somewhere since by default FreeBSD doesn't install debug binaries (maybe when it fails?), so by default dtrace won't be very useful for people... > avg@ tried to fix this a little while ago by changing dtrace to > instead put a breakpoint on r_debug_state in rtld (r248644). This works > for your example, but breaks USDT since that breakpoint apparently fires > before ELF ctors run, and thus before dtrace_dof_init() can run (which is > needed for USDT to work). > > I'm not sure what the best way to fix this is. Perhaps we could add a > second breakpoint to rtld for use by dtrace, or maybe there's some way > to set a breakpoint on the DOF init code. > > It would also be nice if libproc could learn to follow .gnu_debuglink > sections when performing symbol lookup. I use WITH_DEBUG_FILES on my > laptop to keep debug info in separate files, but I still have the same > problem as you with the above example. I'll work on fixing this too. Wow, that is anoying to fix... Good luck.. > > P.S. Why are dtrace and dtraceall seperate? > > dtrace.ko contains the "core" DTrace kernel code. Most of the actual > providers and support code (e.g. fasttrap, sdt) are implemented in their > own modules. dtraceall depends on all the DTrace-related modules, so it's > almost always better to kldload dtraceall instead of kldloading dtrace. > In fact, dtrace(1) was recently changed to automatically kldload > dtraceall if needed, so there's no reason to manually kldload anything > now. Ahh, yep, see that it works on my head machine, nice.. :) -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-dtrace@FreeBSD.ORG Thu Oct 17 20:06:59 2013 Return-Path: Delivered-To: freebsd-dtrace@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 86A32CE1 for ; Thu, 17 Oct 2013 20:06:59 +0000 (UTC) (envelope-from jmg@h2.funkthat.com) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 6B0482EDC for ; Thu, 17 Oct 2013 20:06:59 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id r9HK6wp7078513 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 17 Oct 2013 13:06:58 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id r9HK6wfn078512 for freebsd-dtrace@FreeBSD.org; Thu, 17 Oct 2013 13:06:58 -0700 (PDT) (envelope-from jmg) Date: Thu, 17 Oct 2013 13:06:58 -0700 From: John-Mark Gurney To: freebsd-dtrace@FreeBSD.org Subject: new issues w/ dtrace aborting... Message-ID: <20131017200658.GG56872@funkthat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Thu, 17 Oct 2013 13:06:59 -0700 (PDT) X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Oct 2013 20:06:59 -0000 I'm see this failure which is reproducable: # dtrace -s ./disklatencycmd.d -x evaltime=postinit -c ./catall Tracing... Hit Ctrl-C to end. Now, I have spent some time trying to debug this error and have gotten a stack trace: #0 dt_divide_128 (dividend=0x7fffffffd240, divisor=0, quotient=0x7fffffffd240) at /usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_consume.c:221 #1 0x0000000800a8d343 in dt_stddev (data=0x802bb3e48, normal=1) at /usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_consume.c:372 #2 0x0000000800a9549a in dt_aggregate_valcmp (lhs=, rhs=) at /usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_aggregate.c:120 #3 0x0000000800a93821 in dt_aggregate_varvalcmp (lhs=0x804679240, rhs=0x804679260) at /usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_aggregate.c:950 #4 0x0000000801a3d6aa in qsort () from /lib/libc.so.7 #5 0x0000000800a93c88 in dt_aggregate_walk_sorted (dtp=0x80281d000, func=0x800a8f7d0 , arg=0x7fffffffd498, sfunc=) at /usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_aggregate.c:1275 #6 0x0000000800a951c6 in dtrace_aggregate_print (dtp=0x80281d000, fp=, func=) at /usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_aggregate.c:1880 #7 0x000000000040387f in main (argc=, argv=0x7fffffffd700) at /usr/src/cddl/usr.sbin/dtrace/../../../cddl/contrib/opensolaris/cmd/dtrace/dtrace.c:1921 Now the weird part is that between frame 3 and frame 0, the address of lhs/data changes, yet if you look at the code, it is passing the pointer straight through w/o modification... The value pointed to by lhs is valid and non-zero, but in frame 0 the different pointer is now zero.. The script catall: #!/bin/sh - #since dtrace has issues: find /mnt -type f -exec cat {} + > /dev/null The directory /mnt has a FS that only contains a recent export of HEAD.. I also umount/mount before each run to make sure the disk cache is clear, otherwise it's possible that all the data will be cached in memory, and not perform any io... It looks like it's an issue w/ clear(@stddev) as this is the script I've reduced it to reproduce the failiure: #pragma D option quiet #pragma D option dynvarsize=16m io:::start { start_time[arg0] = timestamp; } io:::done /this->start = start_time[arg0]/ { this->delta = (timestamp - this->start) / 1000; @stddev[args[1]->device_name, args[1]->unit_number] = stddev(this->delta ); start_time[arg0] = 0; } dtrace:::END { clear(@stddev); } Obviously for this to be useful, you'd print out the stddev, but the abort happens either way... Thanks. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."