From owner-freebsd-dtrace@freebsd.org Mon Mar 28 07:41:12 2016 Return-Path: Delivered-To: freebsd-dtrace@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E4430AE09E9 for ; Mon, 28 Mar 2016 07:41:12 +0000 (UTC) (envelope-from spon@sv1149.xserver.jp) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id D06BD1078 for ; Mon, 28 Mar 2016 07:41:12 +0000 (UTC) (envelope-from spon@sv1149.xserver.jp) Received: by mailman.ysv.freebsd.org (Postfix) id CF7A7AE09E4; Mon, 28 Mar 2016 07:41:12 +0000 (UTC) Delivered-To: dtrace@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CF1A0AE09E3 for ; Mon, 28 Mar 2016 07:41:12 +0000 (UTC) (envelope-from spon@sv1149.xserver.jp) Received: from sv1149.xserver.jp (sv1149.xserver.jp [183.90.228.50]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 799CB1074 for ; Mon, 28 Mar 2016 07:41:12 +0000 (UTC) (envelope-from spon@sv1149.xserver.jp) Received: from virusgw8.xserver.jp (virusgw8.xserver.jp [183.90.228.243]) by sv1149.xserver.jp (Postfix) with ESMTP id 3C7CA7B803B4 for ; Mon, 28 Mar 2016 16:41:10 +0900 (JST) Received: from sv1149.xserver.jp (183.90.228.50) by virusgw8.xserver.jp (F-Secure/fsigk_smtp/521/virusgw8.xserver.jp); Mon, 28 Mar 2016 16:41:09 +0900 (JST) X-Virus-Status: clean(F-Secure/fsigk_smtp/521/virusgw8.xserver.jp) Received: by sv1149.xserver.jp (Postfix, from userid 20198) id 288B97B803C5; Mon, 28 Mar 2016 16:41:10 +0900 (JST) To: dtrace@freebsd.org Subject: We could not deliver your parcel, #00592101 Date: Mon, 28 Mar 2016 16:41:10 +0900 From: "FedEx 2Day" Reply-To: "FedEx 2Day" Message-ID: X-Priority: 3 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Mar 2016 07:41:13 -0000 Dear Customer, We could not deliver your item. Please, download Delivery Label attached to this email. Kind regards, Vernon Hill, Sr. Operation Manager. From owner-freebsd-dtrace@freebsd.org Thu Mar 31 17:43:07 2016 Return-Path: Delivered-To: freebsd-dtrace@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B5437AE3891; Thu, 31 Mar 2016 17:43:07 +0000 (UTC) (envelope-from mail@eax.me) Received: from relay12.nicmail.ru (relay12.nicmail.ru [195.208.5.7]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 69B0F106F; Thu, 31 Mar 2016 17:43:06 +0000 (UTC) (envelope-from mail@eax.me) Received: from [109.70.25.214] (port=50579 helo=portege) by f06.mail.nic.ru with esmtp (Exim 5.55) (envelope-from ) id 1algch-000Ln7-0H; Thu, 31 Mar 2016 20:43:11 +0300 Received: from [188.123.231.37] (account mail@eax.me HELO portege) by proxy01.mail.nic.ru (Exim 5.55) with id 1algcT-0007Et-8Q; Thu, 31 Mar 2016 20:42:57 +0300 Date: Thu, 31 Mar 2016 20:42:56 +0300 From: Aleksander Alekseev To: Adrian Chadd Cc: Hans Petter Selasky , "freebsd-hackers@freebsd.org" , Andriy Voskoboinyk , freebsd-dtrace@freebsd.org Subject: Re: I need a little help in fixing `exclusive sleep mutex urtwn0_com_lock` in CURRENT Message-ID: <20160331204256.5cb1fdaf@portege> In-Reply-To: References: <20160330123048.3361a9e4@fujitsu> <56FBBC62.6040905@selasky.org> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Mar 2016 17:43:07 -0000 > So as I understand some code acquired a mutex and didn't release it. > And naturally it has something to do with USB. Now how can I figure > out what code acquired this lock? Best approach I can think of > currently is to find all places where this lock is accessed and add a > debug logging. But perhaps there is a better way? By any chance is it possible to trace mtx_lock / mtx_unlock calls using DTrace? I see number of probes in `dtrace -l` which look like something I need. Unfortunately they are named like knlist_mtx_lock or do_lock_umutex so I can't figure out whether these are probes I looking for or not. cc: freebsd-dtrace@ -- Best regards, Aleksander Alekseev http://eax.me/ From owner-freebsd-dtrace@freebsd.org Thu Mar 31 18:00:30 2016 Return-Path: Delivered-To: freebsd-dtrace@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 26FE5AE3FF6; Thu, 31 Mar 2016 18:00:30 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-pa0-x229.google.com (mail-pa0-x229.google.com [IPv6:2607:f8b0:400e:c03::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E9FCD18AC; Thu, 31 Mar 2016 18:00:29 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-pa0-x229.google.com with SMTP id fe3so71266638pab.1; Thu, 31 Mar 2016 11:00:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=8FZTPvShhj+3LK7LHN9fNMQK6zadYVqSOepvY9OlhkE=; b=0Rvk+i5fTf4txXNnjnlLNg+0edxdsk7YiAhFwnpH8zZYY3omCqFi/B+5CLSK5awtOS buRH1zawhPg/IaH8JMC8XSaefxp7BNkWyIDmUNthT3HuLynAtRWsJ+U/0sXSemArAdc1 63ZBIo0CVwxI53r9MiN8YIr40IbOHmJO2BCBVWx1V5eZ3MiySCaMgdzq45wVV+5yM7zC wbcZLFHesj4b2fpqtM8Xz4Mevww2pcHHbowan9yynXZv5ed/MRi1vnPH6qihdCqKOZ/L 1QAkpRHY0H0qii2Y7lHtYQZbgznQ3qFO2vuGPRKw4JkipBpiU+CWPq+YRip44kxIeW8G khmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=8FZTPvShhj+3LK7LHN9fNMQK6zadYVqSOepvY9OlhkE=; b=Wd2qO8hiuoAZbD5YrCT+39/Dwhm+yTxnw0n1yw+oTaKpsnJ9JOzQMXF8gyljwhgi8c T9rU2Hf0UK1UJUiCkqdb2IaVoWjFeg8/MyLYrEna+XcCJ1cXKXi9U12QGU04FR5o2qq/ 5ZRZ84Gwe8+cdYe1utXjTNah3y/kTvbEaWkmBWbZL2EjWFAwggtbesjk5690/oZShVC7 5T37UI6PBd4+j8/VL2G69O6MBRrXey/76DgyHygj8HBohhusNqyXNKdjDvXb42mO1QIG GmnkJe+tVPdWuHSoSIlnTpnUeMFBd/SF7IOmp//Zka3bvdzOcdRCFEKpdwIpSh3eC8is TOig== X-Gm-Message-State: AD7BkJJ/LS4fY+t/pfEJ1Svk5LSE3XCB1wYzViTqNG6TxaH3ps+I9vB9Noff+FEPcKwAUQ== X-Received: by 10.67.30.163 with SMTP id kf3mr24023412pad.45.1459447229402; Thu, 31 Mar 2016 11:00:29 -0700 (PDT) Received: from wkstn-mjohnston.west.isilon.com (c-67-182-131-225.hsd1.wa.comcast.net. [67.182.131.225]) by smtp.gmail.com with ESMTPSA id e12sm14976613pat.19.2016.03.31.11.00.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 31 Mar 2016 11:00:28 -0700 (PDT) Sender: Mark Johnston Date: Thu, 31 Mar 2016 11:03:33 -0700 From: Mark Johnston To: Aleksander Alekseev Cc: Adrian Chadd , Hans Petter Selasky , "freebsd-hackers@freebsd.org" , Andriy Voskoboinyk , freebsd-dtrace@freebsd.org Subject: Re: I need a little help in fixing `exclusive sleep mutex urtwn0_com_lock` in CURRENT Message-ID: <20160331180333.GA25235@wkstn-mjohnston.west.isilon.com> References: <20160330123048.3361a9e4@fujitsu> <56FBBC62.6040905@selasky.org> <20160331204256.5cb1fdaf@portege> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160331204256.5cb1fdaf@portege> User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Mar 2016 18:00:30 -0000 On Thu, Mar 31, 2016 at 08:42:56PM +0300, Aleksander Alekseev wrote: > > So as I understand some code acquired a mutex and didn't release it. > > And naturally it has something to do with USB. Now how can I figure > > out what code acquired this lock? Best approach I can think of > > currently is to find all places where this lock is accessed and add a > > debug logging. But perhaps there is a better way? > > By any chance is it possible to trace mtx_lock / mtx_unlock calls using > DTrace? I see number of probes in `dtrace -l` which look like > something I need. Unfortunately they are named like knlist_mtx_lock or > do_lock_umutex so I can't figure out whether these are probes I looking > for or not. Yes. You can use the lockstat provider to trace lock events: "dtrace -l -P lockstat". Predicates can be used to isolate events related to a specific lock. For example: # dtrace -n 'lockstat:::adaptive-acquire /args[0]->lock_object.lo_name == "so_snd"/{stack();}' will print a stack every time a socket send buffer lock is acquired. In general, the lock name is specified as an initialization parameter. In your case it'll be "urtwn0_com_lock". lockstat(1) is a command-line program that's good at aggregating data collected from lockstat probes; for the type of debugging you're doing it's probably not very useful. Note that DTrace probably isn't very helpful here if the panic occurs immediately after the event you're interested in occurs, since dtrace(1) won't have time to retrieve the trace record and print it. From owner-freebsd-dtrace@freebsd.org Thu Mar 31 21:23:48 2016 Return-Path: Delivered-To: freebsd-dtrace@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F29ADAE38D0; Thu, 31 Mar 2016 21:23:48 +0000 (UTC) (envelope-from mail@eax.me) Received: from relay15.nicmail.ru (relay15.nicmail.ru [195.208.5.161]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A6EC61873; Thu, 31 Mar 2016 21:23:47 +0000 (UTC) (envelope-from mail@eax.me) Received: from [109.70.25.187] (port=48124 helo=portege) by f19.mail.nic.ru with esmtp (Exim 5.55) (envelope-from ) id 1alk41-000AjB-CT; Fri, 01 Apr 2016 00:23:37 +0300 Received: from [188.123.231.37] (account mail@eax.me HELO portege) by proxy08.mail.nic.ru (Exim 5.55) with id 1alk41-0007vx-QO; Fri, 01 Apr 2016 00:23:37 +0300 Date: Fri, 1 Apr 2016 00:23:37 +0300 From: Aleksander Alekseev To: Mark Johnston Cc: Adrian Chadd , Hans Petter Selasky , Andriy Voskoboinyk , freebsd-dtrace@freebsd.org, "freebsd-hackers@freebsd.org" Subject: Re: I need a little help in fixing `exclusive sleep mutex urtwn0_com_lock` in CURRENT Message-ID: <20160401002337.36700d9b@portege> In-Reply-To: <20160331180333.GA25235@wkstn-mjohnston.west.isilon.com> References: <20160330123048.3361a9e4@fujitsu> <56FBBC62.6040905@selasky.org> <20160331204256.5cb1fdaf@portege> <20160331180333.GA25235@wkstn-mjohnston.west.isilon.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Mar 2016 21:23:49 -0000 > > By any chance is it possible to trace mtx_lock / mtx_unlock calls > > using DTrace? I see number of probes in `dtrace -l` which look like > > something I need. Unfortunately they are named like knlist_mtx_lock > > or do_lock_umutex so I can't figure out whether these are probes I > > looking for or not. > > Yes. You can use the lockstat provider to trace lock events: > "dtrace -l -P lockstat". Predicates can be used to isolate events > related to a specific lock. For example: > > # dtrace -n > 'lockstat:::adaptive-acquire /args[0]->lock_object.lo_name == > "so_snd"/{stack();}' > > will print a stack every time a socket send buffer lock is acquired. > In general, the lock name is specified as an initialization parameter. > In your case it'll be "urtwn0_com_lock". > > lockstat(1) is a command-line program that's good at aggregating data > collected from lockstat probes; for the type of debugging you're doing > it's probably not very useful. > > Note that DTrace probably isn't very helpful here if the panic occurs > immediately after the event you're interested in occurs, since > dtrace(1) won't have time to retrieve the trace record and print it. Thanks, Mark. Here is a funny thing. I thought that by writing debug.witness.watch=0 to /etc/sysctl.conf and rebooting I will give DTrace some time to report who acquired a lock and didn't released it. But in this case kernel crashed differently (I checked twice): http://pastebin.com/raw/f08Fd70B And here is saved DTrace log for lock "urtwn0_com_lock": http://pastebin.com/raw/0X9zgeiY If I'm not wrong everything is OK. No missing mtx_unlock this time. Not sure what to do next so far. And it's pretty late here in UTC+3 already so I will think about this tomorrow. Any advices would be appreciated. -- Best regards, Aleksander Alekseev http://eax.me/ From owner-freebsd-dtrace@freebsd.org Thu Mar 31 22:56:00 2016 Return-Path: Delivered-To: freebsd-dtrace@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0277CAE4CBB; Thu, 31 Mar 2016 22:56:00 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-pa0-x230.google.com (mail-pa0-x230.google.com [IPv6:2607:f8b0:400e:c03::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C55481D85; Thu, 31 Mar 2016 22:55:59 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-pa0-x230.google.com with SMTP id td3so75467517pab.2; Thu, 31 Mar 2016 15:55:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=wF0ojhRAoZ5w6Cg97vdSCdVmt2jf573b43JF3MITYZE=; b=IUpiKC9cfS2pT2URcOU4xGneXxMGcgBvBhD2Ceaz66nkT0OjJnJth59MP5P+jKApiz DSr/GxjfmVOfI1IRzFgg8RqoIOZ7ldxWgBUTGWAbqqknsACpGWRNPFmIpAv/qK2EOKl1 WRNRiIIi96ca7YQe/FzvIJNWq3qK3ZQd+MIM5gNWcoZqIVFRexBnQo7O5vrPGYFtr4pa 9cLWaMDPALZUdcb+CTFds2sX9ypxJhBiHMQFtwj+SHpcD4mdmpm3lirruTqh/dlO+DWN uparZ6T9VkQg5l3fxL7IpG6ZlOXR3tHLyYZZ/XP40SuO4wkpAi8p9REu+z3mr0TLAnxm eSFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=wF0ojhRAoZ5w6Cg97vdSCdVmt2jf573b43JF3MITYZE=; b=H4uQBITO+uTc1iPiz664UKKY6+nCi7wRqLqhJfIsqMq28Jzg2XNrYGZbj5rFAECo0F Wa6Ghla36Y9XNFQog870nmezRd2eDcdbqBXuSJBsPKb3nbBB7BJhrezphYaDvjesnQno EIwyI4/hwXjUhIQ77dDGja34cjIs9FL/Dy+GghNpnDD1xDKELNUdPcQDun0/bSLvthcC tknO50pLMzmcZEPRdZH/Q/nLtrA3j6TMOP/7FfTc373sRCMgoRX+/q8KthhLPHXOogHd UHTiMUoDiGFSDn6wN56z0R91nw0UZc+/zi7FajSV1UchqHTSkIa04vuyePK0CCPBkscX SbCg== X-Gm-Message-State: AD7BkJLC7Wqo//Du/I++zDJWeYpza/nv4qamIwTwsoEqZov+Bz5cXChcrL6ye4SY0BHpCg== X-Received: by 10.66.254.168 with SMTP id aj8mr25866829pad.18.1459464959409; Thu, 31 Mar 2016 15:55:59 -0700 (PDT) Received: from wkstn-mjohnston.west.isilon.com (c-67-182-131-225.hsd1.wa.comcast.net. [67.182.131.225]) by smtp.gmail.com with ESMTPSA id d12sm15709164pfj.85.2016.03.31.15.55.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 31 Mar 2016 15:55:58 -0700 (PDT) Sender: Mark Johnston Date: Thu, 31 Mar 2016 15:59:02 -0700 From: Mark Johnston To: Aleksander Alekseev Cc: Adrian Chadd , Hans Petter Selasky , Andriy Voskoboinyk , freebsd-dtrace@freebsd.org, "freebsd-hackers@freebsd.org" Subject: Re: I need a little help in fixing `exclusive sleep mutex urtwn0_com_lock` in CURRENT Message-ID: <20160331225902.GA32570@wkstn-mjohnston.west.isilon.com> References: <20160330123048.3361a9e4@fujitsu> <56FBBC62.6040905@selasky.org> <20160331204256.5cb1fdaf@portege> <20160331180333.GA25235@wkstn-mjohnston.west.isilon.com> <20160401002337.36700d9b@portege> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160401002337.36700d9b@portege> User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Mar 2016 22:56:00 -0000 On Fri, Apr 01, 2016 at 12:23:37AM +0300, Aleksander Alekseev wrote: > > > By any chance is it possible to trace mtx_lock / mtx_unlock calls > > > using DTrace? I see number of probes in `dtrace -l` which look like > > > something I need. Unfortunately they are named like knlist_mtx_lock > > > or do_lock_umutex so I can't figure out whether these are probes I > > > looking for or not. > > > > Yes. You can use the lockstat provider to trace lock events: > > "dtrace -l -P lockstat". Predicates can be used to isolate events > > related to a specific lock. For example: > > > > # dtrace -n > > 'lockstat:::adaptive-acquire /args[0]->lock_object.lo_name == > > "so_snd"/{stack();}' > > > > will print a stack every time a socket send buffer lock is acquired. > > In general, the lock name is specified as an initialization parameter. > > In your case it'll be "urtwn0_com_lock". > > > > lockstat(1) is a command-line program that's good at aggregating data > > collected from lockstat probes; for the type of debugging you're doing > > it's probably not very useful. > > > > Note that DTrace probably isn't very helpful here if the panic occurs > > immediately after the event you're interested in occurs, since > > dtrace(1) won't have time to retrieve the trace record and print it. > > Thanks, Mark. > > Here is a funny thing. I thought that by writing debug.witness.watch=0 > to /etc/sysctl.conf and rebooting I will give DTrace some time to > report who acquired a lock and didn't released it. But in this case > kernel crashed differently (I checked twice): > > http://pastebin.com/raw/f08Fd70B > > And here is saved DTrace log for lock "urtwn0_com_lock": > > http://pastebin.com/raw/0X9zgeiY > > If I'm not wrong everything is OK. No missing mtx_unlock this time. Looking at your original post, the problem isn't related to a missing unlock. The attempt to lock the ieee80211com is triggering a fault, presumably because it was freed while the lock was dropped. So DTrace isn't going to help much here.