From owner-freebsd-questions@FreeBSD.ORG Tue Feb 28 20:24:42 2012 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DDB521065670 for ; Tue, 28 Feb 2012 20:24:42 +0000 (UTC) (envelope-from msabramo@gmail.com) Received: from mail-tul01m020-f182.google.com (mail-tul01m020-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 9E4EB8FC17 for ; Tue, 28 Feb 2012 20:24:41 +0000 (UTC) Received: by obbwc7 with SMTP id wc7so2717294obb.13 for ; Tue, 28 Feb 2012 12:24:41 -0800 (PST) Received-SPF: pass (google.com: domain of msabramo@gmail.com designates 10.60.28.10 as permitted sender) client-ip=10.60.28.10; Authentication-Results: mr.google.com; spf=pass (google.com: domain of msabramo@gmail.com designates 10.60.28.10 as permitted sender) smtp.mail=msabramo@gmail.com; dkim=pass header.i=msabramo@gmail.com Received: from mr.google.com ([10.60.28.10]) by 10.60.28.10 with SMTP id x10mr7071633oeg.71.1330460681307 (num_hops = 1); Tue, 28 Feb 2012 12:24:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=Fu+Ho4IYxE7HxcE6ph2nXqCRb+meJyyDWW5tlEsRbnw=; b=P3qcrWd9MZ0ymqWmFt1HAAMMmm/O7WmsrfNiQ/oslkIJPU+7EKLwMOyOmICciNSFn8 QXInXpQ5RqbIuvPGaTpG9UFG15A89YIUVprmbbL6MXAUsGw7QGzQNqaG/rCQ28VU7ioT JN6CE7SVfxr8Ji90ci2mIIan55Z0sEbdFTq6o= MIME-Version: 1.0 Received: by 10.60.28.10 with SMTP id x10mr6296537oeg.71.1330460681213; Tue, 28 Feb 2012 12:24:41 -0800 (PST) Received: by 10.60.24.7 with HTTP; Tue, 28 Feb 2012 12:24:41 -0800 (PST) In-Reply-To: References: Date: Tue, 28 Feb 2012 12:24:41 -0800 Message-ID: From: Marc Abramowitz To: freebsd-questions@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: DTrace userland X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Feb 2012 20:24:42 -0000 Here's another way to cause a kernel panic: [marca@freebsd9-0 ~]$ sudo kldload dtraceall [marca@freebsd9-0 ~]$ cat -n test.c 1 #include 2 3 int main() 4 { 5 sleep(15); 6 7 FILE *fp = fopen("hello.txt", "w"); 8 fprintf(fp, "Here I am at %s:%d.\n", __FILE__, __LINE__); 9 fclose(fp); 10 } [marca@freebsd9-0 ~]$ gcc test.c -o test [marca@freebsd9-0 ~]$ sudo dtrace -n 'pid$target:test:main:entry' -c ./test dtrace: description 'pid$target:test:main:entry' matched 1 probe dtrace: buffer size lowered to 1m CPU ID FUNCTION:NAME 0 43030 main:entry (Kernel panic! After reboot....) [marca@freebsd9-0 ~]$ cat hello.txt Here I am at test.c:8. Interestingly, the crash doesn't occur until after the sleep and the fprintf call, so it looks the kernel panic happens as a result of the traced process _exiting_... Marc On Mon, Feb 27, 2012 at 11:10 PM, Marc Abramowitz wrote: > Another strange behavior: > > [Tab 1] > $ /bin/sleep 300 & > [1] 1806 > > [Tab 2] > $ sudo dtrace -n 'pid1806:sleep::entry' > $ echo $? > 158 > > [Tab 1] > [1]+ Killed: 9 /bin/sleep 300 > > Something seems very wrong that DTrace is killing processes and causing > kernel panics. > > Marc > > On Mon, Feb 27, 2012 at 10:22 PM, Marc Abramowitz wrote: > >> I'm using FreeBSD 9.0 on amd64 in VMware Fusion and trying to DTrace >> userland programs. I think I must be doing something wrong. >> >> I recompiled my kernel and world, following the instructions at >> http://wiki.freebsd.org/DTrace and I've read >> http://wiki.freebsd.org/DTrace/userland: >> >> The test.c pid provider example worked fine for me: >> >> $ sudo dtrace -s pid.d -c ./test >> dtrace: script 'pid.d' matched 2 probes >> dtrace: buffer size lowered to 1m >> CPU ID FUNCTION:NAME >> 0 43030 main:entry >> 0 43031 sleep:entry >> 0 43031 sleep:entry >> 0 43031 sleep:entry >> >> As does a simple probe of test.c specified with the -n option: >> >> [marca@freebsd9-0 ~]$ sudo dtrace -n 'pid$target:test:main:entry' -c >> ./test >> dtrace: description 'pid$target:test:main:entry' matched 1 probe >> dtrace: buffer size lowered to 1m >> CPU ID FUNCTION:NAME >> 0 43030 main:entry >> >> When I start trying to dtrace other programs, things don't go so well... >> >> $ sudo dtrace -n ":::entry" -c /usr/local/bin/python >> Python 2.4.5 (#2, Dec 5 2011, 15:19:09) >> [GCC 4.2.1 20070831 patched [FreeBSD]] on freebsd9 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> import os >> >>> os.getpid() >> 1603 >> >>> >> dtrace: failed to control pid 1603: process exited with status 0 >> >> $ sudo dtrace -n 'pid$target:::entry' -c '/bin/cat hello_world.txt' >> dtrace: description 'pid$target:::entry' matched 3315 probes >> dtrace: buffer size lowered to 1m >> CPU ID FUNCTION:NAME >> 0 43448 _rtld_bind:entry >> 0 43903 rlock_acquire:entry >> 0 43125 def_thread_set_flag:entry >> (Had to hit Ctrl-C to exit; it never displayed hello_world.txt to stdout) >> >> [marca@freebsd9-0 /usr/ports/sysutils/coreutils]$ sudo make install >> ... >> [marca@freebsd9-0 /usr/ports/sysutils/coreutils]$ sudo dtrace -n >> 'pid$target:::entry' -c '/usr/local/bin/gcat config.log' >> dtrace: description 'pid$target:::entry' matched 3823 probes >> dtrace: buffer size lowered to 1m >> CPU ID FUNCTION:NAME >> 0 43524 _rtld_bind:entry >> 0 43979 rlock_acquire:entry >> 0 43201 def_thread_set_flag:entry >> ^C >> >> $ sudo dtrace -n 'pid$target:cat:main:entry' -c '/bin/cat hello_world.txt' >> causes a kernel panic. >> According to the core.txt file, it was a "Fatal trap 10: trace trap while >> in kernel mode" and here's the KDB backtrace: >> >> KDB: stack backtrace: >> #0 0xffffffff8089025e at kdb_backtrace+0x5e >> #1 0xffffffff80858ce7 at panic+0x187 >> #2 0xffffffff80b4bf20 at trap_fatal+0x290 >> #3 0xffffffff80b4c540 at trap+0x180 >> #4 0xffffffff80b36963 at calltrap+0x8 >> #5 0xffffffff8162583d at dtrace_assfail+0x2d >> #6 0xffffffff8188aa2e at fasttrap_provider_free+0x1de >> #7 0xffffffff8188ad13 at fasttrap_pid_cleanup_cb+0x1c3 >> #8 0xffffffff8086dfa1 at softclock+0x3a1 >> #9 0xffffffff8082d724 at intr_event_execute_handlers+0x104 >> #10 0xffffffff8082eee4 at ithread_loop+0xa4 >> #11 0xffffffff8082a34f at fork_exit+0x11f >> #12 0xffffffff80b36e8e at fork_trampoline+0xe >> >> [marca@freebsd9-0 /usr/ports/sysutils/coreutils]$ sudo dtrace -n >> 'pid$target:gcat::entry' -c '/usr/local/bin/gcat config.log' >> (Another kernel panic) >> >> I can provide full crash dumps if necessary. >> >> Any idea what's going on here? >> >> Cheers, >> Marc >> >> >> >> >