Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 28 Feb 2012 12:24:41 -0800
From:      Marc Abramowitz <msabramo@gmail.com>
To:        freebsd-questions@freebsd.org
Subject:   Re: DTrace userland
Message-ID:  <CAAgNOZ0U52Va0gNWiXx3C_pTssqr-P+oEqSr0mwkzCvrQLEuMw@mail.gmail.com>
In-Reply-To: <CAAgNOZ2Vx9JsNnBmQaojFjfL7CTCbRTQm_bzT1-DPk3889QgGg@mail.gmail.com>
References:  <CAAgNOZ2niHqw6SLWUkmTch-bKh1HPYrUX20QvLYPF-SKZ7wg8A@mail.gmail.com> <CAAgNOZ2Vx9JsNnBmQaojFjfL7CTCbRTQm_bzT1-DPk3889QgGg@mail.gmail.com>

Next in thread | Previous in thread | Raw E-Mail | Index | Archive | Help
Here's another way to cause a kernel panic:

[marca@freebsd9-0 ~]$ sudo kldload dtraceall
[marca@freebsd9-0 ~]$ cat -n test.c
     1 #include <stdio.h>
     2
     3 int main()
     4 {
     5    sleep(15);
     6
     7    FILE *fp = fopen("hello.txt", "w");
     8    fprintf(fp, "Here I am at %s:%d.\n", __FILE__, __LINE__);
     9    fclose(fp);
    10 }
[marca@freebsd9-0 ~]$ gcc test.c -o test
[marca@freebsd9-0 ~]$ sudo dtrace -n 'pid$target:test:main:entry' -c ./test
dtrace: description 'pid$target:test:main:entry' matched 1 probe
dtrace: buffer size lowered to 1m
CPU     ID                    FUNCTION:NAME
  0  43030                       main:entry
(Kernel panic!  After reboot....)
[marca@freebsd9-0 ~]$ cat hello.txt
Here I am at test.c:8.

Interestingly, the crash doesn't occur until after the sleep and the
fprintf call, so it looks the kernel panic happens as a result of the
traced process _exiting_...

Marc


On Mon, Feb 27, 2012 at 11:10 PM, Marc Abramowitz <msabramo@gmail.com>wrote:

> Another strange behavior:
>
> [Tab 1]
> $ /bin/sleep 300 &
> [1] 1806
>
> [Tab 2]
> $ sudo dtrace -n 'pid1806:sleep::entry'
> $ echo $?
> 158
>
> [Tab 1]
> [1]+  Killed: 9               /bin/sleep 300
>
> Something seems very wrong that DTrace is killing processes and causing
> kernel panics.
>
> Marc
>
> On Mon, Feb 27, 2012 at 10:22 PM, Marc Abramowitz <msabramo@gmail.com>wrote:
>
>> I'm using FreeBSD 9.0 on amd64 in VMware Fusion and trying to DTrace
>> userland programs. I think I must be doing something wrong.
>>
>> I recompiled my kernel and world, following the instructions at
>> http://wiki.freebsd.org/DTrace and I've read
>> http://wiki.freebsd.org/DTrace/userland:
>>
>> The test.c pid provider example worked fine for me:
>>
>> $ sudo dtrace -s pid.d -c ./test
>> dtrace: script 'pid.d' matched 2 probes
>> dtrace: buffer size lowered to 1m
>> CPU     ID                    FUNCTION:NAME
>>   0  43030                       main:entry
>>   0  43031                      sleep:entry
>>   0  43031                      sleep:entry
>>   0  43031                      sleep:entry
>>
>> As does a simple probe of test.c specified with the -n option:
>>
>> [marca@freebsd9-0 ~]$ sudo dtrace -n 'pid$target:test:main:entry' -c
>> ./test
>> dtrace: description 'pid$target:test:main:entry' matched 1 probe
>> dtrace: buffer size lowered to 1m
>> CPU     ID                    FUNCTION:NAME
>>   0  43030                       main:entry
>>
>> When I start trying to dtrace other programs, things don't go so well...
>>
>> $ sudo dtrace -n ":::entry" -c /usr/local/bin/python
>> Python 2.4.5 (#2, Dec  5 2011, 15:19:09)
>> [GCC 4.2.1 20070831 patched [FreeBSD]] on freebsd9
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> import os
>> >>> os.getpid()
>> 1603
>> >>>
>> dtrace: failed to control pid 1603: process exited with status 0
>>
>> $ sudo dtrace -n 'pid$target:::entry' -c '/bin/cat hello_world.txt'
>> dtrace: description 'pid$target:::entry' matched 3315 probes
>> dtrace: buffer size lowered to 1m
>> CPU     ID                    FUNCTION:NAME
>>   0  43448                 _rtld_bind:entry
>>   0  43903              rlock_acquire:entry
>>   0  43125        def_thread_set_flag:entry
>> (Had to hit Ctrl-C to exit; it never displayed hello_world.txt to stdout)
>>
>> [marca@freebsd9-0 /usr/ports/sysutils/coreutils]$ sudo make install
>> ...
>> [marca@freebsd9-0 /usr/ports/sysutils/coreutils]$ sudo dtrace -n
>> 'pid$target:::entry' -c '/usr/local/bin/gcat config.log'
>> dtrace: description 'pid$target:::entry' matched 3823 probes
>> dtrace: buffer size lowered to 1m
>> CPU     ID                    FUNCTION:NAME
>>   0  43524                 _rtld_bind:entry
>>   0  43979              rlock_acquire:entry
>>   0  43201        def_thread_set_flag:entry
>> ^C
>>
>> $ sudo dtrace -n 'pid$target:cat:main:entry' -c '/bin/cat hello_world.txt'
>> causes a kernel panic.
>> According to the core.txt file, it was a "Fatal trap 10: trace trap while
>> in kernel mode" and here's the KDB backtrace:
>>
>> KDB: stack backtrace:
>> #0 0xffffffff8089025e at kdb_backtrace+0x5e
>> #1 0xffffffff80858ce7 at panic+0x187
>> #2 0xffffffff80b4bf20 at trap_fatal+0x290
>> #3 0xffffffff80b4c540 at trap+0x180
>> #4 0xffffffff80b36963 at calltrap+0x8
>> #5 0xffffffff8162583d at dtrace_assfail+0x2d
>> #6 0xffffffff8188aa2e at fasttrap_provider_free+0x1de
>> #7 0xffffffff8188ad13 at fasttrap_pid_cleanup_cb+0x1c3
>> #8 0xffffffff8086dfa1 at softclock+0x3a1
>> #9 0xffffffff8082d724 at intr_event_execute_handlers+0x104
>> #10 0xffffffff8082eee4 at ithread_loop+0xa4
>> #11 0xffffffff8082a34f at fork_exit+0x11f
>> #12 0xffffffff80b36e8e at fork_trampoline+0xe
>>
>> [marca@freebsd9-0 /usr/ports/sysutils/coreutils]$ sudo dtrace -n
>> 'pid$target:gcat::entry' -c '/usr/local/bin/gcat config.log'
>> (Another kernel panic)
>>
>> I can provide full crash dumps if necessary.
>>
>> Any idea what's going on here?
>>
>> Cheers,
>> Marc
>>
>>
>>
>>
>



Want to link to this message? Use this URL: <http://docs.FreeBSD.org/cgi/mid.cgi?CAAgNOZ0U52Va0gNWiXx3C_pTssqr-P+oEqSr0mwkzCvrQLEuMw>