Date: Wed, 30 Oct 2013 12:37:16 +0000 From: symbolics@gmx.com To: dtrace@freebsd.org Subject: Re: Firefox crash during dtrace attach under -CURRENT Message-ID: <20131030123716.GA2037@lemon> In-Reply-To: <20131030081507.GA1674@lemon> References: <20131023203009.GA92945@lemon> <20131024025902.GA2286@charmander> <20131025104706.GB1705@lemon> <20131025145956.GA26814@lemon> <20131027195307.GA3206@charmander.uwaterloo.ca> <20131030081507.GA1674@lemon>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Oct 30, 2013 at 08:15:07AM +0000, symbolics@gmx.com wrote: > On Sun, Oct 27, 2013 at 03:53:07PM -0400, Mark Johnston wrote: > > On Fri, Oct 25, 2013 at 03:59:56PM +0100, symbolics@gmx.com wrote: > > > On Fri, Oct 25, 2013 at 11:47:06AM +0100, symbolics@gmx.com wrote: > > > > On Wed, Oct 23, 2013 at 10:59:02PM -0400, Mark Johnston wrote: > > > > > On Wed, Oct 23, 2013 at 09:30:09PM +0100, symbolics@gmx.com wrote: > > > > > > Hi, > > > > > > > > > > > > http://dtrace.org/blogs/brendan/2011/02/11/dtrace-pid-provider-arguments/ > > > > > > > > > > > > I tried to follow some of the examples but I crash the Firefox process > > > > > > each time. Sometimes DTrace manages to collect a little data before the > > > > > > death. > > > > > > > > > > > > [...] > > > > > > > > > > > > Is this a known problem or should I send a PR? > > > > > > > > > > Thanks for reporting this: I was able to reproduce the crash and managed > > > > > to find a nasty pair of bugs. Could you test the patch below and let me > > > > > know if it fixes the problem for you as well? If you see more crashes, > > > > > please include the backtrace and signo from gdb again; it would likely > > > > > be a different problem that needs to be debugged and fixed separately. > > > > > > > > Hi Mark, > > > > > > > > This helps but there still may be some issues. First time I used this > > > > I found that when I killed the DTrace process Firefox went down too > > > > with a SIGTRAP. I have a possibly unhelpful core from this: > > > > > > > > > > Another data point. I attached to mutt and reviewed some of the calls it > > > was making. Subsequently I killed DTrace, went to to look at other > > > things and a while later when back to check my mail. On attempting to > > > change into a different mail folder mutt died with a SIGTRAP. It seems > > > like DTrace isn't tidying up after itself? > > > > > > (gdb) bt > > > #0 0x0000000800722541 in r_debug_state (rd=0x802425480, m=0x7fffffff6c28) > > > at /usr/home/dm/git/freebsd/libexec/rtld-elf/rtld.c:3491 > > > #1 0x0000000000000000 in ?? () > > > > Ok, I think I've figured out this one too. As you note, dtrace(1) isn't > > cleaning up some of its breakpoints properly when it detaches. In > > particular, it's not stopping the victim process before it tries to > > remove breakpoints using ptrace(2); however, ptrace requires the target > > process to be stopped, else it will return EBUSY. So the breakpoint in > > the rtld gets left behind, and it turns out that r_debug_state() is called > > every time a process tries to dlopen() a shared object. > > > > mutt was a good example since it seems to dlopen() iconv-related stuff > > as I scan through my inbox; one can inspect this with DTrace. :) > > i.e. with something like > > > > 'pid$target::dlopen:entry {trace(copyinstr(arg0));}' > > > > With this observation it becomes easy to reproduce the problem using a > > test program that does something like > > > > while (1) { > > dlopen("/lib/libnonexistent.so.100", RTLD_LAZY); > > sleep(1); > > } > > > > A somewhat crude patch which fixes this for me is below; it just adds > > code to send SIGSTOP to the target process before trying to remove > > breakpoints. Does anyone see any problems with this? Perhaps it should > > be libproc's responsibility to ensure that the victim process is stopped > > before trying a ptrace(PT_IO, ...) to add/remove breakpoints? > > > > Thanks, > > -Mark > > > > Hi Mark, > > I've tried the patch but I can still reproduce the crash using mutt. I > attached to the running mutt process with dtruss and watched that work a > little bit, killed dtruss and carried on using mutt. I then tried to > open a large mail folder and it crashed with a SIGTRAP. Backtrace, FWIW: > > (gdb) bt > #0 0x0000000800722541 in r_debug_state (rd=0x802425480, m=0x7fffffff84a8) > at /usr/home/dm/git/freebsd/libexec/rtld-elf/rtld.c:3491 > #1 0x0000000000000001 in ?? () > #2 0x0000000102c00000 in ?? () > #3 0x0000000000000001 in ?? () > #4 0x00000002028000c0 in ?? () > #5 0x0000000000000001 in ?? () > #6 0x0000000000000002 in ?? () > #7 0x00007fffffff8d00 in ?? () > #8 0x0000000801f25e3c in arena_avail_insert (arena=0xffffffff, > chunk=0x7fffffff84a8, pageind=34395310320, npages=<value optimized out>, > maybe_adjac_pred=<value optimized out>, maybe_adjac_succ=false) > at /usr/obj/usr/home/dm/git/freebsd/lib/libc/jemalloc_arena.c:274 > Previous frame inner to this frame (corrupt stack?) > Current language: auto; currently minimal > Just quickly adding, this crash & trace is perfectly reproducible. I did a second round of kernel building and testing to make sure I hadn't messed up. --sym
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20131030123716.GA2037>