Date: Fri, 26 Mar 2010 09:41:29 -0400 From: John Baldwin <jhb@freebsd.org> To: freebsd-hackers@freebsd.org Cc: Dag-Erling =?iso-8859-15?q?Sm=F8rgrav?= <des@des.no>, Peter Steele <psteele@maxiscale.com> Subject: Re: Puzzling stack trace Message-ID: <201003260941.29154.jhb@freebsd.org> In-Reply-To: <7B9397B189EB6E46A5EE7B4C8A4BB7CB3B5AACDB@MBX03.exg5.exghost.com> References: <7B9397B189EB6E46A5EE7B4C8A4BB7CB3B5AACBE@MBX03.exg5.exghost.com> <86iq8jl0d7.fsf@ds4.des.no> <7B9397B189EB6E46A5EE7B4C8A4BB7CB3B5AACDB@MBX03.exg5.exghost.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Friday 26 March 2010 9:32:46 am Peter Steele wrote:
> >Type "frame 9" and see what it says. If the bug is easily reproducable,
try reproducing it with a debugging version of libc (buildworld with
> >DEBUG_FLAGS=-g)
>
> This crash happened at a production customer site--we have the core and the
matching binary and our logs for the application that crashed but that's all.
We've never seen this particular crash before and cannot reproduce it. The
fscanf() call that failed is repeated on a continual basis as part of a
monitoring thread, so literally thousands of this exact same call have been
made without incident.
>
> The "frame 9" command doesn't show anything useful:
>
> (gdb) frame 9
> #9 0x000000080127374b in memchr () from /lib/libc.so.7
>
> That's it. And yes, the stack trace appears to be wrong. Even the trace
starting from the vsscanf call is wrong. It says that __srget() is the next
function in the stack but vsscanf() doesn't call __srget():
>
> int
> vsscanf(const char * __restrict str, const char * __restrict fmt,
> __va_list ap)
> {
> FILE f;
>
> f._file = -1;
> f._flags = __SRD;
> f._bf._base = f._p = (unsigned char *)str;
> f._bf._size = f._r = strlen(str);
> f._read = eofread;
> f._ub._base = NULL;
> f._lb._base = NULL;
> f._orientation = 0;
> memset(&f._mbstate, 0, sizeof(mbstate_t));
> return (__svfscanf(&f, fmt, ap));
> }
>
> So it seems our application went completely out to lunch. This is
concerning.
Keep in mind that libc doesn't include symbols for static functions, so gdb
will just report the name of the preceding non-static function in the .text
section. If you have a source tree that matches this libc exactly you can
just build a new libc with debugging (make DEBUG_FLAGS=-g) and use that with
gdb to find the real stack trace in many cases. Also, you should see if
__svfscanf() calls __srget(). The __svfscanf() call frame may not show up in
gdb if the compiler re-used the callframe from vsscanf for __svfscanf() as an
optimization.
--
John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201003260941.29154.jhb>