From owner-freebsd-current@FreeBSD.ORG Wed Apr 6 20:31:24 2011 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4E560106566B; Wed, 6 Apr 2011 20:31:24 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id 23FE98FC12; Wed, 6 Apr 2011 20:31:23 +0000 (UTC) Received: from julian-mac.elischer.org (home-nat.elischer.org [67.100.89.137]) (authenticated bits=0) by vps1.elischer.org (8.14.4/8.14.4) with ESMTP id p36KVLTA065397 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 6 Apr 2011 13:31:22 -0700 (PDT) (envelope-from julian@freebsd.org) Message-ID: <4D9CCDB7.3000605@freebsd.org> Date: Wed, 06 Apr 2011 13:31:51 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9 MIME-Version: 1.0 To: John Baldwin References: <4D9A4CE5.5090900@freebsd.org> <4D9B7C92.6030901@freebsd.org> <201104060845.11771.jhb@freebsd.org> <4D9CA00C.20307@freebsd.org> In-Reply-To: <4D9CA00C.20307@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-current@freebsd.org, Justin Hibbits , Navdeep Parhar , FreeBSD Current Subject: Re: KGDB stack traces in the kernel. X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Apr 2011 20:31:24 -0000 On 4/6/11 10:17 AM, Julian Elischer wrote: > On 4/6/11 5:45 AM, John Baldwin wrote: >> On Tuesday, April 05, 2011 4:35:44 pm Navdeep Parhar wrote: >>> On Tue, Apr 5, 2011 at 1:33 PM, Julian >>> Elischer wrote: >>>> On 4/4/11 6:04 PM, Justin Hibbits wrote: >>>> >>> What does ddb do? It always seems to get this stuff correct. >> ddb knows to stop when it gets to a non-kernel address, and it uses >> string >> compares on function names to identify trap frames. For example in >> sys/amd64/amd64/db_trace.c: >> >> >> if (strcmp(name, "calltrap") == 0 || >> strcmp(name, "fork_trampoline") == 0 || >> strcmp(name, "nmi_calltrap") == 0 || >> strcmp(name, "Xdblfault") == 0) >> frame_type = TRAP; >> >> Hah, kgdb just needs to be updated (this is from trgt_amd64.c): >> >> const struct frame_unwind * >> kgdb_trgt_trapframe_sniffer(struct frame_info *next_frame) >> { >> char *pname; >> CORE_ADDR pc; >> >> pc = frame_pc_unwind(next_frame); >> pname = NULL; >> find_pc_partial_function(pc,&pname, NULL, NULL); >> if (pname == NULL) >> return (NULL); >> if (strcmp(pname, "calltrap") == 0 || >> strcmp(pname, "nmi_calltrap") == 0 || >> (pname[0] == 'X'&& pname[1] != '_')) >> return (&kgdb_trgt_trapframe_unwind); >> /* printf("%s: %lx =%s\n", __func__, pc, pname); */ >> return (NULL); >> } >> > > I'll give that a try > >> Can probably just add 'fork_trampoline' to that conditional. I >> think i386 >> needs a similar fix in kgdb. Not sure about other architectures: >> >> Index: trgt_amd64.c >> =================================================================== >> --- trgt_amd64.c (revision 220190) >> +++ trgt_amd64.c (working copy) >> @@ -184,6 +184,7 @@ >> if (pname == NULL) >> return (NULL); >> if (strcmp(pname, "calltrap") == 0 || >> + strcmp(pname, "fork_trampoline") == 0 || BTW this is in 8.x I don't have 9 on a machine I can test this with. While ddb finds fork_trampoline, (k)gdb does not. so I had to add fork_exit a well, but then it gets an error as well. firstly without fork_exit() #0 0xffffffff8144c529 in XXXXXXXX () from blah.ko #1 0xffffffff814dfea5 in XXXXXXXX () from blah.ko #2 0xffffff0063a1f600 in ?? () #3 0xffffffff814dfe30 in XXXXXXXX () from blah.ko #4 0xffffff0063a265f0 in ?? () #5 0xffffff8244290c30 in ?? () #6 0xffffffff805c4b8e in fork_exit (callout=0xffffffff814dfe30 , arg=0x89e475f700000000, frame=0xbad089e445890000) at ../../../kern/kern_fork.c:859 #7 0xf445c7a07d894868 in ?? () #8 0x095d77e800000000 in ?? () #9 0x00095c40e8c78900 in ?? () #10 0x000001bfd8458948 in ?? () [... etc forever] if I add for_exit, it does work but: #6 0xffffff82441c8c30 in ?? () #7 0xffffffff805c4b8e in fork_exit (callout=0xffffffff814dfe30 , arg=0x89e475f700000000, frame=0xbad089e445890000) at ../../../kern/kern_fork.c:859 Previous frame identical to this frame (corrupt stack?) that last error message makes ddd throw up its hands and get confused. it seems to occur all the time however. for example when you use sysctl debug.kdb.enter to enter the debugger, the same thing occurs: this easy to see. #0 kdb_enter (why=0xffffffff80a3494c "sysctl", msg=0xa
) at ../../../kern/subr_kdb.c:352 #1 0xffffffff80626bc9 in kdb_sysctl_enter (oidp=0xffffffff80c6c360, arg1=Variable "arg1" is not available. ) at ../../../kern/subr_kdb.c:174 #2 0xffffffff805fddc3 in sysctl_root (oidp=Variable "oidp" is not available. ) at ../../../kern/kern_sysctl.c:1420 #3 0xffffffff805fe00e in userland_sysctl (td=0xffffff0009a7c000, name=0xffffff824433fa20, namelen=3, old=0x0, oldlenp=Variable "oldlenp" is not available. ) at ../../../kern/kern_sysctl.c:1524 #4 0xffffffff805fe4ba in __sysctl (td=0xffffff0009a7c000, uap=0xffffff824433fbb0) at ../../../kern/kern_sysctl.c:1450 #5 0xffffffff8063429f in syscallenter (td=0xffffff0009a7c000, sa=0xffffff824433fba0) at ../../../kern/subr_trap.c:315 #6 0xffffffff808feafc in syscall (frame=0xffffff824433fc40) at ../../../amd64/amd64/trap.c:888 #7 0xffffffff808e6092 in Xfast_syscall () at ../../../amd64/amd64/exception.S:377 #8 0x000000080073db0c in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) do you have any idea what that is all about? looking at frames 7,8,6 I see: Previous frame inner to this frame (corrupt stack?) (kgdb) up 7 #7 0xffffffff808e6092 in Xfast_syscall () at ../../../amd64/amd64/exception.S:377 377 call syscall Current language: auto; currently asm (kgdb) info frame Stack level 7, frame at 0xffffff824433fc40: rip = 0xffffffff808e6092 in Xfast_syscall (../../../amd64/amd64/exception.S:377); saved rip 0x80073db0c called by frame at 0x7fffffffe190, caller of frame at 0xffffff824433fc40 source language asm. Arglist at 0xffffff824433fc38, args: Locals at 0xffffff824433fc38, Previous frame's sp at 0xffffff824433fcf0 Saved registers: rax at 0xffffff824433fc70, rbx at 0xffffff824433fc78, rcx at 0xffffff824433fc58, rdx at 0xffffff824433fc50, rsi at 0xffffff824433fc48, rdi at 0xffffff824433fc40, rbp at 0xffffff824433fc80, r8 at 0xffffff824433fc60, r9 at 0xffffff824433fc68, r10 at 0xffffff824433fc88, r11 at 0xffffff824433fc90, r12 at 0xffffff824433fc98, r13 at 0xffffff824433fca0, r14 at 0xffffff824433fca8, r15 at 0xffffff824433fcb0, rip at 0xffffff824433fcd8, eflags at 0xffffff824433fce8, cs at 0xffffff824433fce0, ss at 0xffffff824433fcf8 (kgdb) up #8 0x000000080073db0c in ?? () (kgdb) info frame Stack level 8, frame at 0x7fffffffe190: rip = 0x80073db0c; saved rip 0x4023a5 caller of frame at 0xffffff824433fc40 Arglist at 0x7fffffffe180, args: Locals at 0x7fffffffe180, Previous frame's sp is 0x7fffffffe190 Saved registers: rax at 0xffffff824433fc70, rbx at 0xffffff824433fc78, rcx at 0xffffff824433fc58, rdx at 0xffffff824433fc50, rsi at 0xffffff824433fc48, rdi at 0xffffff824433fc40, rbp at 0xffffff824433fc80, r8 at 0xffffff824433fc60, r9 at 0xffffff824433fc68, r10 at 0xffffff824433fc88, r11 at 0xffffff824433fc90, r12 at 0xffffff824433fc98, r13 at 0xffffff824433fca0, r14 at 0xffffff824433fca8, r15 at 0xffffff824433fcb0, rip at 0x7fffffffe188, eflags at 0xffffff824433fce8, cs at 0xffffff824433fce0, ss at 0xffffff824433fcf8 (kgdb) down #7 0xffffffff808e6092 in Xfast_syscall () at ../../../amd64/amd64/exception.S:377 377 call syscall (kgdb) down #6 0xffffffff808feafc in syscall (frame=0xffffff824433fc40) at ../../../amd64/amd64/trap.c:888 888 error = syscallenter(td, &sa); Current language: auto; currently c (kgdb) info frame Stack level 6, frame at 0xffffff824433fc40: rip = 0xffffffff808feafc in syscall (../../../amd64/amd64/trap.c:888); saved rip 0xffffffff808e6092 called by frame at 0xffffff824433fc40, caller of frame at 0xffffff824433fb30 source language c. Arglist at 0xffffff824433fc30, args: frame=0xffffff824433fc40 Locals at 0xffffff824433fc30, Previous frame's sp is 0xffffff824433fc40 Saved registers: rbx at 0xffffff824433fc08, rbp at 0xffffff824433fc30, r12 at 0xffffff824433fc10, r13 at 0xffffff824433fc18, r14 at 0xffffff824433fc20, r15 at 0xffffff824433fc28, rip at 0xffffff824433fc38 (kgdb) >> strcmp(pname, "nmi_calltrap") == 0 || >> (pname[0] == 'X'&& pname[1] != '_')) >> return (&kgdb_trgt_trapframe_unwind); >> Index: trgt_i386.c >> =================================================================== >> --- trgt_i386.c (revision 220190) >> +++ trgt_i386.c (working copy) >> @@ -374,6 +374,7 @@ >> if (strcmp(pname, "dblfault_handler") == 0) >> return (&kgdb_trgt_dblfault_unwind); >> if (strcmp(pname, "calltrap") == 0 || >> + strcmp(pname, "fork_trampoline") == 0 || >> (pname[0] == 'X'&& pname[1] != '_')) >> return (&kgdb_trgt_trapframe_unwind); >> /* printf("%s: %llx =%s\n", __func__, pc, pname); */ >> > > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to > "freebsd-current-unsubscribe@freebsd.org" >