From owner-freebsd-stable@FreeBSD.ORG Thu Oct 26 18:38:06 2006 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 09AD116A416 for ; Thu, 26 Oct 2006 18:38:06 +0000 (UTC) (envelope-from jim@thehousleys.net) Received: from sccrmhc11.comcast.net (sccrmhc11.comcast.net [204.127.200.81]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6F39943D75 for ; Thu, 26 Oct 2006 18:37:54 +0000 (GMT) (envelope-from jim@thehousleys.net) Received: from cat.int.thehousleys.net ([24.62.118.137]) by comcast.net (sccrmhc11) with ESMTP id <2006102618375301100cg8jle>; Thu, 26 Oct 2006 18:37:53 +0000 Received: from [192.168.128.235] (pied.int.thehousleys.net [192.168.128.235]) by cat.int.thehousleys.net (Postfix) with ESMTP id 9EDAC3125E for ; Thu, 26 Oct 2006 14:37:53 -0400 (EDT) Mime-Version: 1.0 (Apple Message framework v752.2) Content-Transfer-Encoding: quoted-printable Message-Id: Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed To: freebsd-stable@freebsd.org From: James Housley Date: Thu, 26 Oct 2006 14:37:52 -0400 X-Mailer: Apple Mail (2.752.2) Subject: Kernel crashes in current 6.2-PRERELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Oct 2006 18:38:06 -0000 My router running FBSD 6.1-RELEASE, 6.1-STABLE and now 6.2-PRERELEASE =20= periodically would reboot. It would always have the same basic =20 information in the kernel panic. I have finally gotten it setup for =20 a debug kernel and a place to store the crash. Here is the information, what are my next steps. Jim Script started on Thu Oct 26 14:28:46 2006 router# kgdb kernel.debug /var/crash/vmcore.0 kgdb: kvm_nlist(_stopped_cpus): kgdb: kvm_nlist(_stoppcbs): [GDB will not be able to debug user-mode threads: /usr/lib/=20 libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and =20 you are welcome to change it and/or distribute copies of it under certain =20 conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for =20 details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode fault virtual address =3D 0x1104768 fault code =3D supervisor read, page not present instruction pointer =3D 0x20:0xc04e50a1 stack pointer =3D 0x28:0xcc680b08 frame pointer =3D 0x28:0xcc680b0c code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, def32 1, gran 1 processor eflags =3D resume, IOPL =3D 0 current process =3D 3690 (ifconfig) trap number =3D 12 panic: page fault Uptime: 5h36m14s Dumping 126 MB (2 chunks) chunk 0: 1MB (160 pages) ... ok chunk 1: 126MB (32240 pages) 110 94 78 62 46 30 14 #0 doadump () at pcpu.h:165 165 __asm __volatile("movl %%fs:0,%0" : "=3Dr" (td)); (kgdb) list *0xc04e50a1 0xc04e50a1 is in turnstile_setowner (/usr/src/sys/kern/=20 subr_turnstile.c:433). 428 429 mtx_assert(&td_contested_lock, MA_OWNED); 430 MPASS(owner->td_proc->p_magic =3D=3D P_MAGIC); 431 MPASS(ts->ts_owner =3D=3D NULL); 432 ts->ts_owner =3D owner; 433 LIST_INSERT_HEAD(&owner->td_contested, ts, ts_link); 434 } 435 436 /* 437 * Malloc a turnstile for a new thread, initialize it and =20 return it. (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc04c4b02 in boot (howto=3D260) at /usr/src/sys/kern/=20 kern_shutdown.c:409 #2 0xc04c4d98 in panic (fmt=3D0xc06683c3 "%s") at /usr/src/sys/kern/kern_shutdown.c:565 #3 0xc0647000 in trap_fatal (frame=3D0xcc680ac8, eva=3D17844072) at /usr/src/sys/i386/i386/trap.c:837 #4 0xc06467e2 in trap (frame=3D {tf_fs =3D -1060962296, tf_es =3D -865599448, tf_ds =3D = -1067515864, =20 tf_edi =3D -1067971588, tf_esi =3D -1050101120, tf_ebp =3D -865596660, =20= tf_isp =3D -865596684, tf_ebx =3D -1050353088, tf_edx =3D -1050353088, =20= tf_ecx =3D 17843956, tf_eax =3D -1050101088, tf_trapno =3D 12, tf_err =3D = 0, =20 tf_eip =3D -1068609375, tf_cs =3D 32, tf_eflags =3D 65543, tf_esp =3D =20= -1050101120, tf_ss =3D -865596624}) at /usr/src/sys/i386/i386/trap.c:270 #5 0xc0635a9a in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #6 0xc04e50a1 in turnstile_setowner (ts=3D0xc164e240, owner=3D0x11046f4) at /usr/src/sys/kern/subr_turnstile.c:432 #7 0xc04e5398 in turnstile_wait (lock=3D0xc0580e5c, owner=3D0x11046f4) at /usr/src/sys/kern/subr_turnstile.c:591 #8 0xc04bb284 in _mtx_lock_sleep (m=3D0xc0580e5c, tid=3D3244866176, = opts=3D0, file=3D0x0, line=3D0) at /usr/src/sys/kern/kern_mutex.c:579 #9 0xc05327df in if_delmulti (ifp=3D0xc0580bfc, sa=3D0x262beca) at /usr/src/sys/net/if.c:2083 #10 0xc0581783 in in6_delmulti (in6m=3D0xc1980980) at /usr/src/sys/netinet6/mld6.c:649 #11 0xc05731c0 in in6_ifdetach (ifp=3D0xc15ba400) at /usr/src/sys/netinet6/in6_ifattach.c:806 #12 0xc05301f4 in if_detach (ifp=3D0xc15ba400) at /usr/src/sys/net/if.c:=20= 665 #13 0xc0535508 in gif_destroy (sc=3D0xc19a3900) at /usr/src/sys/net/=20 if_gif.c:209 #14 0xc05355ba in gif_clone_destroy (ifp=3D0xc168baa0) at /usr/src/sys/net/if_gif.c:226 #15 0xc0533aca in ifc_simple_destroy (ifc=3D0xc06a3520, ifp=3D0x11046f4) at /usr/src/sys/net/if_clone.c:478 #16 0xc0533109 in if_clone_destroy (name=3D0xc18e8140 "gif0") at /usr/src/sys/net/if_clone.c:172 ---Type to continue, or q to quit--- #17 0xc0531cb8 in ifioctl (so=3D0xc19c52c8, cmd=3D2149607801, data=3D0xc18e8140 "gif0", td=3D0xc168ba80) at = /usr/src/sys/net/if.c:=20 1533 #18 0xc04ec9bf in soo_ioctl (fp=3D0xc168baa0, cmd=3D2149607801, =20 data=3D0xc18e8140, active_cred=3D0xc14fad00, td=3D0xc168ba80) at /usr/src/sys/kern/sys_socket.c:214 #19 0xc04e7089 in ioctl (td=3D0xc168ba80, uap=3D0xcc680d04) at = file.h:264 #20 0xc0647317 in syscall (frame=3D {tf_fs =3D 59, tf_es =3D 59, tf_ds =3D 59, tf_edi =3D 134571680, =20= tf_esi =3D 1, tf_ebp =3D -1077942312, tf_isp =3D -865596060, tf_ebx =3D =20= -1077941000, tf_edx =3D 134583517, tf_ecx =3D 0, tf_eax =3D 54, = tf_trapno =3D =20 12, tf_err =3D 2, tf_eip =3D 672448383, tf_cs =3D 51, tf_eflags =3D 646, = =20 tf_esp =3D -1077942340, tf_ss =3D 59}) at /usr/src/sys/i386/i386/trap.c:983 #21 0xc0635aef in Xint0x80_syscall () at /usr/src/sys/i386/i386/=20 exception.s:200 #22 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) up #1 0xc04c4b02 in boot (howto=3D260) at /usr/src/sys/kern/=20 kern_shutdown.c:409 409 doadump(); (kgdb) up #2 0xc04c4d98 in panic (fmt=3D0xc06683c3 "%s") at /usr/src/sys/kern/kern_shutdown.c:565 565 boot(bootopt); (kgdb) up #3 0xc0647000 in trap_fatal (frame=3D0xcc680ac8, eva=3D17844072) at /usr/src/sys/i386/i386/trap.c:837 837 panic("%s", trap_msg[type]); (kgdb) up #4 0xc06467e2 in trap (frame=3D {tf_fs =3D -1060962296, tf_es =3D -865599448, tf_ds =3D = -1067515864, =20 tf_edi =3D -1067971588, tf_esi =3D -1050101120, tf_ebp =3D -865596660, =20= tf_isp =3D -865596684, tf_ebx =3D -1050353088, tf_edx =3D -1050353088, =20= tf_ecx =3D 17843956, tf_eax =3D -1050101088, tf_trapno =3D 12, tf_err =3D = 0, =20 tf_eip =3D -1068609375, tf_cs =3D 32, tf_eflags =3D 65543, tf_esp =3D =20= -1050101120, tf_ss =3D -865596624}) at /usr/src/sys/i386/i386/trap.c:270 270 trap_fatal(&frame, eva); (kgdb) up #5 0xc0635a9a in calltrap () at /usr/src/sys/i386/i386/exception.s:139 139 call trap Current language: auto; currently asm (kgdb) up #6 0xc04e50a1 in turnstile_setowner (ts=3D0xc164e240, owner=3D0x11046f4) at /usr/src/sys/kern/subr_turnstile.c:432 432 ts->ts_owner =3D owner; Current language: auto; currently c (kgdb) list 427 { 428 429 mtx_assert(&td_contested_lock, MA_OWNED); 430 MPASS(owner->td_proc->p_magic =3D=3D P_MAGIC); 431 MPASS(ts->ts_owner =3D=3D NULL); 432 ts->ts_owner =3D owner; 433 LIST_INSERT_HEAD(&owner->td_contested, ts, ts_link); 434 } 435 436 /* (kgdb) p owner $1 =3D (struct thread *) 0x11046f4 (kgdb) p ts $2 =3D (struct turnstile *) 0xc164e240 (kgdb) p tx=08 =08s_link No symbol "ts_link" in current context. (kgdb) router# ^D=08=08exit Script done on Thu Oct 26 14:31:20 2006 -- /"\ ASCII Ribbon Campaign . \ / - NO HTML/RTF in e-mail . X - NO Word docs in e-mail . / \ ----------------------------------------------------------------- jeh@FreeBSD.org http://www.FreeBSD.org The Power to Serve jim@TheHousleys.Net http://www.TheHousleys.net --------------------------------------------------------------------- "Eagles may soar, but weasels don't get sucked into jet engines" -- Anon