Date: Mon, 08 Dec 2003 23:26:31 +0000 From: Robin Breathe <robin@isometry.net> To: current@freebsd.org Cc: John Baldwin <jhb@FreeBSD.org> Subject: Re: Fatal trap 12: page fault while in kernel mode(subr_turnstile.c) w/ trace Message-ID: <3FD508A7.3010901@isometry.net> In-Reply-To: <XFMail.20031208171829.jhb@FreeBSD.org> References: <3FD4F748.4050900@isometry.net> <XFMail.20031208171829.jhb@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
John Baldwin wrote:
> On 08-Dec-2003 Robin Breathe wrote:
>>I've been experiencing the following repeatable panic on recent
>>-CURRENT. This one is against RELENG_5_2, as of around 18:00 U‎T today.
>>Until now I've not been able to get a dump, but thankfully one's finally
>>come :)
>
> If you can reproduce the panic with INVARIANTS, it would be very useful
> to know which, if any, assertions it trips.
Here's the output from DDB with INVARIANTS enabled, does it contain what
you need?
I think I spotted the error in my ways on failing to use previous forced
dumps (it won't dump itself without my doing a 'call doadump' manually),
so I have both DDB and GDB output.
### DDB:
kernel trap 12 with interrupts disabled
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x1103bd00
fault code = supervisor read, page not present
instruction pointer = 0x8:0xc0537aa6
stack pointer = 0x10:0xdcacc960
frame pointer = 0x10:0xdcacc980
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = resume, IOPL = 0
current process = 536 (ngctl)
kernel: type 12 trap, code=0
Stopped at turnstile_wait+0x86: movl 0(%edx),%eax
db> trace
turnstile_wait(0,c47949c8,1103bd00,1cc,250) at turnstile_wait+0x86
_mtx_lock_sleep(c47949c8,0,c06d05f9,250,c4913c7c) at _mtx_lock_sleep+0x125
_mtx_lock_flags(c47949c8,0,c06d05f9,250,c0506aec) at _mtx_lock_flags+0x95
if_detach(c4794808,c4d42200,dcacca5c,c4d7ba51,c4794808) at if_detach+0x394
ether_ifdetach(c4794808,c06d115c,820,c4d42200,c4d42200) at
ether_ifdetach+0x30
ng_eiface_rmnode(c4d42200,0,0,c4d42200,c4d42200) at ng_eiface_rmnode+0x61
ng_rmnode(c4d42200,0,0,0,0) at ng_rmnode+0xc7
ng_generic_msg(c4d42200,c4a04200,0,0,0) at ng_generic_msg+0x11f
ng_apply_item(c4d42200,c4a04200,c06d115c,7d6,c4a04200) at
ng_apply_item+0x365
ng_snd_item(c4a04200,0,c47a0820,0,0) at ng_snd_item+0x7cb
ngc_send(c4ab91e0,0,c1d12e00,c47a07a0,0) at ngc_send+0x146
sosend(c4ab91e0,c47a07a0,dcaccc4c,c1d12e00,0) at sosend+0x44d
kern_sendit(c48a48c0,3,dcacccc4,0,0) at kern_sendit+0x17c
sendit(c48a48c0,3,dcacccc4,0,804f034) at sendit+0x16e
sendto(c48a48c0,dcaccd14,c06e14fe,3ee,6) at sendto+0x5b
syscall(2f,2f,2f,bfbfe9c8,bfbfe9c2) at syscall+0x2c0
Xint0x80_syscall() at Xint0x80_syscall+0x1d
--- syscall (133, FreeBSD ELF32, sendto), eip = 0x280c568f, esp =
0xbfbfe97c, ebp = 0xbfbfebe8 ---
db> call doadump
Dumping 511 MB
16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304
320 336 352 368 384 400 416 432 448 464 480 496
Dump complete
0xf
db> panic
panic: from debugger
cpuid = 0;
Debugger("panic")
Fatal trap 3: breakpoint instruction fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer = 0x8:0xc0672525
stack pointer = 0x10:0xdcacc714
frame pointer = 0x10:0xdcacc720
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = IOPL = 0
current process = 536 (ngctl)
Stopped at turnstile_wait+0x86: movl 0(%edx),%eax
panic: from debugger
cpuid = 0;
Uptime: 10m31s
panic: mi_switch: switch in a critical section
cpuid = 0;
Uptime: 10m31s
panic: msleep
cpuid = 0;
Uptime: 10m31s
panic: msleep
cpuid = 0;
[...repeatead quite a few times...]
Uptime: 10m31s
panic: msleep
cpuid = 0;
Uptime: 10m31s
panic: msleep
cpuid = 0;
Fatal double fault:
eip = 0xc0508286
esp = 0xdcacaffc
ebp = 0xdcacb018
cpuid = 0; apic id = 00
panic: double fault
cpuid = 0;
Uptime: 10m31s
panic: msleep
cpuid = 0;
Uptime: 10m31s
[...repeated more, then stops... machine is silent for about 5 minutes,
then reboots...]
### GDB (ok, I may have been foolish with my past forced dumps):
[twiddle:/home/data/crash]# gdb -k kernel.debug2 vmcore.2
GNU gdb 5.2.1 (FreeBSD)
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-undermydesk-freebsd"...
panic messages:
---
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x1103bd00
fault code = supervisor read, page not present
instruction pointer = 0x8:0xc0537aa6
stack pointer = 0x10:0xdcacc960
frame pointer = 0x10:0xdcacc980
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = resume, IOPL = 0
current process = 536 (ngctl)
Dumping 511 MB
16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304
320 336 352 368 384 400 416 432 448 464 480 496
---
Reading symbols from /boot/kernel/acpi.ko...done.
Loaded symbols for /boot/kernel/acpi.ko
Reading symbols from /boot/kernel/ng_socket.ko...done.
Loaded symbols for /boot/kernel/ng_socket.ko
Reading symbols from /boot/kernel/ng_eiface.ko...done.
Loaded symbols for /boot/kernel/ng_eiface.ko
#0 doadump () at /usr/src/sys/kern/kern_shutdown.c:240
240 dumping++;
(kgdb) bt
#0 doadump () at /usr/src/sys/kern/kern_shutdown.c:240
#1 0xc044c695 in db_fncall (dummy1=1016, dummy2=0, dummy3=331,
dummy4=0xdcacc78c "È\217tÀø\003") at /usr/src/sys/ddb/db_command.c:548
#2 0xc044c3e2 in db_command (last_cmdp=0xc071a400, cmd_table=0x0,
aux_cmd_tablep=0xc06e5e7c, aux_cmd_tablep_end=0xc06e5e80) at
/usr/src/sys/ddb/db_command.c:346
#3 0xc044c525 in db_command_loop () at /usr/src/sys/ddb/db_command.c:472
#4 0xc044f525 in db_trap (type=12, code=0) at /usr/src/sys/ddb/db_trap.c:73
#5 0xc067221c in kdb_trap (type=12, code=0, regs=0xdcacc920) at
/usr/src/sys/i386/i386/db_interface.c:171
#6 0xc0687ea6 in trap_fatal (frame=0xdcacc920, eva=0) at
/usr/src/sys/i386/i386/trap.c:816
#7 0xc0687523 in trap (frame=
{tf_fs = -1066598376, tf_es = 16, tf_ds = 16, tf_edi = 0, tf_esi
= -997570368, tf_ebp = -592656000, tf_isp = -592656052, tf_ebx =
-998684216, tf_edx = 285457664, tf_ecx = 1, tf_eax = 0, tf_trapno = 12,
tf_err = 0, tf_eip = -1068270938, tf_cs = 8, tf_eflags = 65542, tf_esp =
-1066240308, tf_ss = 1}) at /usr/src/sys/i386/i386/trap.c:250
#8 0xc0673c68 in calltrap () at {standard input}:94
#9 0xc0508525 in _mtx_lock_sleep (m=0xc47949c8, opts=0, file=0xc06d05f9
"/usr/src/sys/net/if.c", line=592) at /usr/src/sys/kern/kern_mutex.c:476
#10 0xc0508135 in _mtx_lock_flags (m=0x0, opts=0, file=0xc06d05f9
"/usr/src/sys/net/if.c", line=592) at /usr/src/sys/kern/kern_mutex.c:218
#11 0xc057c9f4 in if_detach (ifp=0xc4794808) at /usr/src/sys/net/if.c:592
#12 0xc057fcb0 in ether_ifdetach (ifp=0xc4794808) at
/usr/src/sys/net/if_ethersubr.c:868
#13 0xc4d7ba51 in ng_eiface_rmnode () from /boot/kernel/ng_eiface.ko
#14 0xc0589b27 in ng_rmnode (node=0xc4d42200, dummy1=0x0, dummy2=0x0,
dummy3=0) at /usr/src/sys/netgraph/ng_base.c:712
#15 0xc058d2df in ng_generic_msg (here=0xc4d42200, item=0xc4a04200,
lasthook=0x0) at /usr/src/sys/netgraph/ng_base.c:2476
#16 0xc058cfa5 in ng_apply_item (node=0xc4d42200, item=0xc4a04200) at
/usr/src/sys/netgraph/ng_base.c:2405
#17 0xc058c9eb in ng_snd_item (item=0xc4a04200, queue=0) at
/usr/src/sys/netgraph/ng_base.c:2252
#18 0xc4d77936 in ngc_send () from /boot/kernel/ng_socket.ko
#19 0xc054e56d in sosend (so=0xc4ab91e0, addr=0xc47a07a0,
uio=0xdcaccc4c, top=0xc1d12e00, control=0x0, flags=0, td=0xc48a48c0) at
/usr/src/sys/kern/uipc_socket.c:715
#20 0xc0552bac in kern_sendit (td=0xc48a48c0, s=3, mp=0xdcacccc4,
flags=0, control=0x0) at /usr/src/sys/kern/uipc_syscalls.c:723
#21 0xc05529fe in sendit (td=0x0, s=0, mp=0xdcacccc4, flags=0) at
/usr/src/sys/kern/uipc_syscalls.c:663
#22 0xc0552d3b in sendto (td=0x0, uap=0x0) at
/usr/src/sys/kern/uipc_syscalls.c:784
#23 0xc0688240 in syscall (frame=
{tf_fs = 47, tf_es = 47, tf_ds = 47, tf_edi = -1077941816, tf_esi
= -1077941822, tf_ebp = -1077941272, tf_isp = -592654988, tf_ebx =
671590912, tf_edx = -1077941824, tf_ecx = 5, tf_eax = 133, tf_trapno =
12, tf_err = 2, tf_eip = 671897231, tf_cs = 31, tf_eflags = 514, tf_esp
= -1077941892, tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:1010
#24 0xc0673cbd in Xint0x80_syscall () at {standard input}:136
---Can't read userspace from dump, or kernel process---
(kgdb) l *0xc0537aa6
0xc0537aa6 is in turnstile_wait (/usr/src/sys/kern/subr_turnstile.c:439).
434 td = curthread;
435 tc = TC_LOOKUP(lock);
436 mtx_assert(&tc->tc_lock, MA_OWNED);
437 MPASS(td->td_turnstile != NULL);
438 MPASS(owner != NULL);
439 MPASS(owner->td_proc->p_magic == P_MAGIC);
440
441 /* If the passed in turnstile is NULL, use this thread's
turnstile. */
442 if (ts == NULL) {
443 ts = td->td_turnstile;
(kgdb)
- Robin
--
Robin Breathe robin@isometry.net +441865741800
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3FD508A7.3010901>