Date: Tue, 28 Dec 2010 16:36:53 GMT From: mike tancsa <mike@sentex.net> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/153497: netgraph panic with ipv6 enabled Message-ID: <201012281636.oBSGaros053635@red.freebsd.org> Resent-Message-ID: <201012281640.oBSGe9a3045275@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 153497
>Category: kern
>Synopsis: netgraph panic with ipv6 enabled
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Tue Dec 28 16:40:09 UTC 2010
>Closed-Date:
>Last-Modified:
>Originator: mike tancsa
>Release: RELENG_8
>Organization:
Sentex Communications
>Environment:
FreeBSD 8.2-PRERELEASE #6: Sun Dec 12 16:25:12 EST 2010
>Description:
Using mpd5 as an LNS with approximately 600 sessions using just IPV4 is nice and stable. However, if I enable ipv6 in mpd5 and on the system, the box will panic anywhere from a few days to a couple of weeks. Enabling WITNESS in the kernel seems to make the issue more acute. The panic appears to be in the same place each time.
I have the core dump files available
panic: page fault
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address = 0x24
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc5f02dc5
stack pointer = 0x28:0xc4f5a928
frame pointer = 0x28:0xc4f5a93c
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 1006 (ng_queue0)
trap number = 12
panic: page fault
cpuid = 1
Uptime: 15d18h26m46s
Physical memory: 2036 MB
Dumping 281 MB: 266panic: bufwrite: buffer is not busy???
cpuid = 1
250
<110>ipfw: 10 Deny UDP 192.168.1.4:19008 64.7.156.213:1204 in via em1
234 218 202 186 170 154 138 122 106 90 74 58 42 26 10
#0 doadump () at pcpu.h:231
231 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) #0 doadump () at pcpu.h:231
#1 0xc068cee3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:419
#2 0xc068d147 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:592
#3 0xc08f082c in trap_fatal (frame=0xc4f5a8e8, eva=36)
at /usr/src/sys/i386/i386/trap.c:946
#4 0xc08f0a90 in trap_pfault (frame=0xc4f5a8e8, usermode=0, eva=36)
at /usr/src/sys/i386/i386/trap.c:859
#5 0xc08f0f39 in trap (frame=0xc4f5a8e8) at /usr/src/sys/i386/i386/trap.c:532
#6 0xc08d825c in calltrap () at /usr/src/sys/i386/i386/exception.s:166
#7 0xc5f02dc5 in ng_address_hook (here=0x0, item=0xc61fd9c0,
hook=0xcd018980, retaddr=0)
at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:3504
#8 0xc60a6bdd in ng_ppp_bypass (node=Variable "node" is not available.
)
at /usr/src/sys/modules/netgraph/ppp/../../../netgraph/ng_ppp.c:901
#9 0xc60a79c5 in ng_ppp_rcvdata (hook=0xcbf6d880, item=0xc61fd9c0)
at /usr/src/sys/modules/netgraph/ppp/../../../netgraph/ng_ppp.c:1524
#10 0xc5f04774 in ng_apply_item (node=0xc60e2e00, item=0xc61fd9c0, rw=0)
at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2336
#11 0xc5f0374f in ng_snd_item (item=0xc61fd9c0, flags=Variable "flags" is not available.
)
at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2253
#12 0xc5f04774 in ng_apply_item (node=0xc645d180, item=0xc61fd9c0, rw=0)
at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2336
#13 0xc5f0374f in ng_snd_item (item=0xc61fd9c0, flags=Variable "flags" is not available.
)
at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2253
#14 0xc5f04774 in ng_apply_item (node=0xc63afe00, item=0xc61fd9c0, rw=0)
at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2336
#15 0xc5f0374f in ng_snd_item (item=0xc61fd9c0, flags=Variable "flags" is not available.
)
at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2253
#16 0xc6094a4c in ng_ksocket_incoming2 (node=0xc6195000, hook=0x0,
arg1=0xc618180c, arg2=0)
at /usr/src/sys/modules/netgraph/ksocket/../../../netgraph/ng_ksocket.c:1153
#17 0xc5f048a9 in ng_apply_item (node=0xc6195000, item=0xc5f0ad40, rw=1)
at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2407
#18 0xc5f059f6 in ngthread (arg=0x0)
at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:3351
#19 0xc06627d1 in fork_exit (callout=0xc5f05890 <ngthread>, arg=0x0,
frame=0xc4f5ad28) at /usr/src/sys/kern/kern_fork.c:845
#20 0xc08d82d4 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:273
(kgdb)
(kgdb) up 7
#7 0xc5f02dc5 in ng_address_hook (here=0x0, item=0xc61fd9c0, hook=0xcd018980, retaddr=0) at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:3504
3504 if ((hook == NULL) ||
(kgdb) list
3499 * Quick sanity check..
3500 * Since a hook holds a reference on it's node, once we know
3501 * that the peer is still connected (even if invalid,) we know
3502 * that the peer node is present, though maybe invalid.
3503 */
3504 if ((hook == NULL) ||
3505 NG_HOOK_NOT_VALID(hook) ||
3506 NG_HOOK_NOT_VALID(peer = NG_HOOK_PEER(hook)) ||
3507 NG_NODE_NOT_VALID(peernode = NG_PEER_NODE(hook))) {
3508
(kgdb) p *hook
$1 = {hk_name = "bypass", '\0' <repeats 25 times>, hk_private = 0x1, hk_flags = 0, hk_type = 0, hk_peer = 0xcb58e180, hk_node = 0xc60e2e00, hk_hooks = {le_next = 0x0,
le_prev = 0xcbf6d8b4}, hk_rcvmsg = 0, hk_rcvdata = 0xc60a9d00 <ng_ppp_rcvdata_bypass>, hk_refs = 2}
(kgdb) p *peer
$2 = {hk_name = "\b\000\000\000 \000\000\000\005\000\000\000\000\000\000\000\035X)\004\036QÆCcmd5\000\000\000", hk_private = 0x0, hk_flags = 0, hk_type = 0, hk_peer = 0x0,
hk_node = 0x0, hk_hooks = {le_next = 0xc613e000, le_prev = 0x7466656c}, hk_rcvmsg = 0x67697232, hk_rcvdata = 0x7468, hk_refs = 0}
(kgdb) p *peer
$3 = {hk_name = "\b\000\000\000 \000\000\000\005\000\000\000\000\000\000\000\035X)\004\036QÆCcmd5\000\000\000", hk_private = 0x0, hk_flags = 0, hk_type = 0, hk_peer = 0x0,
hk_node = 0x0, hk_hooks = {le_next = 0xc613e000, le_prev = 0x7466656c}, hk_rcvmsg = 0x67697232, hk_rcvdata = 0x7468, hk_refs = 0}
(kgdb) p *item
$4 = {el_flags = 5, el_next = {stqe_next = 0x0}, el_dest = 0x0, el_hook = 0x0, body = {da_m = 0xcd160a00, msg = {msg_msg = 0xcd160a00, msg_retaddr = 0}, fn = {fn_fn = {
fn_fn = 0xcd160a00, fn_fn2 = 0xcd160a00}, fn_arg1 = 0x0, fn_arg2 = 0}}, apply = 0x0, depth = 3}
(kgdb) p *peernode
Cannot access memory at address 0x0
(kgdb)
>How-To-Repeat:
make an ipv6 enabled LNS with ~500 connections or more. Wait a week or two.
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201012281636.oBSGaros053635>