Date: Mon, 15 Mar 2010 11:37:41 +0300 From: Nick Filimonov <nick@freenet.ru> To: freebsd-ipfw@freebsd.org Subject: Strange crash in dummynet under high load (7.2-RELEASE) Message-ID: <1268642261.24791.27.camel@kate-laptop>
next in thread | raw e-mail | index | archive | help
Gentlemen,
We're experiencing a kernel panic in dummynet under high load (around
100-110Kpps) in the bridge configuration. It appears that somehow
packets with empty mbuf packet header appear on top of the queue - that
is they have no tag, pointer to the next packet in the queue is null,
etc. Processing such a packet yields null-pointer dereference. The queue
itself appear to have some more packets in it (at least it has non-zero
length and a valid packet on its tail pointer).
I can almost certainly cause this crash by trying to attach dummynet
process to a specific CPU with cpuset even under moderate load;
contributing factor could be that we use new igb multithreaded drivers
that could process more pps simultaneously.
I've attempted to debug the issue by means of modifying the code so
that it wouldn't crash (discarding such packets, granted it results in
mbuf leak from packets remaining in such queues, but it is not that big)
and looking closely on the status of queues at the end of dummynet_io;
they all look good and no null headers encountered there..
Any input or advice would be much appreciated. Output of kgdb and
relevant portions of sysctl.conf is below:
net.link.bridge.ipfw=1
net.inet.ip.fw.one_pass=0
net.inet.ip.fw.dyn_max=65535
net.inet.ip.fw.dyn_buckets=2048
kern.ipc.nmbclusters=204800
net.inet.ip.dummynet.io_fast=1
net.inet.ip.dummynet.max_chain_len=32
net.inet.ip.dummynet.hash_size=32768
bridge001# kgdb kernel.debug /var/crash/vmcore.12
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "i386-marcel-freebsd"...
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
cpuid = 7; apic id = 17
fault virtual address = 0x18
fault code = supervisor write, page not present
instruction pointer = 0x20:0xc08b7100
stack pointer = 0x28:0xe70dac0c
frame pointer = 0x28:0xe70dac18
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 89 (dummynet)
trap number = 12
panic: page fault
cpuid = 7
Uptime: 1h2m23s
Physical memory: 3050 MB
Dumping 222 MB: 207 191 175 159 143 127 111 95 79 63 47 31 15
Reading symbols from /boot/kernel/acpi.ko...Reading symbols
from /boot/kernel/acpi.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/acpi.ko
#0 doadump () at pcpu.h:196
196 __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) bt
#0 doadump () at pcpu.h:196
#1 0xc07cea47 in boot (howto=260)
at /usr/src/sys/kern/kern_shutdown.c:418
#2 0xc07ced19 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:574
#3 0xc0ac5a4c in trap_fatal (frame=0xe70dabcc, eva=24)
at /usr/src/sys/i386/i386/trap.c:939
#4 0xc0ac5cb0 in trap_pfault (frame=0xe70dabcc, usermode=0, eva=24)
at /usr/src/sys/i386/i386/trap.c:852
#5 0xc0ac6632 in trap (frame=0xe70dabcc)
at /usr/src/sys/i386/i386/trap.c:530
#6 0xc0aab74b in calltrap () at /usr/src/sys/i386/i386/exception.s:159
#7 0xc08b7100 in move_pkt (pkt=0xc6ddb100, q=0xc9274000, p=0xc66be200,
len=2048) at /usr/src/sys/netinet/ip_dummynet.c:545
#8 0xc08b7bb0 in ready_event (q=0xc9274000, head=0xe70dac8c,
tail=0xe70dac88)
at /usr/src/sys/netinet/ip_dummynet.c:593
#9 0xc08b9965 in dummynet_task (context=0x0, pending=1)
at /usr/src/sys/netinet/ip_dummynet.c:847
#10 0xc0803cd5 in taskqueue_run (queue=0xc6886400)
at /usr/src/sys/kern/subr_taskqueue.c:282
#11 0xc0803ee8 in taskqueue_thread_loop (arg=0xc0ca0068)
at /usr/src/sys/kern/subr_taskqueue.c:401
#12 0xc07a89c9 in fork_exit (callout=0xc0803e20
<taskqueue_thread_loop>,
arg=0xc0ca0068, frame=0xe70dad38)
at /usr/src/sys/kern/kern_fork.c:810
#13 0xc0aab7c0 in fork_trampoline ()
at /usr/src/sys/i386/i386/exception.s:264
(kgdb) up 7
#7 0xc08b7100 in move_pkt (pkt=0xc6ddb100, q=0xc9274000, p=0xc66be200,
len=2048) at /usr/src/sys/netinet/ip_dummynet.c:545
545 dt->output_time = curr_time + p->delay ;
(kgdb) print *pkt
$1 = {m_hdr = {mh_next = 0x0, mh_nextpkt = 0x0, mh_data = 0xc8f8e800
"",
mh_len = 2048, mh_flags = 1, mh_type = 1, pad = "\000"}, M_dat = {MH
= {
MH_pkthdr = {rcvif = 0x0, header = 0x0, len = 2048, csum_flags =
0,
csum_data = 0, tso_segsz = 0, ether_vtag = 0, tags = {
slh_first = 0x0}}, MH_dat = {MH_ext = {ext_buf = 0xc8f8e800
"",
ext_free = 0, ext_args = 0x0, ext_size = 2048, ref_cnt =
0xc8ef715c,
ext_type = 1},
MH_databuf = "\000���\000\000\000\000\000\000\000\000\000\b\000
\000\\q��\001\000\000\000\205�\233\022\206��>�+V<�K\024B�4�2�=\233?���
\005�l\224�\f^�\\\2041.W\n�gt\237\001�\022%�\v/kg\210����8\226u\227
\001�U\004�_\"z\226����", '\0' <repeats 103 times>}},
M_databuf = "\000\000\000\000\000\000\000\000\000\b", '\0' <repeats
19 times>, "���\000\000\000\000\000\000\000\000\000\b\000\000\\q��\001
\000\000\000\205�\233\022\206��>�+V<�K\024B�4�2�=\233?���\005�l\224�\f^�
\\\2041.W\n�gt\237\001�\022%�\v/kg\210����8\226u\227\001�U\004�_\"z
\226����", '\0' <repeats 103 times>}}
(kgdb) print *p
$2 = {next = {sle_next = 0xc666be00}, pipe_nr = 11, bandwidth =
6000000,
delay = 0, head = 0xc8e3f900, tail = 0xc8e3f900, scheduler_heap =
{size = 0,
elements = 0, offset = 0, p = 0x0}, not_eligible_heap = {size = 0,
elements = 0, offset = 0, p = 0x0}, idle_heap = {size = 0, elements
= 0,
offset = 124, p = 0x0}, V = 0, sum = 0, numbytes = 0, sched_time =
0,
if_name = '\0' <repeats 15 times>, ifp = 0x0, ready = 0, fs = {next =
{
sle_next = 0x0}, fs_nr = 0, flags_fs = 1, pipe = 0xc66be200,
parent_nr = 0, weight = 0, qsize = 50, plr = 0, flow_mask = {dst_ip
= 0,
src_ip = 4294967295, dst_port = 0, src_port = 0, fib = 0 '\0',
proto = 0 '\0', flags = 0 '\0', addr_type = 0 '\0', dst_ip6 = {
__u6_addr = {__u6_addr8 = '\0' <repeats 15 times>, __u6_addr16 =
{0,
0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, src_ip6
= {
__u6_addr = {__u6_addr8 = '\0' <repeats 15 times>, __u6_addr16 =
{0,
0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, flow_id6
= 0,
frag_id6 = 0}, rq_size = 32768, rq_elements = 893, rq =
0xc7c25000,
last_expired = 0, backlogged = 0, w_q = 0, max_th = 0, min_th = 0,
max_p = 0, c_1 = 0, c_2 = 0, c_3 = 0, c_4 = 0, w_q_lookup = 0x0,
lookup_depth = 0, lookup_step = 0, lookup_weight = 0, avg_pkt_size =
0,
max_pkt_size = 0}}
(kgdb) print *q
$3 = {next = 0x0, id = {dst_ip = 0, src_ip = 169950275, dst_port = 0,
src_port = 0, fib = 0 '\0', proto = 0 '\0', flags = 0 '\0',
addr_type = 4 '\004', dst_ip6 = {__u6_addr = {
__u6_addr8 = "\000k���:7�\177�~�\004k��", __u6_addr16 = {27392,
49353,
15068, 50743, 58495, 49278, 27396, 49353}, __u6_addr32 = {
3234425600, 3325508316, 3229541503, 3234425604}}}, src_ip6 = {
__u6_addr = {__u6_addr8 = "��c�\004\000\000\000@�g�\004\000\000",
__u6_addr16 = {60652, 50787, 4, 0, 49728, 50791, 4, 0},
__u6_addr32 = {
3328437484, 4, 3328688704, 4}}}, flow_id6 = 6400,
frag_id6 = 3328688704}, head = 0x0, tail = 0xc72ff200, len = 48,
len_bytes = 33276, numbytes = 6560000, tot_pkts = 12726,
tot_bytes = 10474176, drops = 1695, hash_slot = 29351, avg = 0, count
= 0,
random = 0, q_time = 3739317, fs = 0xc66be278, heap_pos = 0,
sched_time = 3742254, S = 1, F = 0}
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1268642261.24791.27.camel>