Date: Wed, 2 Apr 2008 09:47:40 +0300 From: "Alexander Shulikov" <shulikov@gmail.com> To: freebsd-ipfw@freebsd.org Subject: dummynet panics after 6.2 (kern/121955) Message-ID: <18292fe60804012347m3bf7746dga85621adc8eda3b2@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
See bug: kern/121955
I have received panics with the same configuration of ipfw and
dummynet in 6.3, 7.0-RELEASE, 7.0-STABLE.
But I don't received it in 6.2.
In all panics was corrupted stack.
I use mpd-5.0 for pptp connections and shape traffic with ng_car or
dummynet for different clients.
In 7.0-STABLE on netgraph was added patches, that decreasing using
stack by netgraph. And when I updated from 7.0-RELEASE to 7.0-STABLE,
I received panic rare. Not after 3-5 minutes, but after minimum 3
hours, maximum 2,5 days.
I saw cvsweb, and ip_dummynet in 6.2 and in 6.3.
Between it's two releases was done next changes:
Revision 1.93.2.6: download - view: text, markup, annotated - select for diffs
Wed Mar 21 17:25:15 2007 UTC (12 months, 1 week ago) by oleg
Branches: RELENG_6
CVS tags: RELENG_6_3_BP, RELENG_6_3_0_RELEASE, RELENG_6_3
Diff to: previous 1.93.2.5: preferred, colored; branchpoint 1.93:
preferred, colored; next MAIN 1.94: preferred, colored
Changes since revision 1.93.2.5: +481 -375 lines
MFC rev. 1.102-1.105
- style(9) cleanup
- Use separate thread for servicing dummynet(4).
Utilize taskqueue(9) API.
- Convert
net.inet.ip.dummynet.curr_time
net.inet.ip.dummynet.searches
net.inet.ip.dummynet.search_steps
to SYSCTL_LONG nodes. It will prevent frequent wrap around on 64bit archs.
- Implement simple mechanics for dummynet(4) internal time correction.
Under certain circumstances (system high load, dummynet lock contention, etc)
dummynet's tick counter can be significantly slower than it should be.
(I've observed up to 25% difference on one of my production servers).
Since this counter used for packet scheduling, it's accuracy is vital for
precise bandwidth limitation.
Introduce new sysctl nodes:
net.inet.ip.dummynet.
tick_lost - number of ticks coalesced by taskqueue thread.
tick_adjustment - number of time corrections done.
tick_diff - adjusted vs non-adjusted tick counter difference
tick_delta - last vs 'standard' tick differnece (usec).
tick_delta_sum - accumulated (and not corrected yet) time
difference (usec).
- Use non-recursive mutex. MTX_RECURSE is unnecessary since rev. 1.70
- Pay respect to net.isr.direct: use netisr_dispatch() instead of ip_input()
- purge_flow_set():
Do not leak memory while purging queues which are not bound to pipe.
Diff between files see in file.
After this changes (that mfced from 1.102-1.105) system became working unstable.
I think, that it's changes must be rereaded, because panics on my
server with 6.3 or 7.0, 7.0-STABLE don't repeated in 6.2 with same
configuration.
Last panics was:
# kgdb /root/29.03.08/kernel.debug /var/crash/vmcore.11
kgdb: kvm_nlist(_stopped_cpus):
kgdb: kvm_nlist(_stoppcbs):
[GDB will not be able to debug user-mode threads:
/usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0xf2a9864
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc3f96a9f
stack pointer = 0x28:0xe4614a88
frame pointer = 0x28:0xe4614b7c
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 40 (dummynet)
trap number = 12
panic: page fault
Uptime: 3h55m10s
Physical memory: 1015 MB
Dumping 173 MB: 158 142 126 110 94 78 62 46 30 14
#0 doadump () at pcpu.h:195
195 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) bt
#0 doadump () at pcpu.h:195
#1 0xc056dbf3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2 0xc056ddef in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3 0xc07bccac in trap_fatal (frame=0xe4614a48, eva=254449764) at
/usr/src/sys/i386/i386/trap.c:899
#4 0xc07bcf10 in trap_pfault (frame=0xe4614a48, usermode=0,
eva=254449764) at /usr/src/sys/i386/i386/trap.c:812
#5 0xc07bd869 in trap (frame=0xe4614a48) at /usr/src/sys/i386/i386/trap.c:490
#6 0xc07a786b in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7 0xc3f96a9f in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)
# kgdb /root/29.03.08/kernel.debug /var/crash/vmcore.10
kgdb: kvm_nlist(_stopped_cpus):
kgdb: kvm_nlist(_stoppcbs):
[GDB will not be able to debug user-mode threads:
/usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0xcd
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc3f8ba9f
stack pointer = 0x28:0xe4614a88
frame pointer = 0x28:0xe4614b7c
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 40 (dummynet)
trap number = 12
panic: page fault
Uptime: 3h21m49s
Physical memory: 1015 MB
Dumping 171 MB: 156 140 124 108 92 76 60 44 28 12
#0 doadump () at pcpu.h:195
195 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) bt
#0 doadump () at pcpu.h:195
#1 0xc056dbf3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2 0xc056ddef in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3 0xc07bccac in trap_fatal (frame=0xe4614a48, eva=205) at
/usr/src/sys/i386/i386/trap.c:899
#4 0xc07bcf10 in trap_pfault (frame=0xe4614a48, usermode=0, eva=205)
at /usr/src/sys/i386/i386/trap.c:812
#5 0xc07bd869 in trap (frame=0xe4614a48) at /usr/src/sys/i386/i386/trap.c:490
#6 0xc07a786b in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7 0xc3f8ba9f in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)
# kgdb /root/29.03.08/kernel.debug /var/crash/vmcore.9
kgdb: kvm_nlist(_stopped_cpus):
kgdb: kvm_nlist(_stoppcbs):
[GDB will not be able to debug user-mode threads:
/usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0xf4
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc3f8ca9f
stack pointer = 0x28:0xe4614a88
frame pointer = 0x28:0xe4614b7c
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 40 (dummynet)
trap number = 12
panic: page fault
Uptime: 2d19h13m24s
Physical memory: 1015 MB
Dumping 199 MB: 184 168 152 136 120 104 88 72 56 40 24 8
#0 doadump () at pcpu.h:195
195 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) bt
#0 doadump () at pcpu.h:195
#1 0xc056dbf3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2 0xc056ddef in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3 0xc07bccac in trap_fatal (frame=0xe4614a48, eva=244) at
/usr/src/sys/i386/i386/trap.c:899
#4 0xc07bcf10 in trap_pfault (frame=0xe4614a48, usermode=0, eva=244)
at /usr/src/sys/i386/i386/trap.c:812
#5 0xc07bd869 in trap (frame=0xe4614a48) at /usr/src/sys/i386/i386/trap.c:490
#6 0xc07a786b in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7 0xc3f8ca9f in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)
# kgdb /root/29.03.08/kernel.debug /var/crash/vmcore.8
kgdb: kvm_nlist(_stopped_cpus):
kgdb: kvm_nlist(_stoppcbs):
[GDB will not be able to debug user-mode threads:
/usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0xc2
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc3f93a9f
stack pointer = 0x28:0xe4614a88
frame pointer = 0x28:0xe4614b7c
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 40 (dummynet)
trap number = 12
panic: page fault
Uptime: 11h35m35s
Physical memory: 1015 MB
Dumping 217 MB: 202 186 170 154 138 122 106 90 74 58 42 26 10
#0 doadump () at pcpu.h:195
195 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) bt
#0 doadump () at pcpu.h:195
#1 0xc056dbf3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2 0xc056ddef in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3 0xc07bccac in trap_fatal (frame=0xe4614a48, eva=194) at
/usr/src/sys/i386/i386/trap.c:899
#4 0xc07bcf10 in trap_pfault (frame=0xe4614a48, usermode=0, eva=194)
at /usr/src/sys/i386/i386/trap.c:812
#5 0xc07bd869 in trap (frame=0xe4614a48) at /usr/src/sys/i386/i386/trap.c:490
#6 0xc07a786b in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7 0xc3f93a9f in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)
# kgdb /root/29.03.08/kernel.debug /var/crash/vmcore.7
kgdb: kvm_nlist(_stopped_cpus):
kgdb: kvm_nlist(_stoppcbs):
[GDB will not be able to debug user-mode threads:
/usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0xef
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc3f85a9f
stack pointer = 0x28:0xe4614a88
frame pointer = 0x28:0xe4614b7c
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 40 (dummynet)
trap number = 12
panic: page fault
Uptime: 1d1h54m1s
Physical memory: 1015 MB
Dumping 227 MB: 212 196 180 164 148 132 116 100 84 68 52 36 20 4
#0 doadump () at pcpu.h:195
195 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) bt
#0 doadump () at pcpu.h:195
#1 0xc056dbf3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2 0xc056ddef in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3 0xc07bccac in trap_fatal (frame=0xe4614a48, eva=239) at
/usr/src/sys/i386/i386/trap.c:899
#4 0xc07bcf10 in trap_pfault (frame=0xe4614a48, usermode=0, eva=239)
at /usr/src/sys/i386/i386/trap.c:812
#5 0xc07bd869 in trap (frame=0xe4614a48) at /usr/src/sys/i386/i386/trap.c:490
#6 0xc07a786b in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7 0xc3f85a9f in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)
My sysctl.conf:
# cat /etc/sysctl.conf
net.inet.ip.fw.autoinc_step=1
net.inet.ip.intr_queue_maxlen=256
net.inet.ip.dummynet.hash_size=16384
net.inet.ip.fw.dyn_buckets=1024
net.inet.ip.fastforwarding=1
kern.ipc.somaxconn=2048
kern.ipc.nmbclusters=32768
net.inet.tcp.recvspace=65536
net.inet.tcp.sendspace=131072
net.inet.ip.fw.one_pass=0
Between panic I try change variables:
net.inet.ip.fw.one_pass to 1, but panic repetated after this
My system now:
FreeBSD myhost.mydom 7.0-STABLE FreeBSD 7.0-STABLE #0: Wed Mar 26
07:56:32 EET 2008 root@myhost.mydom:/usr/obj/usr/src/sys/myhost
i386
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?18292fe60804012347m3bf7746dga85621adc8eda3b2>