From owner-freebsd-ipfw@FreeBSD.ORG Wed Apr 2 06:47:44 2008 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A1B78106566B for ; Wed, 2 Apr 2008 06:47:44 +0000 (UTC) (envelope-from shulikov@gmail.com) Received: from fk-out-0910.google.com (fk-out-0910.google.com [209.85.128.188]) by mx1.freebsd.org (Postfix) with ESMTP id ED34A8FC1A for ; Wed, 2 Apr 2008 06:47:43 +0000 (UTC) (envelope-from shulikov@gmail.com) Received: by fk-out-0910.google.com with SMTP id b27so3839420fka.11 for ; Tue, 01 Apr 2008 23:47:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; bh=pMT2scXO12BCT0c0UPDSG/QBZRq63ogC7HaADRL+2aw=; b=NyKElnSTyxeqGFO03IPjQCbCXOu/z3XyjZcl7G37ikJceeO/ZTNEd+NH3qxQPndyM/VzvvVlPxXudJAd951Ix+OBDZ/RCugqxDcYvOqoIg4h49FGt9u+nbYH4KBmGS8RNiOz2TbiZUf/brXx3GHWtPPEsU4xlplesegYEyxc7OE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=VT7RPKOacUELjXlwETeYRmy0kdo7aeM18pPnlWFBy1ZTYQ79ihNU7f+Ief/x8guDI1tCDRNQp8LyUGjvO8MsHA5QCjl9SK+Mym/OyO+uOsMhi27BwkeXRdW6zTB+R3OSfssiPWNwTqAuSi+atc5jqfuSXVjYo9EOjBGlkUwonw4= Received: by 10.82.188.14 with SMTP id l14mr21509383buf.37.1207118860623; Tue, 01 Apr 2008 23:47:40 -0700 (PDT) Received: by 10.82.154.1 with HTTP; Tue, 1 Apr 2008 23:47:40 -0700 (PDT) Message-ID: <18292fe60804012347m3bf7746dga85621adc8eda3b2@mail.gmail.com> Date: Wed, 2 Apr 2008 09:47:40 +0300 From: "Alexander Shulikov" To: freebsd-ipfw@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: dummynet panics after 6.2 (kern/121955) X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Apr 2008 06:47:44 -0000 See bug: kern/121955 I have received panics with the same configuration of ipfw and dummynet in 6.3, 7.0-RELEASE, 7.0-STABLE. But I don't received it in 6.2. In all panics was corrupted stack. I use mpd-5.0 for pptp connections and shape traffic with ng_car or dummynet for different clients. In 7.0-STABLE on netgraph was added patches, that decreasing using stack by netgraph. And when I updated from 7.0-RELEASE to 7.0-STABLE, I received panic rare. Not after 3-5 minutes, but after minimum 3 hours, maximum 2,5 days. I saw cvsweb, and ip_dummynet in 6.2 and in 6.3. Between it's two releases was done next changes: Revision 1.93.2.6: download - view: text, markup, annotated - select for diffs Wed Mar 21 17:25:15 2007 UTC (12 months, 1 week ago) by oleg Branches: RELENG_6 CVS tags: RELENG_6_3_BP, RELENG_6_3_0_RELEASE, RELENG_6_3 Diff to: previous 1.93.2.5: preferred, colored; branchpoint 1.93: preferred, colored; next MAIN 1.94: preferred, colored Changes since revision 1.93.2.5: +481 -375 lines MFC rev. 1.102-1.105 - style(9) cleanup - Use separate thread for servicing dummynet(4). Utilize taskqueue(9) API. - Convert net.inet.ip.dummynet.curr_time net.inet.ip.dummynet.searches net.inet.ip.dummynet.search_steps to SYSCTL_LONG nodes. It will prevent frequent wrap around on 64bit archs. - Implement simple mechanics for dummynet(4) internal time correction. Under certain circumstances (system high load, dummynet lock contention, etc) dummynet's tick counter can be significantly slower than it should be. (I've observed up to 25% difference on one of my production servers). Since this counter used for packet scheduling, it's accuracy is vital for precise bandwidth limitation. Introduce new sysctl nodes: net.inet.ip.dummynet. tick_lost - number of ticks coalesced by taskqueue thread. tick_adjustment - number of time corrections done. tick_diff - adjusted vs non-adjusted tick counter difference tick_delta - last vs 'standard' tick differnece (usec). tick_delta_sum - accumulated (and not corrected yet) time difference (usec). - Use non-recursive mutex. MTX_RECURSE is unnecessary since rev. 1.70 - Pay respect to net.isr.direct: use netisr_dispatch() instead of ip_input() - purge_flow_set(): Do not leak memory while purging queues which are not bound to pipe. Diff between files see in file. After this changes (that mfced from 1.102-1.105) system became working unstable. I think, that it's changes must be rereaded, because panics on my server with 6.3 or 7.0, 7.0-STABLE don't repeated in 6.2 with same configuration. Last panics was: # kgdb /root/29.03.08/kernel.debug /var/crash/vmcore.11 kgdb: kvm_nlist(_stopped_cpus): kgdb: kvm_nlist(_stoppcbs): [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode fault virtual address = 0xf2a9864 fault code = supervisor read, page not present instruction pointer = 0x20:0xc3f96a9f stack pointer = 0x28:0xe4614a88 frame pointer = 0x28:0xe4614b7c code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 40 (dummynet) trap number = 12 panic: page fault Uptime: 3h55m10s Physical memory: 1015 MB Dumping 173 MB: 158 142 126 110 94 78 62 46 30 14 #0 doadump () at pcpu.h:195 195 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:195 #1 0xc056dbf3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc056ddef in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:563 #3 0xc07bccac in trap_fatal (frame=0xe4614a48, eva=254449764) at /usr/src/sys/i386/i386/trap.c:899 #4 0xc07bcf10 in trap_pfault (frame=0xe4614a48, usermode=0, eva=254449764) at /usr/src/sys/i386/i386/trap.c:812 #5 0xc07bd869 in trap (frame=0xe4614a48) at /usr/src/sys/i386/i386/trap.c:490 #6 0xc07a786b in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #7 0xc3f96a9f in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) # kgdb /root/29.03.08/kernel.debug /var/crash/vmcore.10 kgdb: kvm_nlist(_stopped_cpus): kgdb: kvm_nlist(_stoppcbs): [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode fault virtual address = 0xcd fault code = supervisor read, page not present instruction pointer = 0x20:0xc3f8ba9f stack pointer = 0x28:0xe4614a88 frame pointer = 0x28:0xe4614b7c code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 40 (dummynet) trap number = 12 panic: page fault Uptime: 3h21m49s Physical memory: 1015 MB Dumping 171 MB: 156 140 124 108 92 76 60 44 28 12 #0 doadump () at pcpu.h:195 195 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:195 #1 0xc056dbf3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc056ddef in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:563 #3 0xc07bccac in trap_fatal (frame=0xe4614a48, eva=205) at /usr/src/sys/i386/i386/trap.c:899 #4 0xc07bcf10 in trap_pfault (frame=0xe4614a48, usermode=0, eva=205) at /usr/src/sys/i386/i386/trap.c:812 #5 0xc07bd869 in trap (frame=0xe4614a48) at /usr/src/sys/i386/i386/trap.c:490 #6 0xc07a786b in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #7 0xc3f8ba9f in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) # kgdb /root/29.03.08/kernel.debug /var/crash/vmcore.9 kgdb: kvm_nlist(_stopped_cpus): kgdb: kvm_nlist(_stoppcbs): [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode fault virtual address = 0xf4 fault code = supervisor read, page not present instruction pointer = 0x20:0xc3f8ca9f stack pointer = 0x28:0xe4614a88 frame pointer = 0x28:0xe4614b7c code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 40 (dummynet) trap number = 12 panic: page fault Uptime: 2d19h13m24s Physical memory: 1015 MB Dumping 199 MB: 184 168 152 136 120 104 88 72 56 40 24 8 #0 doadump () at pcpu.h:195 195 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:195 #1 0xc056dbf3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc056ddef in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:563 #3 0xc07bccac in trap_fatal (frame=0xe4614a48, eva=244) at /usr/src/sys/i386/i386/trap.c:899 #4 0xc07bcf10 in trap_pfault (frame=0xe4614a48, usermode=0, eva=244) at /usr/src/sys/i386/i386/trap.c:812 #5 0xc07bd869 in trap (frame=0xe4614a48) at /usr/src/sys/i386/i386/trap.c:490 #6 0xc07a786b in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #7 0xc3f8ca9f in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) # kgdb /root/29.03.08/kernel.debug /var/crash/vmcore.8 kgdb: kvm_nlist(_stopped_cpus): kgdb: kvm_nlist(_stoppcbs): [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode fault virtual address = 0xc2 fault code = supervisor read, page not present instruction pointer = 0x20:0xc3f93a9f stack pointer = 0x28:0xe4614a88 frame pointer = 0x28:0xe4614b7c code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 40 (dummynet) trap number = 12 panic: page fault Uptime: 11h35m35s Physical memory: 1015 MB Dumping 217 MB: 202 186 170 154 138 122 106 90 74 58 42 26 10 #0 doadump () at pcpu.h:195 195 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:195 #1 0xc056dbf3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc056ddef in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:563 #3 0xc07bccac in trap_fatal (frame=0xe4614a48, eva=194) at /usr/src/sys/i386/i386/trap.c:899 #4 0xc07bcf10 in trap_pfault (frame=0xe4614a48, usermode=0, eva=194) at /usr/src/sys/i386/i386/trap.c:812 #5 0xc07bd869 in trap (frame=0xe4614a48) at /usr/src/sys/i386/i386/trap.c:490 #6 0xc07a786b in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #7 0xc3f93a9f in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) # kgdb /root/29.03.08/kernel.debug /var/crash/vmcore.7 kgdb: kvm_nlist(_stopped_cpus): kgdb: kvm_nlist(_stoppcbs): [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode fault virtual address = 0xef fault code = supervisor read, page not present instruction pointer = 0x20:0xc3f85a9f stack pointer = 0x28:0xe4614a88 frame pointer = 0x28:0xe4614b7c code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 40 (dummynet) trap number = 12 panic: page fault Uptime: 1d1h54m1s Physical memory: 1015 MB Dumping 227 MB: 212 196 180 164 148 132 116 100 84 68 52 36 20 4 #0 doadump () at pcpu.h:195 195 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:195 #1 0xc056dbf3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc056ddef in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:563 #3 0xc07bccac in trap_fatal (frame=0xe4614a48, eva=239) at /usr/src/sys/i386/i386/trap.c:899 #4 0xc07bcf10 in trap_pfault (frame=0xe4614a48, usermode=0, eva=239) at /usr/src/sys/i386/i386/trap.c:812 #5 0xc07bd869 in trap (frame=0xe4614a48) at /usr/src/sys/i386/i386/trap.c:490 #6 0xc07a786b in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #7 0xc3f85a9f in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) My sysctl.conf: # cat /etc/sysctl.conf net.inet.ip.fw.autoinc_step=1 net.inet.ip.intr_queue_maxlen=256 net.inet.ip.dummynet.hash_size=16384 net.inet.ip.fw.dyn_buckets=1024 net.inet.ip.fastforwarding=1 kern.ipc.somaxconn=2048 kern.ipc.nmbclusters=32768 net.inet.tcp.recvspace=65536 net.inet.tcp.sendspace=131072 net.inet.ip.fw.one_pass=0 Between panic I try change variables: net.inet.ip.fw.one_pass to 1, but panic repetated after this My system now: FreeBSD myhost.mydom 7.0-STABLE FreeBSD 7.0-STABLE #0: Wed Mar 26 07:56:32 EET 2008 root@myhost.mydom:/usr/obj/usr/src/sys/myhost i386