From owner-freebsd-bugs@FreeBSD.ORG Sat Apr 11 08:10:07 2009 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4BE6F1065677 for ; Sat, 11 Apr 2009 08:10:07 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id EB8DC8FC17 for ; Sat, 11 Apr 2009 08:10:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n3B8A2cC051314 for ; Sat, 11 Apr 2009 08:10:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n3B8A2Q3051313; Sat, 11 Apr 2009 08:10:02 GMT (envelope-from gnats) Resent-Date: Sat, 11 Apr 2009 08:10:02 GMT Resent-Message-Id: <200904110810.n3B8A2Q3051313@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Erich Jenkins Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 544181065672 for ; Sat, 11 Apr 2009 08:07:03 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id 42E5D8FC16 for ; Sat, 11 Apr 2009 08:07:03 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.3/8.14.3) with ESMTP id n3B872HL061126 for ; Sat, 11 Apr 2009 08:07:02 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.3/8.14.3/Submit) id n3B872Q6061125; Sat, 11 Apr 2009 08:07:02 GMT (envelope-from nobody) Message-Id: <200904110807.n3B872Q6061125@www.freebsd.org> Date: Sat, 11 Apr 2009 08:07:02 GMT From: Erich Jenkins To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/133595: Kernel Panic at pcpu.h:195 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Apr 2009 08:10:07 -0000 >Number: 133595 >Category: kern >Synopsis: Kernel Panic at pcpu.h:195 >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sat Apr 11 08:10:02 UTC 2009 >Closed-Date: >Last-Modified: >Originator: Erich Jenkins >Release: FreeBSD 7.0 >Organization: Fuujin Group Ltd >Environment: FreeBSD vpngw.fuujinnetworks.com 7.0-RELEASE FreeBSD 7.0-RELEASE #0: Wed Apr 1 22:01:11 CDT 2009 ibe7boo@vpngw.fuujinnetworks.com:/usr/src/sys/i386/compile/KNL04012009 i386 >Description: Kernel Panic and core dump. The box panics and reboots frequently (perhaps 2 times daily). I found a bug report in GNATS that looks similar to this panic (See GNATS 129316 or 127699 for example). All of these faults seem to be similar and the bt for each looks VERY similar, though the displayed "current process" varies from the other bug reports. This box is running IPNAT, IPFW and several instances of OpenVPN, though the problem appears to be related to NAT (though not specifically to IPNAT). KGDB BT Info: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x4 fault code = supervisor read, page not present instruction pointer = 0x20:0xc388994b stack pointer = 0x28:0xdceb59c8 frame pointer = 0x28:0xdceb5a44 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 26 (irq23: vr0) trap number = 12 panic: page fault cpuid = 0 Uptime: 5d7h9m18s Physical memory: 742 MB Dumping 119 MB: 104 88 72 56 40 24 8 #0 doadump () at pcpu.h:195 195 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:195 #1 0xc05ba657 in boot (howto=260) at ../../../kern/kern_shutdown.c:409 #2 0xc05ba919 in panic (fmt=Variable "fmt" is not available. ) at ../../../kern/kern_shutdown.c:563 #3 0xc080c54c in trap_fatal (frame=0xdceb5988, eva=4) at ../../../i386/i386/trap.c:899 #4 0xc080c7b0 in trap_pfault (frame=0xdceb5988, usermode=0, eva=4) at ../../../i386/i386/trap.c:812 #5 0xc080d132 in trap (frame=0xdceb5988) at ../../../i386/i386/trap.c:490 #6 0xc07f3abb in calltrap () at ../../../i386/i386/exception.s:139 #7 0xc388994b in ?? () Previous frame inner to this frame (corrupt stack?) Here's a pciconf -lv output: hostb0@pci0:0:0:0: class=0x060000 card=0x02041106 chip=0x02041106 rev=0x00 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'K8M400 CPU to PCI Bridge' class = bridge subclass = HOST-PCI hostb1@pci0:0:0:1: class=0x060000 card=0x00000000 chip=0x12041106 rev=0x00 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'K8M400 CPU to PCI Bridge' class = bridge subclass = HOST-PCI hostb2@pci0:0:0:2: class=0x060000 card=0x00000000 chip=0x22041106 rev=0x00 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'K8M400 CPU to PCI Bridge' class = bridge subclass = HOST-PCI hostb3@pci0:0:0:3: class=0x060000 card=0x00000000 chip=0x32041106 rev=0x00 hdr=0x00 vendor = 'VIA Technologies Inc' device = '1394 i2c CPU to PCI Bridge' class = bridge subclass = HOST-PCI hostb4@pci0:0:0:4: class=0x060000 card=0x00000000 chip=0x42041106 rev=0x00 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'K8M400 CPU to PCI Bridge' class = bridge subclass = HOST-PCI hostb5@pci0:0:0:7: class=0x060000 card=0x00000000 chip=0x72041106 rev=0x00 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'K8M400 CPU to PCI Bridge' class = bridge subclass = HOST-PCI pcib1@pci0:0:1:0: class=0x060400 card=0x00000000 chip=0xb1881106 rev=0x00 hdr=0x01 vendor = 'VIA Technologies Inc' device = 'VT8237 K8HTB CPU to AGP 2.0/3.0 Bridge' class = bridge subclass = PCI-PCI pcib2@pci0:0:8:0: class=0x060400 card=0x00000000 chip=0x00241011 rev=0x03 hdr=0x01 vendor = 'Digital Equipment Corporation' device = '21151/2 PCI to PCI Bridge' class = bridge subclass = PCI-PCI atapci0@pci0:0:15:0: class=0x01018f card=0x32061565 chip=0x31491106 rev=0x80 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'VT8237 VT6410 SATA RAID Controller' class = mass storage subclass = ATA atapci1@pci0:0:15:1: class=0x01018a card=0x32061565 chip=0x05711106 rev=0x06 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'VT82C586A/B/VT82C686/A/B/VT823x/A/C Bus Master IDE Controller' class = mass storage subclass = ATA isab0@pci0:0:17:0: class=0x060100 card=0x32061565 chip=0x32271106 rev=0x00 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'VT8237 PCI-to-ISA Bridge' class = bridge subclass = PCI-ISA vr0@pci0:0:18:0: class=0x020000 card=0x01021106 chip=0x30651106 rev=0x78 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'VT6102 Rhine II PCI Fast Ethernet Controller||Used by GERICOM in laptop Webengine Advanced' class = network subclass = ethernet hostb6@pci0:0:24:0: class=0x060000 card=0x00000000 chip=0x11001022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' device = '(K8) Athlon 64/Opteron HyperTransport Technology Configuration' class = bridge subclass = HOST-PCI hostb7@pci0:0:24:1: class=0x060000 card=0x00000000 chip=0x11011022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' device = '(K8) Athlon 64/Opteron Address Map' class = bridge subclass = HOST-PCI hostb8@pci0:0:24:2: class=0x060000 card=0x00000000 chip=0x11021022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' device = '(K8) Athlon 64/Opteron DRAM Controller' class = bridge subclass = HOST-PCI hostb9@pci0:0:24:3: class=0x060000 card=0x00000000 chip=0x11031022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' device = '(K8) Athlon 64/Opteron Miscellaneous Control' class = bridge subclass = HOST-PCI vgapci0@pci0:1:0:0: class=0x030000 card=0x12031565 chip=0x31081106 rev=0x01 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'Unknown Via Unichrome S3 VGA - part of a VIA Northbridge' class = display subclass = VGA de0@pci0:2:4:0: class=0x020000 card=0x24001109 chip=0x00091011 rev=0x22 hdr=0x00 vendor = 'Digital Equipment Corporation' device = 'DecChip 21140 Fast Ethernet Adapter' class = network subclass = ethernet de1@pci0:2:5:0: class=0x020000 card=0x00000000 chip=0x00091011 rev=0x22 hdr=0x00 vendor = 'Digital Equipment Corporation' device = 'DecChip 21140 Fast Ethernet Adapter' class = network subclass = ethernet de2@pci0:2:6:0: class=0x020000 card=0x00000000 chip=0x00091011 rev=0x22 hdr=0x00 vendor = 'Digital Equipment Corporation' device = 'DecChip 21140 Fast Ethernet Adapter' class = network subclass = ethernet de3@pci0:2:7:0: class=0x020000 card=0x00000000 chip=0x00091011 rev=0x22 hdr=0x00 vendor = 'Digital Equipment Corporation' device = 'DecChip 21140 Fast Ethernet Adapter' class = network subclass = ethernet I've tried this configuration on Intel and AMD processors, and with multiple makers of NIC's to eliminate the possibility of a hardware problem. The system panics in the same manner on all hardware tested. >How-To-Repeat: Configure IPFW, IPNAT and OpenVPN (bridged not routed VPN) and two subnets behind the box. In this case, the "current process" as reported by BT is an IRQ related request to the vr0 integrated NIC, though this is repeatable with all NIC's I've tested (DEC, Intel, RealTek, and Broadcom). >Fix: >Release-Note: >Audit-Trail: >Unformatted: