Date: Thu, 09 Apr 2009 18:11:26 -0600 From: Erich Jenkins <erich@fuujingroup.com> To: freebsd-bugs@freebsd.org Subject: Kernel Panic Message-ID: <49DE8EAE.7080605@fuujingroup.com>
next in thread | raw e-mail | index | archive | help
I'm hoping someone can shed some light on this issue. I've posted to several lists with no responses. We're running a 7.0 server with a customer kernel (see below) but the results have been the same on GENERIC, and on i386 hardware vs amd64 hardware (both hardware and kernels). This server is responsible for NAT for a few subnets behind it, and it running multiple instances of OpenVPN for connectivity to the subnets (bridged, not routed VPN). The box panics and reboots frequently (perhaps 2 times daily). I found a bug report in GNATS that looks similar to this panic (See GNATS 129316 or 127699 for example). All of these faults seem to be similar and the bt for each looks VERY similar. What am I missing here?? Any assistance would be greatly appreciated at this point. Here's the kernel info for this box: cpu I686_CPU ident KNL04012009 makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols options SCHED_4BSD # 4BSD scheduler options PREEMPTION # Enable kernel thread preemption options INET # InterNETworking options INET6 # IPv6 communications protocols options SCTP # Stream Control Transmission Protocol options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories options UFS_GJOURNAL # Enable gjournal-based UFS journaling options MD_ROOT # MD is a potential root device options NFSCLIENT # Network Filesystem Client options NFSSERVER # Network Filesystem Server options NFS_ROOT # NFS usable as /, requires NFSCLIENT options MSDOSFS # MSDOS Filesystem options CD9660 # ISO 9660 Filesystem options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_PART_GPT # GUID Partition Tables. options GEOM_LABEL # Provides labelization options COMPAT_43TTY # BSD 4.3 TTY compat [KEEP THIS!] options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options COMPAT_FREEBSD5 # Compatible with FreeBSD5 options COMPAT_FREEBSD6 # Compatible with FreeBSD6 options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options KBD_INSTALL_CDEV # install a CDEV entry in /dev options ADAPTIVE_GIANT # Giant mutex is adaptive. options STOP_NMI # Stop CPUS using NMI instead of IPI options AUDIT # Security event auditing options IPFIREWALL options IPDIVERT # To make an SMP kernel, the next two lines are needed options SMP # Symmetric MultiProcessor Kernel device apic # I/O APIC # Bus support. device eisa device pci # ATA and ATAPI devices device ata device atadisk # ATA disk drives device ataraid # ATA RAID drives device atapicd # ATAPI CDROM drives options ATA_STATIC_ID # Static device numbering # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device kbdmux # keyboard multiplexer device vga # VGA video card driver device splash # Splash screen and screen saver support # syscons is the default console driver, resembling an SCO console device sc device agp # support several AGP chipsets # Serial (COM) ports device sio # 8250, 16[45]50 based serial ports device uart # Generic UART driver # PCI Ethernet NICs. device de # DEC/Intel DC21x4x (``Tulip'') device em # Intel PRO/1000 adapter Gigabit Ethernet Card # PCI Ethernet NICs that use the common MII bus controller code. # NOTE: Be sure to keep the 'device miibus' line in order to use these NICs! device miibus # MII bus support device dc # DEC/Intel 21143 and various workalikes device fxp # Intel EtherExpress PRO/100B (82557, 82558) device pcn # AMD Am79C97x PCI 10/100 (precedence over 'le') device rl # RealTek 8129/8139 device sis # Silicon Integrated Systems SiS 900/SiS 7016 device vr # VIA Rhine, Rhine II # Pseudo devices. device loop # Network loopback device random # Entropy device device ether # Ethernet support device sl # Kernel SLIP device ppp # Kernel PPP device tun # Packet tunnel. device pty # Pseudo-ttys (telnet etc) device md # Memory "disks" device gif # IPv6 and IPv4 tunneling device faith # IPv6-to-IPv4 relaying (translation) device firmware # firmware assist module device bpf # Berkeley packet filter Here's the dmesg output (after a panic and reboot): FreeBSD 7.0-RELEASE #0: Wed Apr 1 22:01:11 CDT 2009 ibe7boo@vpngw.fuujinnetworks.com:/usr/src/sys/i386/compile/KNL04012009 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Athlon(tm) 64 Processor 3000+ (1999.80-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x20fc2 Stepping = 2 Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2> Features2=0x1<SSE3> AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow!+,3DNow!> AMD Features2=0x1<LAHF> real memory = 787349504 (750 MB) avail memory = 760778752 (725 MB) ACPI APIC Table: <VIAK8M AWRDACPI> ioapic0 <Version 0.3> irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: <VIAK8M AWRDACPI> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, 2ede0000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0 cpu0: <ACPI CPU> on acpi0 powernow0: <Cool`n'Quiet K8> on cpu0 device_attach: powernow0 attach returned 6 acpi_button0: <Power Button> on acpi0 acpi_button1: <Sleep Button> on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 agp0: <VIA 8380 host to PCI bridge> on hostb0 pcib1: <PCI-PCI bridge> at device 1.0 on pci0 pci1: <PCI bus> on pcib1 vgapci0: <VGA-compatible display> mem 0xf4000000-0xf7ffffff,0xfa000000-0xfaffffff irq 16 at device 0.0 on pci1 pcib2: <PCI-PCI bridge> at device 8.0 on pci0 pci2: <PCI bus> on pcib2 de0: <Digital 21140A Fast Ethernet> port 0xc000-0xc07f mem 0xfc043000-0xfc04307f irq 16 at device 4.0 on pci2 de0: Cogent EM440TX 21140A [10-100Mb/s] pass 2.2 de0: using obsoleted if_watchdog interface de0: Ethernet address: 00:00:d1:1f:cb:d5 de0: [ITHREAD] de1: <Digital 21140A Fast Ethernet> port 0xc100-0xc17f mem 0xfc040000-0xfc04007f irq 17 at device 5.0 on pci2 de1: Cogent EM440TX 21140A [10-100Mb/s] pass 2.2 de1: using obsoleted if_watchdog interface de1: Ethernet address: 00:00:d1:1f:cb:d6 de2: <Digital 21140A Fast Ethernet> port 0xc200-0xc27f mem 0xfc041000-0xfc04107f irq 18 at device 6.0 on pci2 de2: Cogent EM440TX 21140A [10-100Mb/s] pass 2.2 de2: using obsoleted if_watchdog interface de2: Ethernet address: 00:00:d1:1f:cb:d7 de3: <Digital 21140A Fast Ethernet> port 0xc300-0xc37f mem 0xfc042000-0xfc04207f irq 19 at device 7.0 on pci2 de3: Cogent EM440TX 21140A [10-100Mb/s] pass 2.2 de3: using obsoleted if_watchdog interface de3: Ethernet address: 00:00:d1:1f:cb:d8 atapci0: <VIA 6420 SATA150 controller> port 0xdb00-0xdb07,0xe100-0xe103,0xd800-0xd807,0xd900-0xd903,0xda00-0xda0f,0xd000-0xd0ff irq 20 at device 15.0 on pci0 atapci0: [ITHREAD] ata2: <ATA channel 0> on atapci0 ata2: [ITHREAD] ata3: <ATA channel 1> on atapci0 ata3: [ITHREAD] atapci1: <VIA 8237 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xdc00-0xdc0f at device 15.1 on pci0 ata0: <ATA channel 0> on atapci1 ata0: [ITHREAD] ata1: <ATA channel 1> on atapci1 ata1: [ITHREAD] isab0: <PCI-ISA bridge> at device 17.0 on pci0 isa0: <ISA bus> on isab0 vr0: <VIA VT6102 Rhine II 10/100BaseTX> port 0xd400-0xd4ff mem 0xfc101000-0xfc1010ff irq 23 at device 18.0 on pci0 vr0: Quirks: 0x0 miibus0: <MII bus> on vr0 rlphy0: <RTL8201L 10/100 media interface> PHY 1 on miibus0 rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto vr0: using obsoleted if_watchdog interface vr0: Ethernet address: 00:19:21:46:2d:56 vr0: [ITHREAD] acpi_tz0: <Thermal Zone> on acpi0 sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio0: [FILTER] atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] pmtimer0 on isa0 orm0: <ISA Option ROM> at iomem 0xcc000-0xcdfff pnpid ORM0000 on isa0 sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppbus0: <Parallel port bus> on ppc0 ppbus0: [ITHREAD] plip0: <PLIP network interface> on ppbus0 lpt0: <Printer> on ppbus0 lpt0: Interrupt-driven port ppi0: <Parallel I/O> on ppbus0 ppc0: [GIANT-LOCKED] ppc0: [ITHREAD] sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled Timecounter "TSC" frequency 1999795935 Hz quality 800 Timecounters tick every 1.000 msec ipfw2 (+ipv6) initialized, divert enabled, rule-based forwarding disabled, default to deny, logging disabled acd0: CDRW <LITE-ON LTR-24102B/5QSB> at ata0-master UDMA33 ad4: 152627MB <WDC WD1600AAJS-00PSA0 05.06H05> at ata2-master SATA150 Trying to mount root from ufs:/dev/ad4s1a WARNING: / was not properly dismounted WARNING: /tmp was not properly dismounted WARNING: /usr was not properly dismounted WARNING: /var was not properly dismounted IP Filter: v4.1.28 initialized. Default = pass all, Logging = enabled Here's the debug bt: kgdb /usr/src/sys/i386/compile/KNL04072009/kernel.debug /var/crash/vmcore.1 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x4 fault code = supervisor read, page not present instruction pointer = 0x20:0xc388994b stack pointer = 0x28:0xdceb59c8 frame pointer = 0x28:0xdceb5a44 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 26 (irq23: vr0) trap number = 12 panic: page fault cpuid = 0 Uptime: 5d7h9m18s Physical memory: 742 MB Dumping 119 MB: 104 88 72 56 40 24 8 #0 doadump () at pcpu.h:195 195 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:195 #1 0xc05ba657 in boot (howto=260) at ../../../kern/kern_shutdown.c:409 #2 0xc05ba919 in panic (fmt=Variable "fmt" is not available. ) at ../../../kern/kern_shutdown.c:563 #3 0xc080c54c in trap_fatal (frame=0xdceb5988, eva=4) at ../../../i386/i386/trap.c:899 #4 0xc080c7b0 in trap_pfault (frame=0xdceb5988, usermode=0, eva=4) at ../../../i386/i386/trap.c:812 #5 0xc080d132 in trap (frame=0xdceb5988) at ../../../i386/i386/trap.c:490 #6 0xc07f3abb in calltrap () at ../../../i386/i386/exception.s:139 #7 0xc388994b in ?? () Previous frame inner to this frame (corrupt stack?) Here's a pciconf -lv output: hostb0@pci0:0:0:0: class=0x060000 card=0x02041106 chip=0x02041106 rev=0x00 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'K8M400 CPU to PCI Bridge' class = bridge subclass = HOST-PCI hostb1@pci0:0:0:1: class=0x060000 card=0x00000000 chip=0x12041106 rev=0x00 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'K8M400 CPU to PCI Bridge' class = bridge subclass = HOST-PCI hostb2@pci0:0:0:2: class=0x060000 card=0x00000000 chip=0x22041106 rev=0x00 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'K8M400 CPU to PCI Bridge' class = bridge subclass = HOST-PCI hostb3@pci0:0:0:3: class=0x060000 card=0x00000000 chip=0x32041106 rev=0x00 hdr=0x00 vendor = 'VIA Technologies Inc' device = '1394 i2c CPU to PCI Bridge' class = bridge subclass = HOST-PCI hostb4@pci0:0:0:4: class=0x060000 card=0x00000000 chip=0x42041106 rev=0x00 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'K8M400 CPU to PCI Bridge' class = bridge subclass = HOST-PCI hostb5@pci0:0:0:7: class=0x060000 card=0x00000000 chip=0x72041106 rev=0x00 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'K8M400 CPU to PCI Bridge' class = bridge subclass = HOST-PCI pcib1@pci0:0:1:0: class=0x060400 card=0x00000000 chip=0xb1881106 rev=0x00 hdr=0x01 vendor = 'VIA Technologies Inc' device = 'VT8237 K8HTB CPU to AGP 2.0/3.0 Bridge' class = bridge subclass = PCI-PCI pcib2@pci0:0:8:0: class=0x060400 card=0x00000000 chip=0x00241011 rev=0x03 hdr=0x01 vendor = 'Digital Equipment Corporation' device = '21151/2 PCI to PCI Bridge' class = bridge subclass = PCI-PCI atapci0@pci0:0:15:0: class=0x01018f card=0x32061565 chip=0x31491106 rev=0x80 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'VT8237 VT6410 SATA RAID Controller' class = mass storage subclass = ATA atapci1@pci0:0:15:1: class=0x01018a card=0x32061565 chip=0x05711106 rev=0x06 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'VT82C586A/B/VT82C686/A/B/VT823x/A/C Bus Master IDE Controller' class = mass storage subclass = ATA isab0@pci0:0:17:0: class=0x060100 card=0x32061565 chip=0x32271106 rev=0x00 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'VT8237 PCI-to-ISA Bridge' class = bridge subclass = PCI-ISA vr0@pci0:0:18:0: class=0x020000 card=0x01021106 chip=0x30651106 rev=0x78 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'VT6102 Rhine II PCI Fast Ethernet Controller||Used by GERICOM in laptop Webengine Advanced' class = network subclass = ethernet hostb6@pci0:0:24:0: class=0x060000 card=0x00000000 chip=0x11001022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' device = '(K8) Athlon 64/Opteron HyperTransport Technology Configuration' class = bridge subclass = HOST-PCI hostb7@pci0:0:24:1: class=0x060000 card=0x00000000 chip=0x11011022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' device = '(K8) Athlon 64/Opteron Address Map' class = bridge subclass = HOST-PCI hostb8@pci0:0:24:2: class=0x060000 card=0x00000000 chip=0x11021022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' device = '(K8) Athlon 64/Opteron DRAM Controller' class = bridge subclass = HOST-PCI hostb9@pci0:0:24:3: class=0x060000 card=0x00000000 chip=0x11031022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' device = '(K8) Athlon 64/Opteron Miscellaneous Control' class = bridge subclass = HOST-PCI vgapci0@pci0:1:0:0: class=0x030000 card=0x12031565 chip=0x31081106 rev=0x01 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'Unknown Via Unichrome S3 VGA - part of a VIA Northbridge' class = display subclass = VGA de0@pci0:2:4:0: class=0x020000 card=0x24001109 chip=0x00091011 rev=0x22 hdr=0x00 vendor = 'Digital Equipment Corporation' device = 'DecChip 21140 Fast Ethernet Adapter' class = network subclass = ethernet de1@pci0:2:5:0: class=0x020000 card=0x00000000 chip=0x00091011 rev=0x22 hdr=0x00 vendor = 'Digital Equipment Corporation' device = 'DecChip 21140 Fast Ethernet Adapter' class = network subclass = ethernet de2@pci0:2:6:0: class=0x020000 card=0x00000000 chip=0x00091011 rev=0x22 hdr=0x00 vendor = 'Digital Equipment Corporation' device = 'DecChip 21140 Fast Ethernet Adapter' class = network subclass = ethernet de3@pci0:2:7:0: class=0x020000 card=0x00000000 chip=0x00091011 rev=0x22 hdr=0x00 vendor = 'Digital Equipment Corporation' device = 'DecChip 21140 Fast Ethernet Adapter' class = network subclass = ethernet Thanks in advance folks! I appreciate any assistance offered. -- Erich M. Jenkins Fuujin Group Limited PO Box 792 Brainerd, MN 56401 (p) 218-824-5038 (f) 218-824-7516 Visit us on the web at http://www.fuujingroup.com "You should never, never doubt what no one is sure about." -- Gene Wilder
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?49DE8EAE.7080605>