Date: Tue, 8 Aug 2006 00:09:13 GMT From: Dave Clausen <d.clausen@kualo.com> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/101618: kernel panic on multiple Dell PE2850s Message-ID: <200608080009.k7809D2Z056152@www.freebsd.org> Resent-Message-ID: <200608080010.k780AHOx096412@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 101618
>Category: kern
>Synopsis: kernel panic on multiple Dell PE2850s
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Tue Aug 08 00:10:16 GMT 2006
>Closed-Date:
>Last-Modified:
>Originator: Dave Clausen
>Release: FreeBSD 5.5-RELEASE i386
>Organization:
Kualo LTD
>Environment:
Problem exists on multiple Dell PowerEdge 2850s dual Xeons with HT.
System: FreeBSD kernbuild.kgix.net 5.5-RELEASE FreeBSD 5.5-RELEASE #0: Sun May 28 22:29:05 UTC 2006 tech@kernbuild.kgix.net:/usr/obj/usr/src/sys/KGIX-SMP-i386 i386
>Description:
We are experiencing multiple kernel panics per day on a series of Dell PowerEdge 2850 servers each running Cpanel (dcpumon is a cpu monitoring process distributed with cpanel).
Kernel is GENERIC plus firewalling.
dmesg:
Copyright (c) 1992-2006 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 5.5-RELEASE #0: Sun May 28 22:29:05 UTC 2006
tech@kernbuild.kgix.net:/usr/obj/usr/src/sys/KGIX-SMP-i386
ACPI APIC Table: <DELL PE BKC >
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 3.00GHz (2992.51-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0xf43 Stepping = 3
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Hyperthreading: 2 logical CPUs
real memory = 2147221504 (2047 MB)
avail memory = 2095767552 (1998 MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 6
cpu3 (AP): APIC ID: 7
ioapic0: Changing APIC ID to 8
ioapic1: Changing APIC ID to 9
ioapic1: WARNING: intbase 32 != expected base 24
ioapic2: Changing APIC ID to 10
ioapic2: WARNING: intbase 64 != expected base 56
ioapic3: Changing APIC ID to 11
ioapic3: WARNING: intbase 96 != expected base 88
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 32-55 on motherboard
ioapic2 <Version 2.0> irqs 64-87 on motherboard
ioapic3 <Version 2.0> irqs 96-119 on motherboard
npx0: <math processor> on motherboard
npx0: INT 16 interface
acpi0: <DELL PE BKC> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
amr0: <LSILogic MegaRAID 1.51> mem 0xfeac0000-0xfeafffff,0xf80f0000-0xf80fffff irq 46 at device 14.0 on pci2
amr0: <LSILogic PERC 4e/Di> Firmware 521S, BIOS H430, 256MB RAM
pcib3: <ACPI PCI-PCI bridge> at device 0.2 on pci1
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 5.0 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 0.0 on pci5
pci6: <ACPI PCI bus> on pcib6
em0: <Intel(R) PRO/1000 Network Connection, Version - 1.7.35> port 0xecc0-0xecff mem 0xfe7e0000-0xfe7fffff irq 64 at device 7.0 on pci6
em0: Ethernet address: 00:14:22:20:34:5a
pcib7: <ACPI PCI-PCI bridge> at device 0.2 on pci5
pci7: <ACPI PCI bus> on pcib7
em1: <Intel(R) PRO/1000 Network Connection, Version - 1.7.35> port 0xdcc0-0xdcff mem 0xfe5e0000-0xfe5fffff irq 65 at device 8.0 on pci7
em1: Ethernet address: 00:14:22:20:34:5b
pcib8: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci8: <ACPI PCI bus> on pcib8
pcib9: <ACPI PCI-PCI bridge> at device 0.0 on pci8
pci9: <ACPI PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 0.2 on pci8
pci10: <ACPI PCI bus> on pcib10
pcib11: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci11: <ACPI PCI bus> on pcib11
pci11: <display, VGA> at device 13.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH5 UDMA100 controller> port 0xfc00-0xfc0f,0x376,0x170-0x177,0x3f6,0x1f0-0x1f7 at device 31.1 on pci0
ata0: channel #0 on atapci0
ata1: channel #1 on atapci0
fdc0: <floppy drive controller> port 0x3f7,0x3f0-0x3f5 irq 6 drq 2 on acpi0
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> port 0x64,0x60 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xec000-0xeffff,0xc0000-0xcafff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: parallel port not found.
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
Timecounters tick every 10.000 msec
ipfw2 initialized, divert disabled, rule-based forwarding enabled, default to accept, logging limited to 100 packets/entry by default
acd0: CDROM <TEAC CD-ROM CD-224E-N/3.AB> at ata0-master PIO4
amrd0: <LSILogic MegaRAID logical drive> on amr0
amrd0: 69880MB (143114240 sectors) RAID 1 (optimal)
amrd1: <LSILogic MegaRAID logical drive> on amr0
amrd1: 69880MB (143114240 sectors) RAID 0 (optimal)
ses0 at amr0 bus 0 target 6 lun 0
ses0: <PE/PV 1x6 SCSI BP 1.0> Fixed Processor SCSI-2 device
ses0: SAF-TE Compliant Device
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
Mounting root from ufs:/dev/amrd0s1a
WARNING: / was not properly dismounted
kgdb output:
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 06
fault virtual address = 0x1c
fault code = supervisor write, page not present
instruction pointer = 0x8:0xc062dd01
stack pointer = 0x10:0xf15fa780
frame pointer = 0x10:0xf15fa79c
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 23694 (dcpumon)
trap number = 12
panic: page fault
cpuid = 2
boot() called on cpu#0
Uptime: 21h7m11s
Dumping 2047 MB
16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496 512 528 544 560 576 592 608 624 640 656 672 688 704 720 736 752 768 784 800 816 832 848 864 880 896 912 928 944 960 976 992 1008 1024 1040 1056 1072 1088 1104 1120 1136 1152 1168 1184 1200 1216 1232 1248 1264 1280 1296 1312 1328 1344 1360 1376 1392 1408 1424 1440 1456 1472 1488 1504 1520 1536 1552 1568 1584 1600 1616 1632 1648 1664 1680 1696 1712 1728 1744 1760 1776 1792 1808 1824 1840 1856 1872 1888 1904 1920 1936 1952 1968 1984 2000 2016 2032
#0 doadump () at pcpu.h:160
160 __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb)
(kgdb) bt
#0 doadump () at pcpu.h:160
#1 0xc05cca59 in boot (howto=260) at ../../../kern/kern_shutdown.c:412
#2 0xc05ccd7d in panic (fmt=0xc07c4944 "%s")
at ../../../kern/kern_shutdown.c:568
#3 0xc0786410 in trap_fatal (frame=0xf15fa740, eva=28)
at ../../../i386/i386/trap.c:822
#4 0xc078614f in trap_pfault (frame=0xf15fa740, usermode=0, eva=28)
at ../../../i386/i386/trap.c:737
#5 0xc0785d89 in trap (frame=
{tf_fs = -1067581416, tf_es = -1065615344, tf_ds = -1067581424, tf_edi = 4, tf_esi = 0, tf_ebp = -245389412, tf_isp = -245389460, tf_ebx = 131074, tf_edx = -979096576, tf_ecx = 0, tf_eax = 4, tf_trapno = 12, tf_err = 2, tf_eip = -1067262719, tf_cs = 8, tf_eflags = 66118, tf_esp = 7, tf_ss = -245388144})
at ../../../i386/i386/trap.c:427
#6 0xc077385a in calltrap () at ../../../i386/i386/exception.s:140
#7 0xc05e0018 in devclass_find_free_unit (dc=0x0, unit=0)
at ../../../kern/subr_bus.c:1147
#8 0xc058bc85 in procfs_doprocfile (td=0x0, p=0x20002, pn=0xc3674c80,
sb=0xf15fa7f0, uio=0x0) at ../../../fs/procfs/procfs.c:73
#9 0xc058fef4 in pfs_readlink (va=0x0) at pcpu.h:157
#10 0xc0628c1c in kern_readlink (td=0xc5a42c00, path=0x0,
pathseg=UIO_USERSPACE, buf=0x0, bufseg=UIO_USERSPACE, count=1024)
at vnode_if.h:925
---Type <return> to continue, or q <return> to quit---
#11 0xc0628b42 in readlink (td=0xc5a42c00, uap=0x0)
at ../../../kern/vfs_syscalls.c:2197
#12 0xc078674b in syscall (frame=
{tf_fs = 47, tf_es = 47, tf_ds = 47, tf_edi = 135507064, tf_esi = 135655436, tf_ebp = -1077941016, tf_isp = -245387932, tf_ebx = 674068596, tf_edx = -1077942040, tf_ecx = 0, tf_eax = 58, tf_trapno = 47, tf_err = 2, tf_eip = 672570948, tf_cs = 31, tf_eflags = 643, tf_esp = -1077942100, tf_ss = 47})
at ../../../i386/i386/trap.c:1014
#13 0xc07738af in Xint0x80_syscall () at ../../../i386/i386/exception.s:201
#14 0x0000002f in ?? ()
#15 0x0000002f in ?? ()
#16 0x0000002f in ?? ()
#17 0x0813ac78 in ?? ()
#18 0x0815f00c in ?? ()
#19 0xbfbfece8 in ?? ()
#20 0xf15fad64 in ?? ()
#21 0x282d7874 in ?? ()
#22 0xbfbfe8e8 in ?? ()
#23 0x00000000 in ?? ()
#24 0x0000003a in ?? ()
#25 0x0000002f in ?? ()
#26 0x00000002 in ?? ()
#27 0x28169e44 in ?? ()
---Type <return> to continue, or q <return> to quit---
#28 0x0000001f in ?? ()
#29 0x00000283 in ?? ()
#30 0xbfbfe8ac in ?? ()
#31 0x0000002f in ?? ()
#32 0x00000000 in ?? ()
#33 0x00000000 in ?? ()
#34 0x00000000 in ?? ()
#35 0x00000000 in ?? ()
#36 0x415ff000 in ?? ()
#37 0xc8668a98 in ?? ()
#38 0xc5a42c00 in ?? ()
#39 0xf15fa600 in ?? ()
#40 0xf15fa5e8 in ?? ()
#41 0xc3547300 in ?? ()
#42 0xc05dd0c3 in sched_switch (td=0x815f00c, newtd=0x282d7874, flags=Cannot access memory at address 0xbfbfecf8
)
at ../../../kern/sched_4bsd.c:881
Previous frame inner to this frame (corrupt stack?)
(kgdb)
>How-To-Repeat:
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200608080009.k7809D2Z056152>