Date: Wed, 10 Dec 2008 15:25:20 GMT From: Mike <mike@nux.co.uk> To: freebsd-gnats-submit@FreeBSD.org Subject: i386/129550: crash with PAE kernel Message-ID: <200812101525.mBAFPKoC070074@www.freebsd.org> Resent-Message-ID: <200812101530.mBAFU14l092117@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 129550
>Category: i386
>Synopsis: crash with PAE kernel
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: freebsd-i386
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Wed Dec 10 15:30:01 UTC 2008
>Closed-Date:
>Last-Modified:
>Originator: Mike
>Release: 7
>Organization:
Nux
>Environment:
FreeBSD redbus-lingo1950.nux.co.uk 7.0-RELEASE-p6 FreeBSD 7.0-RELEASE-p6 #7: Tue Dec 2 00:48:03 GMT 2008 razvan@redbus-lingo1950.nux.co.uk:/usr/obj/usr/src/sys/PAE i386
>Description:
Hi,
We have a problem with dell poweredge 1950 server and freebsd 7.
The machine is used as a jail host with one jail running (with ezjail-admin), other than the standard stuff like syslog and ssh the main progam causing any siginificant work is dovecot - this has been compiled with kqueue support.
Other than the below is there any other information we can provide to diagnose this problem - the machine is a live server with about 250gb of Maildirs on it with about 200 users connected at the same time.
>From /var/log/messages:
Dec 10 10:00:39 redbus-lingo1950 su: razvan to root on /dev/ttyp0
Dec 10 14:21:49 redbus-lingo1950 kernel:
Dec 10 14:21:49 redbus-lingo1950 kernel:
Dec 10 14:21:49 redbus-lingo1950 kernel: Fatal trap 12: page fault while in kernel mode
Dec 10 14:21:49 redbus-lingo1950 kernel: cpuid = 3; apic id = 03
Dec 10 14:21:49 redbus-lingo1950 kernel: fault virtual address = 0x0
Dec 10 14:21:49 redbus-lingo1950 kernel: fault code = supervisor read, page not
Dec 10 14:21:49 redbus-lingo1950 kernel: present
Dec 10 14:21:49 redbus-lingo1950 kernel: instruction pointer = 0x20:0x0
Dec 10 14:21:49 redbus-lingo1950 kernel: stack pointer = 0x28:0xf4955a28
Dec 10 14:21:49 redbus-lingo1950 kernel: frame pointer = 0x28:0xf4955a48
Dec 10 14:21:49 redbus-lingo1950 kernel: code segment = base 0x0, limit 0xfffff, type 0x1b
Dec 10 14:21:49 redbus-lingo1950 kernel: = DPL 0,
Dec 10 14:21:49 redbus-lingo1950 kernel:
Dec 10 14:38:57 redbus-lingo1950 syslogd: kernel boot file is /boot/kernel/kernel
redbus-lingo1950# kgdb kernel.debug /var/crash/vmcore.0
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 57762 (imap)
trap number = 12
panic: page fault
cpuid = 3
Uptime: 8d13h27m30s
Physical memory: 8175 MB
Dumping 475 MB: 460 444 428 412 396 380 364 348 332 316 300 284 268 252 236 220 204 188 172 156 140 124 108 92 76 60 44 28 12
#0 doadump () at pcpu.h:195
195 __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) list *0x20:0x0
A syntax error in expression, near `:0x0'.
(kgdb) list "0x20:0x0"
No source file named 0x20.
(kgdb) backtrace
#0 doadump () at pcpu.h:195
#1 0xc04ed227 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2 0xc04ed4e9 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3 0xc072f24c in trap_fatal (frame=0xf49559e8, eva=0) at /usr/src/sys/i386/i386/trap.c:899
#4 0xc072f4e0 in trap_pfault (frame=0xf49559e8, usermode=0, eva=0) at /usr/src/sys/i386/i386/trap.c:812
#5 0xc072fe8c in trap (frame=0xf49559e8) at /usr/src/sys/i386/i386/trap.c:490
#6 0xc071482b in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7 0x00000000 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) quit
Other info:
redbus-lingo1950# uname -a
FreeBSD redbus-lingo1950.nux.co.uk 7.0-RELEASE-p6 FreeBSD 7.0-RELEASE-p6 #7: Tue Dec 2 00:48:03 GMT 2008 razvan@redbus-lingo1950.nux.co.uk:/usr/obj/usr/src/sys/PAE i386
redbus-lingo1950# mount
/dev/mfid0s1a on / (ufs, local)
devfs on /dev (devfs, local)
/dev/mfid0s1g on /home (ufs, local, soft-updates)
/dev/mfid0s1e on /tmp (ufs, local, soft-updates)
/dev/mfid0s1f on /usr (ufs, local, soft-updates)
/dev/mfid0s1d on /var (ufs, local, soft-updates)
/home/jails/basejail on /home/jails/mail.lingo24.com/basejail (nullfs, local, read-only)
devfs on /home/jails/mail.lingo24.com/dev (devfs, local)
redbus-lingo1950# sysctl kern.sched.name
kern.sched.name: ULE
Kernel config is standard PAE but with ULE and the following added:
# make pf in kernel as wwell
device pf
device pflog
# need to add nullfs
device nullfs
redbus-lingo1950# dmesg
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-RELEASE-p6 #7: Tue Dec 2 00:48:03 GMT 2008
razvan@redbus-lingo1950.nux.co.uk:/usr/obj/usr/src/sys/PAE
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU L5335 @ 2.00GHz (1997.03-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0x6fb Stepping = 11
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x4e33d<SSE3,RSVD2,MON,DS_CPL,VMX,TM2,SSSE3,CX16,xTPR,PDCM,DCA>
AMD Features=0x20100000<NX,LM>
AMD Features2=0x1<LAHF>
Cores per package: 4
real memory = 9395240960 (8960 MB)
avail memory = 8394002432 (8005 MB)
ACPI APIC Table: <DELL PE_SC3 >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP): APIC ID: 3
cpu4 (AP): APIC ID: 4
cpu5 (AP): APIC ID: 5
cpu6 (AP): APIC ID: 6
cpu7 (AP): APIC ID: 7
ioapic0: Changing APIC ID to 8
ioapic0 <Version 2.0> irqs 0-23 on motherboard
registered firmware set <isp_1040>
registered firmware set <isp_1040_it>
registered firmware set <isp_1080>
registered firmware set <isp_1080_it>
registered firmware set <isp_12160>
registered firmware set <isp_12160_it>
registered firmware set <isp_2100>
registered firmware set <isp_2200>
registered firmware set <isp_2300>
registered firmware set <isp_2322>
registered firmware set <isp_2400>
kbd1 at kbdmux0
acpi0: <DELL PE_SC3> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
cpu0: <ACPI CPU> on acpi0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
p4tcc1: <CPU Frequency Thermal Control> on cpu1
cpu2: <ACPI CPU> on acpi0
p4tcc2: <CPU Frequency Thermal Control> on cpu2
cpu3: <ACPI CPU> on acpi0
p4tcc3: <CPU Frequency Thermal Control> on cpu3
cpu4: <ACPI CPU> on acpi0
p4tcc4: <CPU Frequency Thermal Control> on cpu4
cpu5: <ACPI CPU> on acpi0
p4tcc5: <CPU Frequency Thermal Control> on cpu5
cpu6: <ACPI CPU> on acpi0
p4tcc6: <CPU Frequency Thermal Control> on cpu6
cpu7: <ACPI CPU> on acpi0
p4tcc7: <CPU Frequency Thermal Control> on cpu7
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci4: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci4
pci5: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 0.0 on pci5
pci6: <ACPI PCI bus> on pcib3
pcib4: <PCI-PCI bridge> at device 0.0 on pci6
pci7: <PCI bus> on pcib4
bce0: <Broadcom NetXtreme II BCM5708 1000Base-T (B2)> mem 0xf4000000-0xf5ffffff irq 16 at device 0.0 on pci7
miibus0: <MII bus> on bce0
brgphy0: <BCM5708C 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bce0: Ethernet address: 00:1d:09:65:93:41
bce0: [ITHREAD]
bce0: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); F/W (0x03050C05); Flags( MFW MSI )
pcib5: <ACPI PCI-PCI bridge> at device 1.0 on pci5
pci8: <ACPI PCI bus> on pcib5
pcib6: <PCI-PCI bridge> at device 0.3 on pci4
pci9: <PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci1: <ACPI PCI bus> on pcib7
mfi0: <LSI MegaSAS 1078> port 0xec00-0xecff mem 0xfc480000-0xfc4bffff,0xfc440000-0xfc47ffff irq 16 at device 0.0 on pci1
mfi0: Megaraid SAS driver Ver 2.00
mfi0: 2433 (282234152s/0x0020/0) - Shutdown command received from host
mfi0: 2434 (4278190083s/0x0020/0) - PCI 0x041000 0x0460 0x041028 0x041f0c: Firmware initialization started (PCI ID 0060/1000/1f0c/1028)
mfi0: 2435 (4278190083s/0x0020/0) - Type 18: Firmware version 1.11.52-0349
mfi0: 2436 (4278190083s/0x0008/0) - Battery Present
mfi0: 2437 (4278190083s/0x0020/0) - Type 18: Package version 6.0.1-0080
mfi0: 2438 (4278190101s/0x0004/0) - PD 32(e0/s255) event: Enclosure (SES) discovered on PD 20(c None/p0)
mfi0: 2439 (4278190101s/0x0002/0) - PD 32(e32/s255) event: Inserted: Encl PD 20
mfi0: 2440 (4278190101s/0x0002/0) - Type 29: Inserted: PD 20(c None/p0) Info: enclPd=20, scsiType=d, portMap=09, sasAddr=5001c030dd3f2b00,0000000000000000
mfi0: 2441 (4278190101s/0x0002/0) - PD 00(e32/s0) event: Inserted: PD 00(e0x20/s0)
mfi0: 2442 (4278190101s/0x0002/0) - Type 29: Inserted: PD 00(e0x20/s0) Info: enclPd=20, scsiType=0, portMap=00, sasAddr=5000c5000336be59,0000000000000000
mfi0: 2443 (4278190101s/0x0002/0) - PD 01(e32/s1) event: Inserted: PD 01(e0x20/s1)
mfi0: 2444 (4278190101s/0x0002/0) - Type 29: Inserted: PD 01(e0x20/s1) Info: enclPd=20, scsiType=0, portMap=01, sasAddr=5000c5000336b375,0000000000000000
mfi0: 2445 (282234233s/0x0020/0) - Adapter ticks 282234233 elapsed 59s: Time established as 12/10/08 14:23:53; (59 seconds since power on)
mfi0: 2446 (282234242s/0x0008/0) - Battery temperature is normal
mfi0: 2447 (282234242s/0x0008/0) - Current capacity of the battery is above threshold
mfi0: [ITHREAD]
pcib8: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci10: <ACPI PCI bus> on pcib8
pcib9: <PCI-PCI bridge> at device 5.0 on pci0
pci11: <PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci12: <ACPI PCI bus> on pcib10
pcib11: <PCI-PCI bridge> at device 7.0 on pci0
pci13: <PCI bus> on pcib11
pcib12: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib12
pcib13: <PCI-PCI bridge> at device 0.0 on pci2
pci3: <PCI bus> on pcib13
bce1: <Broadcom NetXtreme II BCM5708 1000Base-T (B2)> mem 0xf8000000-0xf9ffffff irq 16 at device 0.0 on pci3
miibus1: <MII bus> on bce1
brgphy1: <BCM5708C 10/100/1000baseTX PHY> PHY 1 on miibus1
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bce1: Ethernet address: 00:1d:09:65:93:3f
bce1: [ITHREAD]
bce1: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); F/W (0x03050C05); Flags( MFW MSI )
pci0: <serial bus, USB> at device 29.0 (no driver attached)
pci0: <serial bus, USB> at device 29.1 (no driver attached)
pci0: <serial bus, USB> at device 29.2 (no driver attached)
pci0: <serial bus, USB> at device 29.3 (no driver attached)
pci0: <serial bus, USB> at device 29.7 (no driver attached)
pcib14: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci14: <ACPI PCI bus> on pcib14
vgapci0: <VGA-compatible display> port 0xdc00-0xdcff mem 0xd8000000-0xdfffffff,0xfc2d0000-0xfc2dffff irq 19 at device 13.0 on pci14
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A, console
sio0: [FILTER]
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc8fff,0xc9000-0xc9fff,0xd2000-0xd2fff,0xec000-0xeffff pnpid ORM0000 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x100>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
acd0: CDROM <TEAC CD-ROM CD-224E-N/3.AC> at ata0-master UDMA33
mfid0: <MFI Logical Disk> on mfi0
mfid0: 380928MB (780140544 sectors) RAID volume '' is optimal
SMP: AP CPU #1 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #3 Launched!
Trying to mount root from ufs:/dev/mfid0s1a
WARNING: / was not properly dismounted
redbus-lingo1950#
>How-To-Repeat:
unknown
>Fix:
unknown
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200812101525.mBAFPKoC070074>