Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 15 Jun 2004 22:34:57 +0200 (CEST)
From:      Tamas TEVESZ <ice@wormhole.hu>
To:        freebsd-hackers@freebsd.org
Subject:   4.10-RELEASE and -STABLE crashing regularly under load
Message-ID:  <Pine.LNX.4.58.0406152233570.25116@dawn.royalcomp.hu>

next in thread | raw e-mail | index | archive | help

hi folks,

[i've posted the following message to -bugs@ a while ago, but
then was directed here by a friend freebsder. while reposting, i
also corrected some minor facts i missed in the previous post]

i have a dell poweredge 2600 (4G ram, 2x2.8ghz xeon cpus, some disk,
full dmesg below), running a heavily loaded website (apache13, php, cgi,
pure-ftpd). this is a brand new 4.10-release install, brought to sync
with -stable, both exhibit the exact same problem), which every once
in a while crashes badly. 4.10-R did that every ~2.5 days, 4.10-S did
it for the first time after one day. (before that, system was running
4.9-stable on a poweredge 4600 with one xeon cpu, no ht, no smp,
no nothing like that, and was very stable).

i cannot entirely rule out bad hardware as this is a brand new system,
but we haven't had many problems with dell stuff before.

everything i think to be related is included below; if anything else
is needed just please tell so.

thanks in advance.


misc related information:
==================================

# sysctl machdep.hlt_logical_cpus
machdep.hlt_logical_cpus: 1
# kldstat
Id Refs Address    Size     Name
 1    2 0xc0100000 1e5214   kernel
 2    1 0xc02e6000 21c4     accf_http.ko
#


information from gdb:
==================================

# gdb -k kernel.debug.1 vmcore.1
GNU gdb 4.18 (FreeBSD)
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-unknown-freebsd"...Deprecated bfd_read called at /usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line 2627 in elfstab_build_psymtabs
Deprecated bfd_read called at /usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line 933 in fill_symbuf

SMP 4 cpus
IdlePTD at physical address 0x00309000
initial pcb at physical address 0x0027b3c0
panicstr: page fault
panic messages:
---
Fatal trap 12: page fault while in kernel mode
mp_lock = 02000002; cpuid = 2; lapic.id = 06000000
fault virtual address   = 0xbfc00000
fault code    = supervisor write, page not present
instruction pointer     = 0x8:0xc0213fd9
stack pointer         = 0x10:0xfc749e04
frame pointer         = 0x10:0xfc749e10
code segment   = base 0x0, limit 0xfffff, type 0x1b
      = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 73408 (grep)
interrupt mask  = none <- SMP: XXX
trap number    = 12
panic: page fault
mp_lock = 02000002; cpuid = 2; lapic.id = 06000000
boot() called on cpu#2

syncing disks... 45 7
done
Uptime: 1d0h14m54s
amr0: flushing cache...done
amr1: flushing cache...done

dumping to dev #amrd/0x20001, offset 1048960
dump 3583 3582 3581 3580 3579 3578 3577 3576 3575 3574 3573 3572 3571 [...]
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
487   if (dumping++) {
(kgdb) bt
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
#1  0xc01664df in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:316
#2  0xc0166938 in poweroff_wait (junk=0xc024ff79, howto=-1071318481) at /usr/src/sys/kern/kern_shutdown.c:595
#3  0xc0217e98 in trap_fatal (frame=0xfc749dc4, eva=3217031168) at /usr/src/sys/i386/i386/trap.c:974
#4  0xc0217b29 in trap_pfault (frame=0xfc749dc4, usermode=0, eva=3217031168) at /usr/src/sys/i386/i386/trap.c:867
#5  0xc02176c7 in trap (frame={tf_fs = 24, tf_es = -68485104, tf_ds = 134610960, tf_edi = -99396280, tf_esi = 0, tf_ebp = -59466224,
      tf_isp = -59466256, tf_ebx = 3, tf_edx = -1043777528, tf_ecx = 0, tf_eax = 1245573123, tf_trapno = 12, tf_err = 2, tf_eip = -1071562791,
      tf_cs = 8, tf_eflags = 66054, tf_esp = 134660096, tf_ss = 134660096}) at /usr/src/sys/i386/i386/trap.c:466
#6  0xc0213fd9 in pmap_qenter (va=0, m=0xfa135548, count=4) at /usr/src/sys/i386/i386/pmap.c:848
#7  0xc017711a in pipe_build_write_buffer (wpipe=0xfa135520, uio=0xfc749ed0) at /usr/src/sys/kern/sys_pipe.c:594
#8  0xc01772e0 in pipe_direct_write (wpipe=0xfa135520, uio=0xfc749ed0) at /usr/src/sys/kern/sys_pipe.c:709
#9  0xc0177682 in pipe_write (fp=0xce43cec0, uio=0xfc749ed0, cred=0xcd0c6080, flags=0, p=0xfbeb3ee0) at /usr/src/sys/kern/sys_pipe.c:827
#10 0xc0175a05 in dofilewrite (p=0xfbeb3ee0, fp=0xce43cec0, fd=1, buf=0x8068000, nbyte=16384, offset=-1, flags=0)
    at /usr/src/sys/sys/file.h:163
#11 0xc01758be in write (p=0xfbeb3ee0, uap=0xfc749f80) at /usr/src/sys/kern/sys_generic.c:329
#12 0xc02181c9 in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47, tf_edi = 134643712, tf_esi = 672187864, tf_ebp = -1077937456,
      tf_isp = -59465772, tf_ebx = 672188332, tf_edx = 672187864, tf_ecx = 0, tf_eax = 4, tf_trapno = 12, tf_err = 2, tf_eip = 672141560,
      tf_cs = 31, tf_eflags = 663, tf_esp = -1077937500, tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:1175
#13 0xc02056fb in Xint0x80_syscall ()
#14 0x280fedd2 in ?? ()
#15 0x280fed41 in ?? ()
#16 0x280fbc26 in ?? ()
#17 0x280a50d5 in ?? ()
#18 0x804ec04 in ?? ()
#19 0x804edc6 in ?? ()
#20 0x804eec5 in ?? ()
#21 0x804f0c7 in ?? ()
#22 0x804f3a4 in ?? ()
#23 0x80500f3 in ?? ()
#24 0x8049046 in ?? ()
(kgdb) list *0xc0213fd9
0xc0213fd9 is in pmap_qenter (/usr/src/sys/i386/i386/pmap.c:848).
843  void
844  pmap_qenter(vm_offset_t va, vm_page_t *m, int count)
845  {
846   while (count-- > 0) {
847     pt_entry_t *pte = vtopte(va);
848     *pte = VM_PAGE_TO_PHYS(*m) | PG_RW | PG_V | pgeflag;
849  #ifdef SMP
850     cpu_invlpg((void *)va);
851  #else
852     invltlb_1pg(va);
(kgdb)

dmesg:
==================================

Copyright (c) 1992-2004 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD 4.10-STABLE #3: Mon Jun 14 12:39:29 CEST 2004
    root@mammut.swi.hu:/usr/obj/usr/src/sys/MAMMUT
Timecounter "i8254"  frequency 1193182 Hz
CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2791.01-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf29  Stepping = 9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Hyperthreading: 2 logical CPUs
real memory  = 3757899776 (3669824K bytes)
avail memory = 3660173312 (3574388K bytes)
Changing APIC ID for IO APIC #0 from 0 to 8 on chip
Changing APIC ID for IO APIC #1 from 0 to 9 on chip
Changing APIC ID for IO APIC #2 from 0 to 10 on chip
Changing APIC ID for IO APIC #3 from 0 to 11 on chip
Changing APIC ID for IO APIC #4 from 0 to 12 on chip
Programming 24 pins in IOAPIC #0
IOAPIC #0 intpin 2 -> irq 0
Programming 24 pins in IOAPIC #1
Programming 24 pins in IOAPIC #2
Programming 24 pins in IOAPIC #3
Programming 24 pins in IOAPIC #4
FreeBSD/SMP: Multiprocessor motherboard: 4 CPUs
 cpu0 (BSP): apic id:  0, version: 0x00050014, at 0xfee00000
 cpu1 (AP):  apic id:  1, version: 0x00050014, at 0xfee00000
 cpu2 (AP):  apic id:  6, version: 0x00050014, at 0xfee00000
 cpu3 (AP):  apic id:  7, version: 0x00050014, at 0xfee00000
 io0 (APIC): apic id:  8, version: 0x00178020, at 0xfec00000
 io1 (APIC): apic id:  9, version: 0x00178020, at 0xfec80000
 io2 (APIC): apic id: 10, version: 0x00178020, at 0xfec81000
 io3 (APIC): apic id: 11, version: 0x00178020, at 0xfec82000
 io4 (APIC): apic id: 12, version: 0x00178020, at 0xfec82800
Preloaded elf kernel "kernel" at 0xc02ea000.
Preloaded elf module "accf_http.ko" at 0xc02ea09c.
Warning: Pentium 4 CPU: PSE disabled
module_register_init: MOD_LOAD (accf_http, c0180dc0, 0xc02e7a60) error 17
Pentium Pro MTRR support enabled
md0: Malloc disk
Using $PIR table, 12 entries at 0xc00fc160
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <Host to PCI bridge> on motherboard
IOAPIC #0 intpin 16 -> irq 2
pci0: <PCI bus> on pcib0
pcib1: <PCI to PCI bridge (vendor=8086 device=2543)> at device 2.0 on pci0
pci1: <PCI bus> on pcib1
pci1: <unknown card> (vendor=0x8086, dev=0x1461) at 28.0
pcib2: <PCI to PCI bridge (vendor=8086 device=1460)> at device 29.0 on pci1
pci2: <PCI bus> on pcib2
pci1: <unknown card> (vendor=0x8086, dev=0x1461) at 30.0
pcib3: <PCI to PCI bridge (vendor=8086 device=1460)> at device 31.0 on pci1
IOAPIC #1 intpin 4 -> irq 5
pci3: <PCI bus> on pcib3
em0: <Intel(R) PRO/1000 Network Connection, Version - 1.7.25> port 0xece0-0xecff mem 0xfdcc0000-0xfdcdffff,0xfdce0000-0xfdcfffff irq 5 at device 1.0 on pci3
em0:  Speed:N/A  Duplex:N/A
pcib4: <PCI to PCI bridge (vendor=8086 device=2545)> at device 3.0 on pci0
pci4: <PCI bus> on pcib4
pci4: <unknown card> (vendor=0x8086, dev=0x1461) at 28.0
pcib5: <PCI to PCI bridge (vendor=8086 device=1460)> at device 29.0 on pci4
pci5: <PCI bus> on pcib5
pci4: <unknown card> (vendor=0x8086, dev=0x1461) at 30.0
pcib6: <PCI to PCI bridge (vendor=8086 device=1460)> at device 31.0 on pci4
pci6: <PCI bus> on pcib6
pcib7: <PCI to PCI bridge (vendor=8086 device=2547)> at device 4.0 on pci0
pci7: <PCI bus> on pcib7
pci7: <unknown card> (vendor=0x8086, dev=0x1461) at 28.0
pcib8: <PCI to PCI bridge (vendor=8086 device=1460)> at device 29.0 on pci7
IOAPIC #3 intpin 0 -> irq 7
pci8: <PCI bus> on pcib8
amr0: <LSILogic MegaRAID> mem 0xf7ff0000-0xf7ffffff irq 7 at device 8.0 on pci8
amr0: <LSILogic PERC 4/Di> Firmware 2.48, BIOS 1.06, 128MB RAM
pci7: <unknown card> (vendor=0x8086, dev=0x1461) at 30.0
pcib9: <PCI to PCI bridge (vendor=8086 device=1460)> at device 31.0 on pci7
pci10: <PCI bus> on pcib9
pcib10: <PCI to PCI bridge (vendor=8086 device=b154)> at device 6.0 on pci10
IOAPIC #4 intpin 1 -> irq 11
pci11: <PCI bus> on pcib10
pcib11: <PCI to PCI bridge (vendor=8086 device=b154)> at device 0.0 on pci11
IOAPIC #4 intpin 0 -> irq 13
pci12: <PCI bus> on pcib11
amr1: <LSILogic MegaRAID> mem 0xe8000000-0xefffffff irq 13 at device 0.0 on pci12
amr1: <LSILogic PERC 3/DC> Firmware 1.80, BIOS 3.29, 128MB RAM
pci11: <unknown card> (vendor=0x1077, dev=0x1216) at 1.0 irq 11
pci0: <UHCI USB controller> at 29.0 irq 2
pcib12: <Intel 82801BA/BAM (ICH2) Hub to PCI bridge> at device 30.0 on pci0
pci13: <PCI bus> on pcib12
pci13: <ATI Mach64-GR graphics accelerator> at 4.0
isab0: <PCI to ISA bridge (vendor=8086 device=2480)> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH3 ATA100 controller> port 0xfc00-0xfc0f,0-0x3,0-0x7,0-0x3,0-0x7 irq 2 at device 31.1 on pci0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
orm0: <Option ROMs> at iomem 0xc0000-0xc7fff,0xec000-0xeffff on isa0
pmtimer0 on isa0
fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 8250
sio1: configured irq 3 not in bitmap of probed irqs 0
APIC_IO: Testing 8254 interrupt delivery
APIC_IO: routing 8254 via IOAPIC #0 intpin 2
IP packet filtering initialized, divert disabled, rule-based forwarding enabled, default to accept, logging disabled
IP Filter: v3.4.31 initialized.  Default = pass all, Logging = enabled
ata0-slave: ATAPI identify retries exceeded
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
acd0: CDROM <TEAC CD-ROM CD-224E> at ata0-master PIO4
amrd0: <LSILogic MegaRAID logical drive> on amr0
amrd0: 34680MB (71024640 sectors) RAID 1 (optimal)
amrd1: <LSILogic MegaRAID logical drive> on amr0
amrd1: 139900MB (286515200 sectors) RAID 0 (optimal)
amrd2: <LSILogic MegaRAID logical drive> on amr1
amrd2: 450814MB (923267072 sectors) RAID 5 (optimal)
amrd3: <LSILogic MegaRAID logical drive> on amr1
amrd3: 450814MB (923267072 sectors) RAID 5 (optimal)
SMP: AP CPU #3 Launched!
pass0 at amr0 bus 0 target 6 lun 0
pass0: <PE/PV 1x6 SCSI BP 1.1> Fixed Processor SCSI-2 device
pass1 at amr1 bus 0 target 6 lun 0
pass1: <DELL PV22XS E.14> Fixed Processor SCSI-3 device
pass2 at amr1 bus 1 target 6 lun 0
pass2: <DELL PV22XS E.14> Fixed Processor SCSI-3 device
Mounting root from ufs:/dev/amrd0s1a
WARNING: / was not properly dismounted

config:
==================================

machine		i386
cpu		I686_CPU
ident		mammut
maxusers	0

options         NMBCLUSTERS=32768
options		PMAP_SHPGPERPROC=256
options		ACCEPT_FILTER_HTTP

makeoptions	DEBUG=-g

options 	INET			#InterNETworking
options 	FFS			#Berkeley Fast Filesystem
options 	FFS_ROOT		#FFS usable as root device [keep this!]
options 	SOFTUPDATES		#Enable FFS soft updates support
options 	UFS_DIRHASH		#Improve performance on big directories
options 	MFS			#Memory Filesystem
options 	MD_ROOT			#MD is a potential root device
options 	CD9660			#ISO 9660 Filesystem
options 	CD9660_ROOT		#CD-ROM usable as root, CD9660 required
options 	PROCFS			#Process filesystem
options 	COMPAT_43		#Compatible with BSD 4.3 [KEEP THIS!]
options 	SCSI_DELAY=15000	#Delay (in ms) before probing SCSI
options 	UCONSOLE		#Allow users to grab the console
options 	USERCONFIG		#boot -c editor
options 	VISUAL_USERCONFIG	#visual boot -c editor
options 	KTRACE			#ktrace(1) support
options 	SYSVMSG			#SYSV-style message queues
options 	SYSVSEM			#SYSV-style semaphores
options 	P1003_1B		#Posix P1003_1B real-time extensions
options 	_KPOSIX_PRIORITY_SCHEDULING
options 	ICMP_BANDLIM		#Rate limit bad replies
options 	KBD_INSTALL_CDEV	# install a CDEV entry in /dev

options 	SMP			# Symmetric MultiProcessor Kernel
options 	APIC_IO			# Symmetric (APIC) I/O
options		IPFILTER
options		IPFILTER_LOG
options		IPFIREWALL
options		IPFIREWALL_DEFAULT_TO_ACCEPT

device		isa
device		eisa
device		pci

device		fdc0	at isa? port IO_FD1 irq 6 drq 2
device		fd0	at fdc0 drive 0

device		ata0	at isa? port IO_WD1 irq 14
device		ata
device		atapicd			# ATAPI CDROM drives
options 	ATA_STATIC_ID		#Static device numbering

device		scbus		# SCSI bus (required)
device		da		# Direct Access (disks)
device		pass		# Passthrough device (direct SCSI access)

device		amr		# AMI MegaRAID

device		atkbdc0	at isa? port IO_KBD
device		atkbd0	at atkbdc? irq 1 flags 0x1

device		vga0	at isa?

device		sc0	at isa? flags 0x100

device		npx0	at nexus? port IO_NPX irq 13

device		sio0	at isa? port IO_COM1 flags 0x10 irq 4
device		sio1	at isa? port IO_COM2 irq 3

device		miibus		# MII bus support
device		em

pseudo-device	loop		# Network loopback
pseudo-device	ether		# Ethernet support
pseudo-device	pty		# Pseudo-ttys (telnet etc)
pseudo-device	md		# Memory "disks"
pseudo-device	bpf		#Berkeley packet filter

-- 
[-]

``Early to rise, early to bed, makes a man healthy, wealthy and dead.''



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.LNX.4.58.0406152233570.25116>