Date: Sun, 10 Jul 2005 16:13:23 +0200 (CEST) From: Blaz Zupan <blaz@si.FreeBSD.org> To: FreeBSD-gnats-submit@FreeBSD.org Subject: kern/83220: Daily crashes on 5.4 SMP (with backtrace) Message-ID: <20050710141323.4264E55475@titanic.medinet.si> Resent-Message-ID: <200507101420.j6AEKPAN021780@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 83220
>Category: kern
>Synopsis: Daily crashes on 5.4 SMP (with backtrace)
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Sun Jul 10 14:20:24 GMT 2005
>Closed-Date:
>Last-Modified:
>Originator: Blaz Zupan
>Release: FreeBSD 5.4-RELEASE
>Organization:
>Environment:
System: FreeBSD in-4.mail.amis.net 5.4-RELEASE-p3 FreeBSD 5.4-RELEASE-p3 #0: Tue Jul 5 18:37:15 CEST 2005 blaz@bigbrother.amis.net:/usr/obj/usr/src5/sys/DL380 i386
Copyright (c) 1992-2005 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 5.4-RELEASE-p3 #0: Tue Jul 5 18:37:15 CEST 2005
blaz@bigbrother.amis.net:/usr/obj/usr/src5/sys/DL380
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 3.06GHz (3050.08-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0xf29 Stepping = 9
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Hyperthreading: 2 logical CPUs
real memory = 1073717248 (1023 MB)
avail memory = 1045372928 (996 MB)
ACPI APIC Table: <COMPAQ 00000083>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 6
cpu3 (AP): APIC ID: 7
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-15 on motherboard
ioapic1 <Version 1.1> irqs 16-31 on motherboard
ioapic2 <Version 1.1> irqs 32-47 on motherboard
ioapic3 <Version 1.1> irqs 48-63 on motherboard
npx0: <math processor> on motherboard
npx0: INT 16 interface
acpi0: <COMPAQ P29> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 1000
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x920-0x923 on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <display, VGA> at device 3.0 (no driver attached)
pci0: <base peripheral> at device 4.0 (no driver attached)
pci0: <base peripheral> at device 4.2 (no driver attached)
isab0: <PCI-ISA bridge> at device 15.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <ServerWorks CSB5 UDMA100 controller> port 0x2000-0x200f,0x376,0x170-0x177,0x3f6,0x1f0-0x1f7 at device 15.1 on pci0
ata0: channel #0 on atapci0
ata1: channel #1 on atapci0
ohci0: <OHCI (generic) USB controller> mem 0xf5ef0000-0xf5ef0fff irq 7 at device 15.2 on pci0
usb0: OHCI version 1.0, legacy support
usb0: SMM does not respond, resetting
usb0: <OHCI (generic) USB controller> on ohci0
usb0: USB revision 1.0
uhub0: (0x1166) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 4 ports with 4 removable, self powered
pcib1: <ACPI Host-PCI bridge> on acpi0
pci1: <ACPI PCI bus> on pcib1
ciss0: <Compaq Smart Array 5i> port 0x3000-0x30ff mem 0xf7cf0000-0xf7cf3fff,0xf7dc0000-0xf7dfffff irq 30 at device 3.0 on pci1
pcib2: <ACPI Host-PCI bridge> on acpi0
pci2: <ACPI PCI bus> on pcib2
bge0: <Broadcom BCM5703 Gigabit Ethernet, ASIC rev. 0x1002> mem 0xf7ef0000-0xf7efffff irq 29 at device 1.0 on pci2
miibus0: <MII bus> on bge0
brgphy0: <BCM5703 10/100/1000baseTX PHY> on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto
bge0: Ethernet address: 00:0e:7f:20:22:91
bge1: <Broadcom BCM5703 Gigabit Ethernet, ASIC rev. 0x1002> mem 0xf7ee0000-0xf7eeffff irq 31 at device 2.0 on pci2
miibus1: <MII bus> on bge1
brgphy1: <BCM5703 10/100/1000baseTX PHY> on miibus1
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto
bge1: Ethernet address: 00:0e:7f:20:22:90
pcib3: <ACPI Host-PCI bridge> on acpi0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI Host-PCI bridge> on acpi0
pci6: <ACPI PCI bus> on pcib4
pci6: <base peripheral, PCI hot-plug controller> at device 30.0 (no driver attached)
acpi_tz0: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x64,0x60 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: model Generic PS/2 mouse, device ID 0
sio0: <Standard PC COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
fdc0: <floppy drive controller (FDE)> port 0x3f2-0x3f5 irq 6 drq 2 on acpi0
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
orm0: <ISA Option ROMs> at iomem 0xee000-0xeffff,0xcc000-0xcd7ff,0xc8000-0xcbfff,0xc0000-0xc7fff on isa0
pmtimer0 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 10.000 msec
IP Filter: v3.4.35 initialized. Default = pass all, Logging = enabled
acd0: CDROM <COMPAQ CD-ROM SN-124/N104> at ata0-master PIO4
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
da0 at ciss0 bus 0 target 0 lun 0
da0: <COMPAQ RAID 5 VOLUME OK> Fixed Direct Access SCSI-0 device
da0: 135.168MB/s transfers
da0: 69455MB (142245120 512 byte sectors: 255H 32S/T 17432C)
Kernel config file:
machine i386
cpu I686_CPU
ident DL380
options SCHED_4BSD # 4BSD scheduler
options INET # InterNETworking
options INET6 # IPv6 communications protocols
options FFS # Berkeley Fast Filesystem
options SOFTUPDATES # Enable FFS soft updates support
options UFS_ACL # Support for access control lists
options UFS_DIRHASH # Improve performance on big directories
options MD_ROOT # MD is a potential root device
options GEOM_GPT # GUID Partition Tables.
options COMPAT_43 # Compatible with BSD 4.3 [KEEP THIS!]
options COMPAT_FREEBSD4 # Compatible with FreeBSD4
options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI
options KTRACE # ktrace(1) support
options SYSVSHM # SYSV-style shared memory
options SYSVMSG # SYSV-style message queues
options SYSVSEM # SYSV-style semaphores
options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options ADAPTIVE_GIANT # Giant mutex is adaptive.
options NMBCLUSTERS=12000
options IPFILTER
options IPFILTER_LOG
options SMP
options INCLUDE_CONFIG_FILE
options KDB_STOP_NMI
options KDB
options DDB
makeoptions DEBUG=-g #Build kernel with gdb(1) debug symbols
device apic # I/O APIC
device isa
device eisa
device pci
device fdc
device ata
device atapicd # ATAPI CDROM drives
options ATA_STATIC_ID # Static device numbering
device scbus # SCSI bus (required for SCSI)
device da # Direct Access (disks)
device ciss # Compaq Smart RAID 5*
device atkbdc # AT keyboard controller
device atkbd # AT keyboard
device psm # PS/2 mouse
device vga # VGA video card driver
device sc
device agp # support several AGP chipsets
device npx
device pmtimer
device sio # 8250, 16[45]50 based serial ports
device miibus # MII bus support
device bge # Broadcom BCM570xx Gigabit Ethernet
device loop # Network loopback
device mem # Memory and kernel memory devices
device io # I/O device
device random # Entropy device
device ether # Ethernet support
device pty # Pseudo-ttys (telnet etc)
device md # Memory "disks"
device bpf # Berkeley packet filter
device ohci # OHCI PCI->USB interface
device usb # USB Bus (required)
device ukbd # Keyboard
device ums # Mouse
>Description:
Crash happens with both HTT turned on and off in the BIOS. Machine is a
heavily loaded incoming mail server, running postfix, amavisd-new and
F-Secure under Linux emulation. ipfilter is running.
See also this thread on freebsd-stable:
http://lists.freebsd.org/pipermail/freebsd-stable/2005-July/016767.html
Below is a backtrace on a crashdump. Crashdump and kernel compiled with -g
are available on request.
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
#0 doadump () at pcpu.h:159
159 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) bt
#0 doadump () at pcpu.h:159
#1 0xc044b006 in db_fncall (dummy1=0, dummy2=0, dummy3=-1067606609, dummy4=0xe4b6c9d0 "üɶä(\205]Àèɶäìɶä\222\a")
at /usr/src5/sys/ddb/db_command.c:531
#2 0xc044ae14 in db_command (last_cmdp=0xc0674644, cmd_table=0x0, aux_cmd_tablep=0xc064226c, aux_cmd_tablep_end=0xc0642270)
at /usr/src5/sys/ddb/db_command.c:349
#3 0xc044aedc in db_command_loop () at /usr/src5/sys/ddb/db_command.c:455
#4 0xc044ca75 in db_trap (type=12, code=0) at /usr/src5/sys/ddb/db_main.c:221
#5 0xc04e6599 in kdb_trap (type=12, code=0, tf=0xe4b6cb3c) at /usr/src5/sys/kern/subr_kdb.c:468
#6 0xc05f4c79 in trap_fatal (frame=0xe4b6cb3c, eva=36) at /usr/src5/sys/i386/i386/trap.c:812
#7 0xc05f43e9 in trap (frame=
{tf_fs = -1040580584, tf_es = -1029439472, tf_ds = 16, tf_edi = -1038000128, tf_esi = -1066898900, tf_ebp = -457782384, tf_isp = -457782424, tf_ebx = -1040530304, tf_edx = -1040524364, tf_ecx = -1040524544, tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = -1068574101, tf_cs = 8, tf_eflags = 65683, tf_esp = 180, tf_ss = 0}) at /usr/src5/sys/i386/i386/trap.c:255
#8 0xc05e283a in calltrap () at /usr/src5/sys/i386/i386/exception.s:140
#9 0xc1fa0018 in ?? ()
#10 0xc2a40010 in ?? ()
#11 0x00000010 in ?? ()
#12 0xc2216000 in ?? ()
#13 0xc0686a2c in tcbinfo ()
#14 0xe4b6cb90 in ?? ()
#15 0xe4b6cb68 in ?? ()
#16 0xc1fac480 in ?? ()
#17 0xc1fadbb4 in ?? ()
#18 0xc1fadb00 in ?? ()
#19 0x00000000 in ?? ()
#20 0x0000000c in ?? ()
#21 0x00000000 in ?? ()
#22 0xc04eda6b in propagate_priority (td=0xc2216000) at /usr/src5/sys/kern/subr_turnstile.c:243
#23 0xc04ee225 in turnstile_wait (ts=0xc1fadb00, lock=0xc0686a2c, owner=0xc2216000)
at /usr/src5/sys/kern/subr_turnstile.c:556
#24 0xc04c5ced in _mtx_lock_sleep (m=0xc0686a2c, td=0xc1fac480, opts=0, file=0x0, line=0)
at /usr/src5/sys/kern/kern_mutex.c:552
#25 0xc0559ad8 in tcp_usr_rcvd (so=0x0, flags=0) at /usr/src5/sys/netinet/tcp_usrreq.c:602
#26 0xc0506103 in soreceive (so=0xc27bf798, psa=0x0, uio=0xe4b6cc88, mp0=0x0, controlp=0x0, flagsp=0x0)
at /usr/src5/sys/kern/uipc_socket.c:1395
#27 0xc04f4bd9 in soo_read (fp=0x0, uio=0xe4b6cc88, active_cred=0xc2884a80, flags=0, td=0xc1fac480)
at /usr/src5/sys/kern/sys_socket.c:91
#28 0xc04ee865 in dofileread (td=0xc1fac480, fp=0xc2e17bb0, fd=10, buf=0x0, nbyte=4096, offset=Unhandled dwarf expression opcode 0x93
) at file.h:233
#29 0xc04ee72f in read (td=0xc1fac480, uap=0xe4b6cd14) at /usr/src5/sys/kern/sys_generic.c:107
#30 0xc05f4fe7 in syscall (frame=
{tf_fs = 47, tf_es = 47, tf_ds = -1078001617, tf_edi = 10, tf_esi = 300, tf_ebp = -1077942168, tf_isp = -457781900, tf_ebx = 134822152, tf_edx = 0, tf_ecx = 10, tf_eax = 3, tf_trapno = 0, tf_err = 2, tf_eip = 672556795, tf_cs = 31, tf_eflags = 658, tf_esp = -1077942212, tf_ss = 47}) at /usr/src5/sys/i386/i386/trap.c:1009
#31 0xc05e288f in Xint0x80_syscall () at /usr/src5/sys/i386/i386/exception.s:201
#32 0x0000002f in ?? ()
#33 0x0000002f in ?? ()
#34 0xbfbf002f in ?? ()
#35 0x0000000a in ?? ()
#36 0x0000012c in ?? ()
#37 0xbfbfe868 in ?? ()
#38 0xe4b6cd74 in ?? ()
#39 0x08093908 in ?? ()
#40 0x00000000 in ?? ()
#41 0x0000000a in ?? ()
#42 0x00000003 in ?? ()
#43 0x00000000 in ?? ()
#44 0x00000002 in ?? ()
#45 0x281666fb in ?? ()
#46 0x0000001f in ?? ()
#47 0x00000292 in ?? ()
#48 0xbfbfe83c in ?? ()
#49 0x0000002f in ?? ()
#50 0x00000000 in ?? ()
#51 0x00000000 in ?? ()
#52 0x00000000 in ?? ()
#53 0x00000000 in ?? ()
#54 0x2c75b000 in ?? ()
#55 0xc22de000 in ?? ()
#56 0xc1fac480 in ?? ()
#57 0xe4b6ccac in ?? ()
#58 0xe4b6cc94 in ?? ()
#59 0xc1f26000 in ?? ()
#60 0xc04ded13 in sched_switch (td=0x12c, newtd=0x8093908, flags=Cannot access memory at address 0xbfbfe878
) at /usr/src5/sys/kern/sched_4bsd.c:881
Previous frame inner to this frame (corrupt stack?)
(kgdb) quit
>How-To-Repeat:
Only happens on SMP boxes, a very simmilar HP DL380 G3 box with only a
single processor has currently an uptime of 12 days, while a SMP box
crashes at least once daily.
>Fix:
Unknown
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20050710141323.4264E55475>