Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 16 Mar 2004 09:08:18 -0500
From:      Sven Willenberger <sven@dmv.com>
To:        freebsd-current@freebsd.org
Subject:   kmem_map too small, revisited
Message-ID:  <1079446098.23554.49.camel@lanshark.dmv.com>

next in thread | raw e-mail | index | archive | help
Preamble: I am looking for assistance in how to better troubleshoot this
issue of kmem_map. I have googled and read the newsgroups to no avail.

The Issue: Ever since having migrated from 4.x to 5.x I have been having
an issue (on 2 different hardware setups) with spontaneous reboots or
system hangs, usually identified by the kmem_map too small message. The
hardware is predominantly Supermicro dual Xeon processor machines
(though a single xeon non S/M machine exhibits similar behavior). 

The machines are used essentially as email filtering machines running
sendmail/mimedefang/spamassassin and process upwards of 150K messages
per day each. With the exception of 1 single processor machine, all have
dual procs with 1GB of ECC DRAM.

Where should I begin to better diagnose what exactly is going on here? 

Latest dmesg:
Sorry, need DDB option to print backtrace
panic: kmem_malloc(16384): kmem_map too small: 588120064 total allocated
cpuid = 2;
boot() called on cpu#2

syncing disks, buffers remaining... 6073 6073 6070 6070 6070 6070 6070
6070 6070 6070 6070 6070 6070 6070 6070 6070 6070 6070 6070 6070 6070
6070
Copyright (c) 1992-2004 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights
reserved.
FreeBSD 5.2.1-RC #0: Mon Feb  2 10:01:14 EST 2004
    svenw@cartman.dmv.com:/usr/obj/usr/src/sys/CARTMAN
Preloaded elf kernel "/boot/kernel/kernel" at 0xc081d000.
Preloaded elf module "/boot/kernel/acpi.ko" at 0xc081d2bc.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 2.00GHz (1996.60-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf27  Stepping = 7
 
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Hyperthreading: 2 logical CPUs
real memory  = 1073217536 (1023 MB)
avail memory = 1032687616 (984 MB)
ACPI APIC Table: <PTLTD          APIC  >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  6
 cpu3 (AP): APIC ID:  7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic2 <Version 2.0> irqs 48-71 on motherboard
Pentium Pro MTRR support enabled
npx0: [FAST]
npx0: <math processor> on motherboard
npx0: INT 16 interface
acpi0: <PTLTD    RSDT  > on motherboard
pcibios: BIOS version 2.10
Using $PIR table, 25 entries at 0xc00fde30
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
acpi_cpu0: <CPU> on acpi0
acpi_cpu1: <CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <unknown> at device 0.1 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pcib1: could not get PCI interrupt routing table for \\_SB_.PCI0.HLB_ -
AE_NOT_FOUND
pci1: <ACPI PCI bus> on pcib1
pci1: <base peripheral, interrupt controller> at device 28.0 (no driver
attached)
pcib2: <ACPI PCI-PCI bridge> at device 29.0 on pci1
pci2: <ACPI PCI bus> on pcib2
em0: <Intel(R) PRO/1000 Network Connection, Version - 1.7.19> port
0x3000-0x301f mem 0xfc200000-0xfc21ffff,0xfc220000-0xfc23ffff irq 54 at
device 3.0 on pci2
em0:  Speed:N/A  Duplex:N/A
pci1: <base peripheral, interrupt controller> at device 30.0 (no driver
attached)
pcib3: <ACPI PCI-PCI bridge> at device 31.0 on pci1
pci3: <ACPI PCI bus> on pcib3
ahc0: <Adaptec aic7899 Ultra160 SCSI adapter> port 0x4000-0x40ff mem
0xfc300000-0xfc300fff irq 32 at device 2.0 on pci3
aic7899: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
ahc1: <Adaptec aic7899 Ultra160 SCSI adapter> port 0x4400-0x44ff mem
0xfc301000-0xfc301fff irq 33 at device 2.1 on pci3
aic7899: Ultra160 Wide Channel B, SCSI Id=7, 32/253 SCBs
uhci0: <Intel 82801CA/CAM (ICH3) USB controller USB-A> port
0x2000-0x201f irq 16 at device 29.0 on pci0
usb0: <Intel 82801CA/CAM (ICH3) USB controller USB-A> on uhci0
usb0: USB revision 1.0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <Intel 82801CA/CAM (ICH3) USB controller USB-B> port
0x2020-0x203f irq 19 at device 29.1 on pci0
usb1: <Intel 82801CA/CAM (ICH3) USB controller USB-B> on uhci1
usb1: USB revision 1.0
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <Intel 82801CA/CAM (ICH3) USB controller USB-C> port
0x2040-0x205f irq 18 at device 29.2 on pci0
usb2: <Intel 82801CA/CAM (ICH3) USB controller USB-C> on uhci2
usb2: USB revision 1.0
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
pcib4: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pci4: <display, VGA> at device 4.0 (no driver attached)
fxp0: <Intel 82550 Pro/100 Ethernet> port 0x5400-0x543f mem
0xfc420000-0xfc43ffff,0xfc401000-0xfc401fff irq 22 at device 5.0 on pci4
fxp0: Ethernet address 00:30:48:24:ca:e1
miibus0: <MII bus> on fxp0
inphy0: <i82555 10/100 media interface> on miibus0
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH3 UDMA100 controller> port
0x2060-0x206f,0x374-0x377,0x170-0x177,0x3f4-0x3f7,0x1f0-0x1f7 mem
0xfc000000-0xfc0003ff at device 31.1 on pci0
ata0: at 0x1f0 irq 14 on atapci0
ata0: [MPSAFE]
ata1: at 0x170 irq 15 on atapci0
ata1: [MPSAFE]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x64,0x60 irq 1 on acpi0
sio0 port 0x3f8-0x3ff irq 4 on acpi0
sio0: type 16550A
sio1 port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
fdc0: <Enhanced floppy controller (i82077, NE72065 or clone)> port
0x3f7,0x3f0-0x3f5 irq 6 drq 2 on acpi0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
ppc0 port 0x778-0x77f,0x378-0x37f irq 7 drq 3 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/9 bytes threshold
ppbus0: <Parallel port bus> on ppc0
ppi0: <Parallel I/O> on ppbus0
pmtimer0 on isa0
orm0: <Option ROMs> at iomem
0xe0000-0xe3fff,0xdc000-0xdffff,0xc8000-0xc8fff,0xc0000-0xc7fff on isa0
sc0: <System console> on isa0
sc0: VGA <16 virtual consoles, flags=0x200>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on
isa0
Timecounters tick every 10.000 msec
Waiting 7 seconds for SCSI devices to settle
GEOM: create disk da0 dp=0xc6ba9c50
da0 at ahc1 bus 0 target 3 lun 0
da0: <IBM IC35L018UWD210-0 S5CS> Fixed Direct Access SCSI-3 device
da0: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged
Queueing Enabled
da0: 17501MB (35843670 512 byte sectors: 255H 63S/T 2231C)
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
Mounting root from ufs:/dev/da0s1a
WARNING: / was not properly dismounted
WARNING: /tmp was not properly dismounted
WARNING: /usr was not properly dismounted
WARNING: /var was not properly dismounted
/var: mount pending error: blocks 1364 files 126

Snippet from the kernel CONF file on what is now a 5.2.1-RC machine:

# $FreeBSD: src/sys/i386/conf/GENERIC,v 1.384.2.2 2003/05/31 15:18:41
scottl Exp $

machine         i386
cpu             I586_CPU
cpu             I686_CPU
ident           GENERIC

#To statically compile in device wiring instead of /boot/device.hints
#hints          "GENERIC.hints"         #Default places to look for
devices.

#makeoptions    DEBUG=-g                #Build kernel with gdb(1) debug
symbols

options         SCHED_4BSD              #4BSD scheduler
options         INET                    #InterNETworking
options         INET6                   #IPv6 communications protocols
options         FFS                     #Berkeley Fast Filesystem
options         SOFTUPDATES             #Enable FFS soft updates support
options         UFS_ACL                 #Support for access control
lists
options         UFS_DIRHASH             #Improve performance on big
directories
options         MD_ROOT                 #MD is a potential root device
options         NFSCLIENT               #Network Filesystem Client
options         NFSSERVER               #Network Filesystem Server
options         NFS_ROOT                #NFS usable as root device,
requires NFSCLIENT
options         MSDOSFS                 #MSDOS Filesystem
options         CD9660                  #ISO 9660 Filesystem
options         PROCFS                  #Process filesystem (requires
PSEUDOFS)
options         PSEUDOFS                #Pseudo-filesystem framework
options         COMPAT_43               #Compatible with BSD 4.3 [KEEP
THIS!]
options         COMPAT_FREEBSD4         #Compatible with FreeBSD4
options         SCSI_DELAY=7000         #Delay (in ms) before probing
SCSI
options         KTRACE                  #ktrace(1) support
options         SYSVSHM                 #SYSV-style shared memory
options         SYSVMSG                 #SYSV-style message queues
options         SYSVSEM                 #SYSV-style semaphores
options         _KPOSIX_PRIORITY_SCHEDULING #Posix P1003_1B real-time
extensions
options         KBD_INSTALL_CDEV        # install a CDEV entry in /dev
options         AHC_REG_PRETTY_PRINT    # Print register bitfields in
debug
                                        # output.  Adds ~128k to driver.
options         AHD_REG_PRETTY_PRINT    # Print register bitfields in
debug
                                        # output.  Adds ~215k to driver.

# Debugging for use in -current
#options        DDB                     #Enable the kernel debugger
#options        INVARIANTS              #Enable calls of extra sanity
checking
options         INVARIANT_SUPPORT       #Extra sanity checks of internal
structures, required by INVARIANTS
#options        WITNESS                 #Enable checks to detect
deadlocks and cycles
#options        WITNESS_SKIPSPIN        #Don't run witness on spinlocks
for speed

# Deal with kmem issues
options                 VM_KMEM_SIZE_SCALE="2"
options                 VM_KMEM_SIZE_MAX="(512*1024*1024)"
#options                        KVA_PAGES=640

# To make an SMP kernel, the next two are needed
options         SMP                     # Symmetric MultiProcessor
Kernel
device          apic                    # Symmetric (APIC) I/O

Thanks in advance,

Sven




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1079446098.23554.49.camel>