From owner-freebsd-stable@FreeBSD.ORG Wed Oct 25 22:19:57 2006 Return-Path: X-Original-To: freebsd-stable@FreeBSD.org Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5186416A403; Wed, 25 Oct 2006 22:19:57 +0000 (UTC) (envelope-from stb@lassitu.de) Received: from koef.zs64.net (koef.zs64.net [212.12.50.230]) by mx1.FreeBSD.org (Postfix) with ESMTP id 34FC943D5C; Wed, 25 Oct 2006 22:19:55 +0000 (GMT) (envelope-from stb@lassitu.de) Received: (from stb@koef.zs64.net) (authenticated) by koef.zs64.net (8.13.8/8.13.8) with ESMTP id k9PMJkSr003224 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Thu, 26 Oct 2006 00:19:47 +0200 (CEST) (envelope-from stb@lassitu.de) In-Reply-To: <20061025183308.L33725@fledge.watson.org> References: <20061025183308.L33725@fledge.watson.org> Mime-Version: 1.0 (Apple Message framework v752.2) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <838FCA83-20F8-4A09-A025-E69956032F86@lassitu.de> Content-Transfer-Encoding: 7bit From: Stefan Bethke Date: Thu, 26 Oct 2006 00:19:45 +0200 To: Robert Watson X-Mailer: Apple Mail (2.752.2) Cc: Andreas Sons , FreeBSD Stable Subject: Re: panic: kmem_map too small X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Oct 2006 22:19:57 -0000 Am 25.10.2006 um 19:36 schrieb Robert Watson: > > On Wed, 25 Oct 2006, Stefan Bethke wrote: > >> We're consistely getting this panic even under smallish loads. >> I've experimented with various values for VM_KMEM_SIZE_MAX (384, >> 512, 768 and 1024 MB), but the boxes are still panicking after >> even short periods (a few hours) just doing a buildworld, a few >> ports compiles, or even when logging out of the console(?). >> >> Would logging vm.zone every few minutes help detecting a kernel >> memory leak? >> >> I'm at a complete loss as to the actual cause of these. The ACPI >> errors precede the panic below by only a few seconds, so I'd guess >> they're a symptom, not a cause. We've tried with ACPI disabled in >> the BIOS, but we also got these panics, so we re-enabled it. >> >> The hardware is a Tyan GT20 (2865) with a single-core Opteron, two >> gigs of RAM and two or three SATA disks. >> >> Any ideas? > > Try the commands "show uma" and "show malloc" in DDB to print the > memory allocations made by the uma(9) and malloc(9) kernel memory > allocators. This output may be sufficient to suggest to us where a > kernel memory leak, if any, might be taking place. Also, a stack > trace ("trace") never hurts; if something is sitting there > allocating a lot of memory at high speed, it could well be that the > last call to the memory allocator is the one leaking. Thanks, see below. I'll let it sit in the debugger, if someone has some more ideas for what to look at. This panic was triggered by me trying to ssh into the box; it had been sitting idle since the last reboot around 1900 CEST. Note that I had to break into the debugger manually. I'm assuming that the memory shortage is so severe that dumping cannot be initiated, so the kernel hangs... Thanks, Stefan login: cblock_alloc_cblocks: M_NOWAIT malloc failed, trying M_WAITOK panic: kmem_malloc(4096): kmem_map too small: 699756544 total allocated Uptime: 5h4m51s KDB: enter: Break sequence on console [thread pid 1545 tid 100066 ] Stopped at kdb_enter+0x30: leave db> show uma Zone Allocs Frees Used Cache FFS2 dinode 529 80 449 1 FFS1 dinode 0 0 0 0 FFS inode 529 80 449 15 Mountpoints 4 0 4 8 SWAPMETA 0 0 0 0 rtentry 5 0 5 53 unpcb 784 772 12 44 ripcb 0 0 0 0 sackhole 0 0 0 0 tcpreass 0 0 0 0 hostcache 2 1 1 99 syncache 105 105 0 78 tcptw 1 1 0 156 tcpcb 117 109 8 16 inpcb 117 109 8 36 udpcb 288 276 12 32 ipq 0 0 0 0 socket 1190 1158 32 12 KNOTE 58 58 0 112 PIPE 655 654 1 17 NFSNODE 0 0 0 0 NFSMOUNT 0 0 0 0 DIRHASH 29 0 29 7 NAMEI 30174 30174 0 12 L VFS Cache 0 0 0 0 S VFS Cache 5865 5409 456 48 VNODEPOLL 0 0 0 0 VNODE 565 82 483 7 ata_composite 0 0 0 0 ata_request 63093 63093 0 76 g_bio 310921 310921 0 261 ACL UMA zone 0 0 0 0 mbuf_jumbo_16k 0 0 0 0 mbuf_jumbo_9k 0 0 0 0 mbuf_jumbo_pagesize 0 0 0 0 mbuf_cluster 384 0 384 6 mbuf 1725 1724 1 140 mbuf_packet 23439 23119 320 64 VMSPACE 1508 1478 30 22 UPCALL 0 0 0 0 KSEGRP 77 0 77 43 THREAD 77 0 77 13 PROC 1547 1478 69 8 Files 10873 10787 86 73 4096 5805 5704 101 18 2048 271 89 182 0 1024 3055 3008 47 145 512 4305 4270 35 13 256 4213 3909 304 26 128 22332 20502 1830 0 64 9997360 9992724 4636 9985362 32 4944 2141 2803 135 16 10069985 10067371 2614 228 mt_zone 158 0 158 78 DP fakepg 0 0 0 0 PV ENTRY 785101 775096 10005 1595 MAP ENTRY 59298 58673 625 159 KMAP ENTRY 170937 170911 26 142 MAP 7 0 7 33 VM OBJECT 29822 28947 875 82 128 Bucket 44 2 42 0 64 Bucket 32 1 31 11 32 Bucket 24 1 23 5 16 Bucket 28 0 28 22 UMA Hash 6 1 5 25 UMA RCntSlabs 195 0 195 27 UMA Slabs 340 41 299 55 UMA Zones 67 0 67 23 UMA Kegs 67 0 67 5 db> show malloc Type Allocs Frees Used MADT Table 0 0 0 acpipwr 0 0 0 acpi_perf 0 0 0 acpidev 93 0 93 acpisem 17 0 17 acpicmbat 0 0 0 PCI Link 64 0 64 acpitask 2 2 0 acpica 20052640 20049612 3028 KTRACE 100 0 100 prison 0 0 0 $PIR 0 0 0 DEVFS3 98 1 97 nexusdev 3 0 3 MP Table 0 0 0 memdesc 1 0 1 legacydrv 0 0 0 ithread 66 0 66 I/O APIC 1 0 1 zombie 1478 1478 0 proc-args 4944 4914 30 kqueue 52 52 0 kenv 114 1 113 atkbddev 2 0 2 sigio 1 0 1 file desc to leader 0 0 0 VM pgdata 2 0 2 file desc 1570 1500 70 DEVFS2 0 0 0 USBHC 0 0 0 cdev 21 0 21 USBdev 9 6 3 UMAHash 3 2 1 UFS mount 9 0 9 UFS quota 0 0 0 UFS dirhash 30 0 30 savedino 0 0 0 newdirblk 0 0 0 dirrem 0 0 0 mkdir 0 0 0 diradd 0 0 0 freefile 0 0 0 freeblks 0 0 0 freefrag 0 0 0 allocindir 0 0 0 indirdep 0 0 0 allocdirect 0 0 0 bmsafemap 0 0 0 newblk 1 0 1 inodedep 1 0 1 pagedep 1 0 1 rpcclnt 0 0 0 p1003.1b 1 0 1 agp 0 0 0 NFS daemon 5 0 5 NFSV3 srvdesc 0 0 0 NFS srvsock 1 0 1 nlminfo 0 0 0 NFS lock 0 0 0 NFS DirectIO 0 0 0 NFS hash 0 0 0 NFSV3 diroff 0 0 0 NFSV3 bigfh 0 0 0 NFS req 0 0 0 NFS srvsock 0 0 0 idmap 0 0 0 NFS4 dev 0 0 0 syncache 1 0 1 USB 31 0 31 hostcache 1 0 1 ip_moptions 0 0 0 Export Host 0 0 0 in_multi 3 0 3 igmp 0 0 0 routetbl 55 41 14 entropy 1024 0 1024 ata_dma 6 0 6 ad_driver 2 0 2 vlan 0 0 0 tun 0 0 0 lo 1 0 1 arpcom 2 0 2 clone 2 0 2 ether_multi 14 2 12 ifaddr 22 0 22 ifnet 4 0 4 BPF 3 0 3 ata_generic 3 0 3 Export Host 0 0 0 vnodemarker 4068 4068 0 mount 251 175 76 vnodes 1 0 1 VFS hash 1 0 1 Export Host 1 0 1 cluster_save buffer 0 0 0 vfscache 1 0 1 BIO buffer 56 9 47 isadev 18 0 18 pcb 57 35 22 soname 2881 2877 4 mbuf_tag 2 2 0 mbextcnt 0 0 0 accf 0 0 0 ptys 1 0 1 ttys 3070 1839 1231 shm 1 0 1 sem 4 0 4 msg 4 0 4 iov 581 581 0 select 0 0 0 ioctlops 1346 1346 0 Unitno 8 2 6 turnstiles 91 0 91 taskqueue 9 0 9 stack 0 0 0 MD sectors 0 0 0 MD disk 0 0 0 sleep queues 91 0 91 sbuf 246 246 0 rman 542 366 176 LED 0 0 0 kbdmux 6 0 6 acd_driver 1 0 1 kobj 134 19 115 eventhandler 44 0 44 devstat 8 0 8 mirror_data 0 0 0 bus-sc 1841 1759 82 bus 4342 3549 793 SWAP 2 0 2 umtx 90 0 90 sysctltmp 219 219 0 sysctloid 3189 0 3189 sysctl 557 557 0 uidinfo 141 136 5 plimit 1251 1237 14 ata_pci 0 0 0 cred 10711 10697 14 subproc 1626 1478 148 proc 2 0 2 session 227 203 24 pgrp 237 213 24 GEOM 429 331 98 mtx_pool 1 0 1 module 180 0 180 pfs_vncache 0 0 0 free 0 0 0 pfs_nodes 0 0 0 ip6ndp 0 0 0 ip6opt 0 0 0 temp 10112 10099 13 devbuf 1563 1 1562 cache 0 0 0 lockf 84 78 6 DEVFS 13 1 12 linker 54 24 30 DEVFS_RULE 0 0 0 DEVFS1 86 0 86 db> trace Tracing pid 1545 tid 100066 td 0xc6893d80 kdb_enter(c06e1497,c6523cf8,200820e,418500,c06ff738,...) at kdb_enter +0x30 siointr1(c6523c00,c6530100,c639a4c4,ffc0d61c,c069241c,...) at siointr1 +0xe1 siointr(c6523c00,c6893d80,ffc0d61c,0,c6893d80,...) at siointr+0x3f intr_execute_handlers(c639a4c4,ffc0d634,ffc0d678,c068e5e3,38,...) at intr_execute_handlers+0x10c lapic_handle_intr(38) at lapic_handle_intr+0x3a Xapic_isr1() at Xapic_isr1+0x33 --- interrupt, eip = 0xc0698f02, esp = 0xffc0d678, ebp = 0xffc0d678 --- is_dumpable(9095000,0,0,0,0,...) at is_dumpable+0x22 minidumpsys(c0723a00,c052f034,ffc0d730,c052fa08,c0745c34,...) at minidumpsys+0xde dumpsys(c0723a00,ffc0d7bc,c050a7b4,c06d0ba4,104,...) at dumpsys+0x1c doadump(c06d0ba4,104,c06d0c9d,c052aa40,ffc0d7a8,...) at doadump+0x4c boot(104,c0723900,c06ddf3b,ffc0d7f4,100,...) at boot+0x514 panic(c06ddf3b,1000,29b57000,ffc0d824,a,...) at panic+0x13d kmem_malloc(c104b0c0,1000,2,ffc0d87c,c063891b,...) at kmem_malloc+0xab page_alloc(c1055880,1000,ffc0d86f,2,ffc0d877,...) at page_alloc+0x27 slab_zalloc(c1055880,2,c1055880,1,0,...) at slab_zalloc+0x11b uma_zone_slab(c1055880,2,c1055880,1,c0704f60,...) at uma_zone_slab+0x106 uma_zalloc_bucket(c1055880,2,1,2f,4d,...) at uma_zalloc_bucket+0x17f uma_zalloc_arg(c1055880,0,2,c1056b40,c69a4780,...) at uma_zalloc_arg +0x319 malloc(80,c07029a0,2,4d,c6517400,...) at malloc+0x58 cblock_alloc_cblocks(4d,c6517400,c6517400,c6893d80,ffc0d978,...) at cblock_alloc_cblocks+0x5d clist_alloc_cblocks(c6517400,2000,2000,c6517400,c6517400,...) at clist_alloc_cblocks+0x77 ttsetwater(c6517400,8,c69a2b80,ffc0d9b4,c0548da3,...) at ttsetwater+0xa1 tty_open(c6897b00,c6517400,0,c0703040,ffc0d9bc,...) at tty_open+0x7d ptsopen(c6897b00,3,2000,c6893d80,0,...) at ptsopen+0x103 giant_open(c6897b00,3,2000,c6893d80,ffc0da18,...) at giant_open+0x71 devfs_open(ffc0da50,ffc0db0c,c058423d,c06f75e0,ffc0da50,...) at devfs_open+0x272 VOP_OPEN_APV(c06f75e0,ffc0da50,c071be20,c66ac000,0,...) at VOP_OPEN_APV+0x43 vn_open_cred(ffc0dbc0,ffc0dcc0,0,c6971580,6,...) at vn_open_cred+0x49d vn_open(ffc0dbc0,ffc0dcc0,0,6,ffc0db58,...) at vn_open+0x33 kern_open(c6893d80,bfbfe050,0,3,0,...) at kern_open+0xc8 open(c6893d80,ffc0dd04,c,c6893d80,8089000,...) at open+0x36 syscall(3b,3b,3b,8077274,4,...) at syscall+0x362 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (5, FreeBSD ELF32, open), eip = 0x28300747, esp = 0xbfbfe01c, ebp = 0xbfbfe078 --- -- Stefan Bethke Fon +49 170 346 0140