From owner-freebsd-stable@FreeBSD.ORG Thu Oct 26 13:29:05 2006 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1041A16A560; Thu, 26 Oct 2006 13:29:05 +0000 (UTC) (envelope-from stb@lassitu.de) Received: from koef.zs64.net (koef.zs64.net [212.12.50.230]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4B59443D55; Thu, 26 Oct 2006 13:29:03 +0000 (GMT) (envelope-from stb@lassitu.de) Received: (from stb@koef.zs64.net) (authenticated) by koef.zs64.net (8.13.8/8.13.8) with ESMTP id k9QDSsQD032832 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Thu, 26 Oct 2006 15:28:54 +0200 (CEST) (envelope-from stb@lassitu.de) In-Reply-To: <20152564-EC53-4210-9590-23817A6A3E42@lassitu.de> References: <20061025183308.L33725@fledge.watson.org> <838FCA83-20F8-4A09-A025-E69956032F86@lassitu.de> <45400286.9020402@samsco.org> <20061026091253.J69980@fledge.watson.org> <20061026111723.K33725@fledge.watson.org> <20152564-EC53-4210-9590-23817A6A3E42@lassitu.de> Mime-Version: 1.0 (Apple Message framework v752.2) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Stefan Bethke Date: Thu, 26 Oct 2006 15:30:17 +0200 To: Robert Watson X-Mailer: Apple Mail (2.752.2) Cc: Andreas Sons , FreeBSD Stable Subject: Re: panic: kmem_map too small X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Oct 2006 13:29:05 -0000 Am 26.10.2006 um 12:36 schrieb Stefan Bethke: > Am 26.10.2006 um 12:20 schrieb Robert Watson: > >> After a bit of looking at the output, etc, I agree with your >> conclusion that what's there now is lacking. The attached patch, >> committed to -CURRENT but not yet to -STABLE, makes the "show >> malloc" DDB output a bit more like the "vmstat -m" output, in that >> it summarizes the allocation counts and adds the memory use >> information. Sample output: > > I patched up the box; here's the output right after rebooting into > the new kernel. Once it panics again, I'll post the results. > > db> show malloc > Type InUse MemUse Requests > MADT Table 0 0K 0 > acpipwr 0 0K 0 > acpi_perf 0 0K 0 > acpidev 93 3K 93 > acpisem 17 2K 17 > acpicmbat 0 0K 0 > PCI Link 64 6K 64 > acpitask 0 0K 2 > acpica 3000 158K 42603 > KTRACE 100 13K 100 > prison 0 0K 0 > $PIR 0 0K 0 > DEVFS3 95 12K 96 > nexusdev 3 1K 3 > MP Table 0 0K 0 > memdesc 1 4K 1 > legacydrv 0 0K 0 > ithread 66 6K 66 > I/O APIC 1 1K 1 > zombie 0 0K 649 > proc-args 28 2K 345 > kqueue 0 0K 30 > kenv 113 8K 114 > atkbddev 2 1K 2 > sigio 1 1K 1 > file desc to leader 0 0K 0 > VM pgdata 2 65K 2 > file desc 68 17K 717 > DEVFS2 0 0K 0 > USBHC 0 0K 0 > cdev 19 3K 19 > USBdev 3 1K 9 > UMAHash 1 1K 3 > UFS mount 9 19K 9 > UFS quota 0 0K 0 > UFS dirhash 27 5K 27 > savedino 0 0K 0 > newdirblk 0 0K 0 > dirrem 0 0K 0 > mkdir 0 0K 0 > diradd 0 0K 0 > freefile 0 0K 0 > freeblks 0 0K 0 > freefrag 0 0K 0 > allocindir 0 0K 0 > indirdep 0 0K 0 > allocdirect 0 0K 0 > bmsafemap 0 0K 0 > newblk 1 1K 1 > inodedep 1 256K 1 > pagedep 1 64K 1 > rpcclnt 0 0K 0 > p1003.1b 1 1K 1 > agp 0 0K 0 > NFS daemon 5 10K 5 > NFSV3 srvdesc 0 0K 0 > NFS srvsock 1 1K 1 > nlminfo 0 0K 0 > NFS lock 0 0K 0 > NFS DirectIO 0 0K 0 > NFS hash 0 0K 0 > NFSV3 diroff 0 0K 0 > NFSV3 bigfh 0 0K 0 > NFS req 0 0K 0 > NFS srvsock 0 0K 0 > idmap 0 0K 0 > NFS4 dev 0 0K 0 > syncache 1 8K 1 > USB 31 3K 31 > hostcache 1 24K 1 > ip_moptions 0 0K 0 > Export Host 0 0K 0 > in_multi 3 1K 3 > igmp 0 0K 0 > routetbl 14 2K 55 > entropy 1024 64K 1024 > ata_dma 6 1K 6 > ad_driver 2 1K 2 > vlan 0 0K 0 > tun 0 0K 0 > lo 1 1K 1 > arpcom 2 1K 2 > clone 2 8K 2 > ether_multi 12 1K 14 > ifaddr 22 5K 22 > ifnet 4 4K 4 > BPF 3 1K 3 > ata_generic 3 3K 3 > Export Host 0 0K 0 > vnodemarker 0 0K 74 > mount 76 3K 225 > vnodes 1 1K 1 > VFS hash 1 256K 1 > Export Host 1 1K 1 > cluster_save buffer 0 0K 0 > vfscache 1 512K 1 > BIO buffer 41 82K 45 > isadev 18 2K 18 > pcb 22 5K 43 > soname 4 1K 431 > mbuf_tag 0 0K 2 > mbextcnt 0 0K 0 > accf 0 0K 0 > ptys 0 0K 0 > ttys 1072 152K 2543 > shm 1 12K 1 > sem 4 7K 4 > msg 4 25K 4 > iov 0 0K 292 > select 0 0K 0 > ioctlops 0 0K 465 > Unitno 6 1K 8 > turnstiles 91 6K 91 > taskqueue 9 1K 9 > stack 0 0K 0 > MD sectors 0 0K 0 > MD disk 0 0K 0 > sleep queues 91 3K 91 > sbuf 0 0K 246 > rman 176 11K 542 > LED 0 0K 0 > kbdmux 6 9K 6 > acd_driver 1 2K 1 > kobj 115 230K 134 > eventhandler 44 3K 44 > devstat 8 17K 8 > mirror_data 0 0K 0 > bus-sc 82 32K 1841 > bus 793 38K 4342 > SWAP 2 549K 2 > umtx 90 6K 90 > sysctltmp 0 0K 197 > sysctloid 3189 97K 3189 > sysctl 0 0K 151 > uidinfo 4 2K 7 > plimit 13 4K 149 > ata_pci 0 0K 0 > cred 11 2K 1266 > subproc 146 292K 795 > proc 2 8K 2 > session 23 3K 26 > pgrp 23 2K 26 > GEOM 98 12K 429 > mtx_pool 1 8K 1 > module 180 12K 180 > pfs_vncache 0 0K 0 > free 0 0K 0 > pfs_nodes 0 0K 0 > ip6ndp 0 0K 0 > ip6opt 0 0K 0 > temp 13 211K 4630 > devbuf 1562 3592K 1563 > cache 0 0K 0 > lockf 6 1K 42 > DEVFS 12 1K 13 > linker 30 2K 54 > DEVFS_RULE 0 0K 0 > DEVFS1 84 21K 84 Panic a few minutes ago: login: -0255: *** Error: UtCallocate: Could not allocate size 30 -0255: *** Error: UtCallocate: Could not allocate size 30 -0255: *** Error: UtCallocate: Could not allocate size 30 ACPI-1304: *** Error: Method execution failed [\SWFS] (Node 0xc63ee220), AE_NO_MEMORY ACPI-1304: *** Error: Method execution failed [\RBYT] (Node 0xc63ee1a0), AE_NO_MEMORY ACPI-1304: *** Error: Method execution failed [\RTMP] (Node 0xc63ee380), AE_NO_MEMORY ACPI-1304: *** Error: Method execution failed [\_TZ_.THRM._TMP] (Node 0xc63ede00), AE_NO_MEMORY panic: kmem_malloc(16384): kmem_map too small: 699756544 total allocated KDB: enter: panic [thread pid 1386 tid 100059 ] Stopped at kdb_enter+0x30: leave db> Uptime: 2h43m42s KDB: enter: Break sequence on console [thread pid 1386 tid 100059 ] Stopped at kdb_enter+0x30: leave db> show malloc Type InUse MemUse Requests MADT Table 0 0K 0 acpipwr 0 0K 0 acpi_perf 0 0K 0 acpidev 93 3K 93 acpisem 17 2K 17 acpicmbat 0 0K 0 PCI Link 64 6K 64 acpitask 0 0K 2 acpica 3024 159K 20026966 KTRACE 100 13K 100 prison 0 0K 0 $PIR 0 0K 0 DEVFS3 97 13K 98 nexusdev 3 1K 3 MP Table 0 0K 0 memdesc 1 4K 1 legacydrv 0 0K 0 ithread 66 6K 66 I/O APIC 1 1K 1 zombie 0 0K 1312 proc-args 36 2K 3017 kqueue 0 0K 68 kenv 113 8K 114 atkbddev 2 1K 2 sigio 1 1K 1 file desc to leader 0 0K 0 VM pgdata 2 65K 2 file desc 76 19K 1402 DEVFS2 0 0K 0 USBHC 0 0K 0 cdev 21 3K 21 USBdev 3 1K 9 UMAHash 1 1K 3 UFS mount 9 19K 9 UFS quota 0 0K 0 UFS dirhash 30 6K 30 savedino 0 0K 0 newdirblk 0 0K 0 dirrem 0 0K 0 mkdir 0 0K 0 diradd 0 0K 0 freefile 0 0K 0 freeblks 0 0K 0 freefrag 0 0K 0 allocindir 0 0K 0 indirdep 0 0K 0 allocdirect 0 0K 0 bmsafemap 0 0K 0 newblk 1 1K 1 inodedep 1 256K 1 pagedep 1 64K 1 rpcclnt 0 0K 0 p1003.1b 1 1K 1 agp 0 0K 0 NFS daemon 5 10K 5 NFSV3 srvdesc 0 0K 0 NFS srvsock 1 1K 1 nlminfo 0 0K 0 NFS lock 0 0K 0 NFS DirectIO 0 0K 0 NFS hash 0 0K 0 NFSV3 diroff 0 0K 0 NFSV3 bigfh 0 0K 0 NFS req 0 0K 0 NFS srvsock 0 0K 0 idmap 0 0K 0 NFS4 dev 0 0K 0 syncache 1 8K 1 USB 31 3K 31 hostcache 1 24K 1 ip_moptions 0 0K 0 Export Host 0 0K 0 in_multi 3 1K 3 igmp 0 0K 0 routetbl 14 2K 61 entropy 1024 64K 1024 ata_dma 6 1K 6 ad_driver 2 1K 2 vlan 0 0K 0 tun 0 0K 0 lo 1 1K 1 arpcom 2 1K 2 clone 2 8K 2 ether_multi 12 1K 14 ifaddr 22 5K 22 ifnet 4 4K 4 BPF 3 1K 3 ata_generic 3 3K 3 Export Host 0 0K 0 vnodemarker 0 0K 2158 mount 76 3K 225 vnodes 1 1K 1 VFS hash 1 256K 1 Export Host 1 1K 1 cluster_save buffer 0 0K 3 vfscache 1 512K 1 BIO buffer 59 118K 63 isadev 18 2K 18 pcb 21 5K 66 soname 4 1K 1866 mbuf_tag 0 0K 2 mbextcnt 0 0K 0 accf 0 0K 0 ptys 1 1K 1 ttys 1073 153K 2859 shm 1 12K 1 sem 4 7K 4 msg 4 25K 4 iov 0 0K 415 select 0 0K 0 ioctlops 0 0K 1188 Unitno 6 1K 8 turnstiles 121 8K 121 taskqueue 9 1K 9 stack 0 0K 0 MD sectors 0 0K 0 MD disk 0 0K 0 sleep queues 121 4K 121 sbuf 0 0K 248 rman 176 11K 542 LED 0 0K 0 kbdmux 6 9K 6 acd_driver 1 2K 1 kobj 115 230K 134 eventhandler 44 3K 44 devstat 8 17K 8 mirror_data 0 0K 0 bus-sc 82 32K 1841 bus 793 38K 4342 SWAP 2 549K 2 umtx 120 8K 120 sysctltmp 0 0K 414 sysctloid 3189 97K 3189 sysctl 0 0K 563 uidinfo 5 2K 84 plimit 15 4K 1085 ata_pci 0 0K 0 cred 14 2K 7927 subproc 175 329K 1487 proc 2 8K 2 session 25 4K 169 pgrp 25 2K 182 GEOM 98 12K 429 mtx_pool 1 8K 1 module 180 12K 180 pfs_vncache 0 0K 0 free 0 0K 0 pfs_nodes 0 0K 0 ip6ndp 0 0K 0 ip6opt 0 0K 0 temp 13 211K 7942 devbuf 1562 3592K 1563 cache 0 0K 0 lockf 6 1K 138 DEVFS 12 1K 13 linker 30 2K 54 DEVFS_RULE 0 0K 0 DEVFS1 86 22K 86 db> show uma Zone Allocs Frees Used Cache FFS2 dinode 746 84 662 43 FFS1 dinode 0 0 0 0 FFS inode 746 84 662 63 Mountpoints 4 0 4 8 SWAPMETA 0 0 0 0 rtentry 6 1 5 53 unpcb 492 482 10 46 ripcb 0 0 0 0 sackhole 0 0 0 0 tcpreass 0 0 0 0 hostcache 4 2 2 98 syncache 59 59 0 78 tcptw 2 2 0 156 tcpcb 71 62 9 15 inpcb 71 62 9 35 udpcb 253 241 12 32 ipq 0 0 0 0 socket 817 786 31 13 KNOTE 72 72 0 112 PIPE 584 580 4 14 NFSNODE 0 0 0 0 NFSMOUNT 0 0 0 0 DIRHASH 29 0 29 7 NAMEI 26944 26944 0 12 L VFS Cache 8 8 0 26 S VFS Cache 4903 4217 686 98 VNODEPOLL 0 0 0 0 VNODE 782 85 697 31 ata_composite 0 0 0 0 ata_request 3378 3377 1 56 g_bio 13513 13509 4 170 ACL UMA zone 0 0 0 0 mbuf_jumbo_16k 0 0 0 0 mbuf_jumbo_9k 0 0 0 0 mbuf_jumbo_pagesize 0 0 0 0 mbuf_cluster 384 0 384 6 mbuf 3835 3827 8 133 mbuf_packet 15060 14740 320 64 VMSPACE 1348 1312 36 16 UPCALL 0 0 0 0 KSEGRP 98 0 98 62 THREAD 98 0 98 22 PROC 1387 1312 75 23 Files 9078 8992 86 73 4096 5448 5341 107 12 2048 274 80 194 4 1024 2238 2191 47 145 512 2395 2360 35 13 256 3829 3497 332 28 128 18254 16580 1674 246 64 9990754 9986054 4700 9980755 32 4837 2006 2831 107 16 10047751 10045138 2613 229 mt_zone 158 0 158 78 DP fakepg 0 0 0 0 PV ENTRY 702520 691839 10681 4109 MAP ENTRY 55134 54502 632 320 KMAP ENTRY 170993 170967 26 142 MAP 7 0 7 33 VM OBJECT 25648 24666 982 207 128 Bucket 44 2 42 0 64 Bucket 34 1 33 9 32 Bucket 25 0 25 3 16 Bucket 30 0 30 20 UMA Hash 6 1 5 25 UMA RCntSlabs 195 0 195 27 UMA Slabs 376 68 308 46 UMA Zones 67 0 67 23 UMA Kegs 67 0 67 5 db> -- Stefan Bethke Fon +49 170 346 0140