From owner-freebsd-current@FreeBSD.ORG Wed May 12 07:25:06 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5F2BE16A4CF for ; Wed, 12 May 2004 07:25:06 -0700 (PDT) Received: from melusine.cuivre.fr.eu.org (melusine.cuivre.fr.eu.org [82.225.155.84]) by mx1.FreeBSD.org (Postfix) with ESMTP id BFE4743D46 for ; Wed, 12 May 2004 07:25:04 -0700 (PDT) (envelope-from thomas@FreeBSD.ORG) Received: by melusine.cuivre.fr.eu.org (Postfix, from userid 1000) id 47CBC2C41F; Wed, 12 May 2004 16:25:41 +0200 (CEST) Date: Wed, 12 May 2004 16:25:41 +0200 From: Thomas Quinot To: freebsd-current@freebsd.org Message-ID: <20040512142541.GA65967@melusine.cuivre.fr.eu.org> References: <1079446098.23554.49.camel@lanshark.dmv.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1079446098.23554.49.camel@lanshark.dmv.com> X-message-flag: WARNING! Using Outlook can damage your computer. User-Agent: Mutt/1.5.6i Subject: Re: kmem_map too small, revisited (5.2.1-REL-p1) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 May 2004 14:25:06 -0000 * Sven Willenberger, 2004-03-16 : > The Issue: Ever since having migrated from 4.x to 5.x I have been having > an issue (on 2 different hardware setups) with spontaneous reboots or > system hangs, usually identified by the kmem_map too small message. The I just got a similar reboot on a 5.2.1-REL-p1 box (ie with the cred leak fix and the SA 04:04 TCP reassembly queue fix, but with rev. 1.217.2.2 of tcp_input.c). Could that explain the problem? panic: kmem_malloc(16384): kmem_map too small: 275247104 total allocated panic: from debugger Uptime: 61d14h24m27s Dumping 2047 MB Here is an excerpt of the backtrace: #9 0xc06c8b88 in calltrap () at {standard input}:94 #10 0xc0568a55 in panic ( fmt=0xc07324a1 "kmem_malloc(%ld): kmem_map too small: %ld total allocated") at /usr/src/RELENG_5_2/sys/kern/kern_shutdown.c:534 #11 0xc0691370 in kmem_malloc (map=0xc10310a0, size=16384, flags=1026) at /usr/src/RELENG_5_2/sys/vm/vm_kern.c:342 #12 0xc06a2777 in page_alloc (zone=0x0, bytes=0, pflag=0x0, wait=0) at /usr/src/RELENG_5_2/sys/vm/uma_core.c:842 #13 0xc06a4302 in uma_large_malloc (size=16384, wait=1026) at /usr/src/RELENG_5_2/sys/vm/uma_core.c:2024 #14 0xc055d52b in malloc (size=16384, type=0xc0769100, flags=1026) at /usr/src/RELENG_5_2/sys/kern/kern_malloc.c:253 #15 0xc0671f55 in softdep_disk_io_initiation (bp=0xd4a3ff10) at /usr/src/RELENG_5_2/sys/ufs/ffs/ffs_softdep.c:3493 #16 0xc0523bd7 in spec_xstrategy (vp=0xc87db618, bp=0xd4a3ff10) at /usr/src/RELENG_5_2/sys/sys/buf.h:413 #17 0xc0523d72 in spec_specstrategy (ap=0xe66b1b68) at /usr/src/RELENG_5_2/sys/fs/specfs/spec_vnops.c:534 #18 0xc0522eb8 in spec_vnoperate (ap=0x0) at /usr/src/RELENG_5_2/sys/fs/specfs/spec_vnops.c:122 #19 0xc06874fc in ufs_strategy (ap=0x0) at vnode_if.h:1141 #20 0xc0688268 in ufs_vnoperate (ap=0x0) at /usr/src/RELENG_5_2/sys/ufs/ufs/ufs_vnops.c:2793 #21 0xc05ae5bd in bwrite (bp=0xd4a3ff10) at vnode_if.h:1116 #22 0xc05b03c2 in vfs_bio_awrite (bp=0xd4a3ff10) at /usr/src/RELENG_5_2/sys/kern/vfs_bio.c:1715 #23 0xc0679b8c in ffs_fsync (ap=0xe66b1c94) at /usr/src/RELENG_5_2/sys/ufs/ffs/ffs_vnops.c:268 #24 0xc0678d04 in ffs_sync (mp=0xc82b0000, waitfor=2, cred=0xd40a4600, ---Type to continue, or q to quit--- td=0xd39cd500) at vnode_if.h:627 #25 0xc05c419e in sync (td=0xd39cd500, uap=0xe66b1d14) at /usr/src/RELENG_5_2/sys/kern/vfs_syscalls.c:141 #26 0xc06d7d80 in syscall (frame= {tf_fs = -1078001617, tf_es = -1078001617, tf_ds = -1078001617, tf_edi = 0, tf_esi = 136427152, tf_ebp = -1077943608, tf_isp = -429187724, tf_ebx = 0, tf_edx = 0, tf_ecx = 805742720, tf_eax = 36, tf_trapno = 12, tf_err = 2, tf_eip = 675618159, tf_cs = 31, tf_eflags = 531, tf_esp = -1077943620, tf_ss = 47}) at /usr/src/RELENG_5_2/sys/i386/i386/trap.c:1010 #27 0xc06c8bdd in Xint0x80_syscall () at {standard input}:136 ---Can't read userspace from dump, or kernel process--- By walking the kmemstatistics data in the crashdump, I was able to reconstruct kmem usage information (only the biggest figures are listed here): Type MemUse devbuf 1081216 vfscache 1048576 inodedep 998272 lockf 798080 subproc 764928 SWAP 561216 kobj 239616 ttys 213120 temp 164720 linker 162032 mbufmgr 142608 diradd 118176 acpica 98016 allocdirect 93952 kqueue 83200 pagedep 81408 BPF 74688 entropy 65536 pfs_fileno 40960 mount 34752 I am keeping the crash dump at hand, please let me know if there is any other piece of data I can provide to help identify the origin of the problem. Thanks! Thomas. -- Thomas.Quinot@Cuivre.FR.EU.ORG