From owner-freebsd-stable@FreeBSD.ORG Fri Nov 20 00:45:30 2009 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 40B50106566C for ; Fri, 20 Nov 2009 00:45:30 +0000 (UTC) (envelope-from randy@psg.com) Received: from ran.psg.com (ran.psg.com [IPv6:2001:418:1::36]) by mx1.freebsd.org (Postfix) with ESMTP id 2252D8FC1E for ; Fri, 20 Nov 2009 00:45:30 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=rmac.psg.com) by ran.psg.com with esmtp (Exim 4.69 (FreeBSD)) (envelope-from ) id 1NBHcv-000316-3f; Fri, 20 Nov 2009 00:45:29 +0000 Received: from host-40-47.meeting.ietf.org.psg.com (localhost [127.0.0.1]) by rmac.psg.com (Postfix) with ESMTP id 945882C229DE; Fri, 20 Nov 2009 09:45:28 +0900 (JST) Date: Fri, 20 Nov 2009 09:45:28 +0900 Message-ID: From: Randy Bush To: FreeBSD-STABLE Mailing List References: User-Agent: Wanderlust/2.15.5 (Almost Unreal) Emacs/22.3 Mule/5.0 (SAKAKI) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII Subject: 7.2 dies in midnight run again X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Nov 2009 00:45:30 -0000 i think the issue is how to tune for zfs i386 with 4G of RAM RELENG_7 cvsupped Nov 18 02:42 GMT panic: kmem_malloc(65536): kmem_map too small: 535019520 total allocated cpuid = 0 Uptime: 13h15m1s Physical memory: 3958 MB Dumping 637 MB: 622 606 590 574 558 542 526 510 494 478 462 446 430 414 398 382 366 350 334 318 302 286 270 254 238 222 206 190 174 158 142 126 110 94 78 62 46 30 14 Dump complete Automatic reboot in 15 seconds - press a key on the console to abort and it did not auto reboot # cat /boot/loader.conf.local ipfw_load=YES umass_load=YES zfs_load=YES vm.kmem_size=536870912 vm.kmem_size_max=1073741824 vfs.zfs.prefetch_disable=1 it has zfs # zpool status pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror ONLINE 0 0 0 twed1 ONLINE 0 0 0 twed2 ONLINE 0 0 0 errors: No known data errors but boots and has root on ufs # df -H Filesystem Size Used Avail Capacity Mounted on /dev/twed0s1a 260M 199M 40M 83% / devfs 1.0k 1.0k 0B 100% /dev /dev/twed0s1h 65M 2.3M 57M 4% /root procfs 4.1k 4.1k 0B 100% /proc tank 147G 17M 147G 0% /tank tank/usr 167G 20G 147G 12% /usr tank/usr/home 216G 68G 147G 32% /usr/home tank/var 149G 2.3G 147G 2% /var tank/var/spool 148G 531M 147G 0% /var/spool /dev/md0 130M 12k 119M 0% /tmp # kgdb kernel.debug /usr/home/crash/vmcore.20 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... Unread portion of the kernel message buffer: panic: kmem_malloc(65536): kmem_map too small: 535019520 total allocated cpuid = 0 Uptime: 13h15m1s Physical memory: 3958 MB Dumping 637 MB: 622 606 590 574 558 542 526 510 494 478 462 446 430 414 398 382 366 350 334 318 302 286 270 254 238 222 206 190 174 158 142 126 110 94 78 62 46 30 14 Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/zfs.ko Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. done. Loaded symbols for /boot/kernel/opensolaris.ko Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from /boot/kernel/ipfw.ko.symbols...done. done. Loaded symbols for /boot/kernel/ipfw.ko Reading symbols from /boot/kernel/umass.ko...Reading symbols from /boot/kernel/umass.ko.symbols...done. done. Loaded symbols for /boot/kernel/umass.ko Reading symbols from /boot/kernel/cam.ko...Reading symbols from /boot/kernel/cam.ko.symbols...done. done. Loaded symbols for /boot/kernel/cam.ko Reading symbols from /boot/kernel/usb.ko...Reading symbols from /boot/kernel/usb.ko.symbols...done. done. Loaded symbols for /boot/kernel/usb.ko #0 doadump () at pcpu.h:196 196 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); (kgdb) back #0 doadump () at pcpu.h:196 #1 0xc052b0b6 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418 #2 0xc052b39e in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:574 #3 0xc06dfb54 in kmem_malloc (map=0xc107108c, size=65536, flags=2) at /usr/src/sys/vm/vm_kern.c:305 #4 0xc06d6317 in page_alloc (zone=0x0, bytes=65536, pflag=0xf67ee4a7 "\002", wait=2) at /usr/src/sys/vm/uma_core.c:952 #5 0xc06d8e20 in uma_large_malloc (size=65536, wait=2) at /usr/src/sys/vm/uma_core.c:2706 #6 0xc05189e8 in malloc (size=65536, mtp=0xc0989060, flags=2) at /usr/src/sys/kern/kern_malloc.c:393 #7 0xc0897a61 in zfs_kmem_alloc (size=65536, kmflags=2) at /usr/src/sys/modules/zfs/../../cddl/compat/opensolaris/kern/opensolaris_kmem.c:74 #8 0xc090bf4a in zio_buf_alloc (size=65536) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:207 #9 0xc08f3472 in vdev_cache_read (zio=0xd39d0708) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_cache.c:188 #10 0xc090c145 in zio_vdev_io_start (zio=0xd39d0708) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1816 #11 0xc090c7f0 in zio_execute (zio=0xd39d0708) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:998 #12 0xc08f6bda in vdev_mirror_io_start (zio=0xd77e7708) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:303 #13 0xc090c7f0 in zio_execute (zio=0xd77e7708) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:998 #14 0xc08f6bda in vdev_mirror_io_start (zio=0xdbd22960) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:303 #15 0xc090c7f0 in zio_execute (zio=0xdbd22960) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:998 #16 0xc08ad39d in arc_read_nolock (pio=0xd1844960, spa=0xc7746000, bp=0xcc548640, done=0xc08b0600 , private=0xd9c89e38, priority=0, zio_flags=1, arc_flags=0xf67ee854, zb=0xf67ee834) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2762 #17 0xc08ad878 in arc_read (pio=0xd1844960, spa=0xc7746000, bp=0xcc548640, pbuf=0xc9b29134, done=0xc08b0600 , private=0xd9c89e38, priority=0, zio_flags=1, arc_flags=0xf67ee854, zb=0xf67ee834) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2507 #18 0xc08b0ada in dbuf_read (db=0xd9c89e38, zio=0xd1844960, flags=14) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:521 #19 0xc08b1142 in dbuf_findbp (dn=0xcba92000, level=Variable "level" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1381 #20 0xc08b1269 in dbuf_hold_impl (dn=0xcba92000, level=0 '\0', blkid=0, fail_sparse=0, tag=0x0, dbp=0xf67ee8f0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1617 #21 0xc08b2529 in dbuf_hold (dn=0xcba92000, blkid=0, tag=0x0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1689 #22 0xc08b48cc in dmu_buf_hold (os=0xc774b3d0, object=167123, offset=0, tag=0x0, dbp=0xf67ee95c) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:101 #23 0xc0900044 in zap_lockdir (os=0xc774b3d0, obj=167123, tx=0x0, lti=RW_READER, fatreader=1, adding=0, zapp=0xf67eeba0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:388 #24 0xc09009fd in zap_cursor_retrieve (zc=0xf67eeb9c, za=0xf67eea84) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:1004 #25 0xc0925a2b in zfs_freebsd_readdir (ap=0xf67eec00) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:2156 #26 0xc07382f2 in VOP_READDIR_APV (vop=0xc098b560, a=0xf67eec00) at vnode_if.c:1407 #27 0xc05becae in kern_getdirentries (td=0xd5a01000, fd=19, buf=0x29061000
, count=4096, basep=0xf67eec74) at vnode_if.h:747 #28 0xc05beec1 in getdirentries (td=0xd5a01000, uap=0xf67eecfc) at /usr/src/sys/kern/vfs_syscalls.c:3776 #29 0xc072cfc5 in syscall (frame=0xf67eed38) at /usr/src/sys/i386/i386/trap.c:1101 #30 0xc0711380 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:262 #31 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) randy