Date: Thu, 4 Jun 2009 09:48:15 -0500 (CDT) From: Tim Chase <tim@chase2k.com> To: freebsd-stable@freebsd.org Subject: kmem map too small panic after updating to STABLE-7 r192996 Message-ID: <alpine.LFD.2.00.0906040932280.27799@localhost.localdomain>
next in thread | raw e-mail | index | archive | help
Hello,
I decided to give the new zfs code a try and upgraded my stable-7 system
and discovered a panic that I can reproduce at will.
This is an i386 system with 1.5GB of RAM and a single zfs pool:
appbuild# zpool status
pool: backup
state: ONLINE
status: The pool is formatted using an older on-disk format. The pool can
still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'. Once this is done, the
pool will no longer be accessible on older software versions.
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
backup ONLINE 0 0 0
mirror ONLINE 0 0 0
ad4 ONLINE 0 0 0
ad6 ONLINE 0 0 0
errors: No known data errors
I had previously used the following zfs tuning:
vm.kmem_size="512M"
vm.kmem_size_max="512M"
vfs.zfs.arc_max="100M"
After getting this panic, I removed these tunables. The panic happens
in either case.
Reproducing the panic is easy: I keep a copy of the FreeBSD svn tree
in a compressed zfs file system (compression ratio runs around 1.35x).
An "svn update" (or the requisite "svn cleanup" following a system crash)
in my checkout area will _always_ result in a "kmem map too small" panic.
Here's a stack trace from my most recent panic:
(kgdb) bt
#0 doadump () at pcpu.h:196
#1 0xc052c963 in boot (howto=260) at /root/stable-7/sys/kern/kern_shutdown.c:418
#2 0xc052cb9d in panic (fmt=Variable "fmt" is not available.
) at /root/stable-7/sys/kern/kern_shutdown.c:574
#3 0xc069fa2a in kmem_malloc (map=0xc105408c, size=131072, flags=2) at /root/stable-7/sys/vm/vm_kern.c:305
#4 0xc0696587 in page_alloc (zone=0x0, bytes=131072, pflag=0xe6dbcb47 "\002", wait=2)
at /root/stable-7/sys/vm/uma_core.c:952
#5 0xc0699000 in uma_large_malloc (size=131072, wait=2) at /root/stable-7/sys/vm/uma_core.c:2706
#6 0xc051af38 in malloc (size=131072, mtp=0xc094f060, flags=2) at /root/stable-7/sys/kern/kern_malloc.c:393
#7 0xc085db00 in zfs_kmem_alloc (size=131072, kmflags=2)
at /root/stable-7/sys/modules/zfs/../../cddl/compat/opensolaris/kern/opensolaris_kmem.c:74
#8 0xc08d1c79 in zio_buf_alloc (size=131072)
at /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:207
#9 0xc08bcc40 in vdev_queue_io_to_issue (vq=0xc492b334, pending_limit=Variable "pending_limit" is not available.
)
at /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:227
#10 0xc08bce42 in vdev_queue_io_done (zio=0xd420a000)
at /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:313
#11 0xc08d4162 in zio_vdev_io_done (zio=0xd420a000)
at /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1847
#12 0xc08d2532 in zio_execute (zio=0xd420a000)
at /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:998
#13 0xc0862aa0 in taskq_thread (arg=0xc44a10cc)
at /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/os/taskq.c:854
#14 0xc0506816 in fork_exit (callout=0xc0862960 <taskq_thread>, arg=0xc44a10cc, frame=0xe6dbcd38)
at /root/stable-7/sys/kern/kern_fork.c:811
#15 0xc06d4670 in fork_trampoline () at /root/stable-7/sys/i386/i386/exception.s:264
(kgdb) print panicstr
$1 = 0xc0792320 "kmem_malloc(131072): kmem_map too small: 325656576 total allocated"
The arc size is now auto-tuned as:
kstat.zfs.misc.arcstats.c_min: 26214400
kstat.zfs.misc.arcstats.c_max: 209715200
and while monitoring kstat.zfs.misc.arcstats.size I noticed it fluctuated
between around 140 and 150 million. Interestingly enough, immediately
before the last panic, It (arcstats.size) had just been reduced from
150038268 to 128790020.
I'm going to continue to poke around in my core dumps and see if I can
figure out what's going on. Any hints as to what to look for or monitor
while the system is running would be appreciated.
Thanks,
Tim
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.LFD.2.00.0906040932280.27799>