Date: Thu, 4 Jun 2009 14:13:44 -0700 From: Kip Macy <kmacy@freebsd.org> To: Tim Chase <tim@chase2k.com> Cc: freebsd-stable@freebsd.org Subject: Re: kmem map too small panic after updating to STABLE-7 r192996 Message-ID: <3c1674c90906041413n15e2ce88q9f279818641515e7@mail.gmail.com> In-Reply-To: <alpine.LFD.2.00.0906040932280.27799@localhost.localdomain> References: <alpine.LFD.2.00.0906040932280.27799@localhost.localdomain>
next in thread | previous in thread | raw e-mail | index | archive | help
As I mentioned in the initial e-mail, auto-tuning is only safe to rely on on amd64. -Kip On Thu, Jun 4, 2009 at 7:48 AM, Tim Chase <tim@chase2k.com> wrote: > Hello, > > I decided to give the new zfs code a try and upgraded my stable-7 system > and discovered a panic that I can reproduce at will. > > This is an i386 system with 1.5GB of RAM and a single zfs pool: > > =A0 =A0 =A0 =A0appbuild# zpool status > =A0 =A0 =A0 =A0 =A0pool: backup > =A0 =A0 =A0 =A0 state: ONLINE > =A0 =A0 =A0 =A0status: The pool is formatted using an older on-disk forma= t. =A0The > pool can > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0still be used, but some features are unava= ilable. > =A0 =A0 =A0 =A0action: Upgrade the pool using 'zpool upgrade'. =A0Once th= is is done, > the > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0pool will no longer be accessible on older= software versions. > =A0 =A0 =A0 =A0 scrub: none requested > =A0 =A0 =A0 =A0config: > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0NAME =A0 =A0 =A0 =A0STATE =A0 =A0 READ WRI= TE CKSUM > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0backup =A0 =A0 =A0ONLINE =A0 =A0 =A0 0 =A0= =A0 0 =A0 =A0 0 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0mirror =A0 =A0ONLINE =A0 =A0 =A0 0 =A0= =A0 0 =A0 =A0 0 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0ad4 =A0 =A0 ONLINE =A0 =A0 =A0 0 = =A0 =A0 0 =A0 =A0 0 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0ad6 =A0 =A0 ONLINE =A0 =A0 =A0 0 = =A0 =A0 0 =A0 =A0 0 > > =A0 =A0 =A0 =A0errors: No known data errors > > I had previously used the following zfs tuning: > > =A0 =A0 =A0 =A0vm.kmem_size=3D"512M" > =A0 =A0 =A0 =A0vm.kmem_size_max=3D"512M" > =A0 =A0 =A0 =A0vfs.zfs.arc_max=3D"100M" > > After getting this panic, I removed these tunables. =A0The panic happens > in either case. > > Reproducing the panic is easy: I keep a copy of the FreeBSD svn tree > in a compressed zfs file system (compression ratio runs around 1.35x). > An "svn update" (or the requisite "svn cleanup" following a system crash) > in my checkout area will _always_ result in a "kmem map too small" panic. > Here's a stack trace from my most recent panic: > > (kgdb) bt > #0 =A0doadump () at pcpu.h:196 > #1 =A00xc052c963 in boot (howto=3D260) at > /root/stable-7/sys/kern/kern_shutdown.c:418 > #2 =A00xc052cb9d in panic (fmt=3DVariable "fmt" is not available. > ) at /root/stable-7/sys/kern/kern_shutdown.c:574 > #3 =A00xc069fa2a in kmem_malloc (map=3D0xc105408c, size=3D131072, flags= =3D2) at > /root/stable-7/sys/vm/vm_kern.c:305 > #4 =A00xc0696587 in page_alloc (zone=3D0x0, bytes=3D131072, pflag=3D0xe6d= bcb47 > "\002", wait=3D2) > =A0 =A0at /root/stable-7/sys/vm/uma_core.c:952 > #5 =A00xc0699000 in uma_large_malloc (size=3D131072, wait=3D2) at > /root/stable-7/sys/vm/uma_core.c:2706 > #6 =A00xc051af38 in malloc (size=3D131072, mtp=3D0xc094f060, flags=3D2) a= t > /root/stable-7/sys/kern/kern_malloc.c:393 > #7 =A00xc085db00 in zfs_kmem_alloc (size=3D131072, kmflags=3D2) > =A0 =A0at > /root/stable-7/sys/modules/zfs/../../cddl/compat/opensolaris/kern/opensol= aris_kmem.c:74 > #8 =A00xc08d1c79 in zio_buf_alloc (size=3D131072) > =A0 =A0at > /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/= fs/zfs/zio.c:207 > #9 =A00xc08bcc40 in vdev_queue_io_to_issue (vq=3D0xc492b334, > pending_limit=3DVariable "pending_limit" is not available. > ) > =A0 =A0at > /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/= fs/zfs/vdev_queue.c:227 > #10 0xc08bce42 in vdev_queue_io_done (zio=3D0xd420a000) > =A0 =A0at > /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/= fs/zfs/vdev_queue.c:313 > #11 0xc08d4162 in zio_vdev_io_done (zio=3D0xd420a000) > =A0 =A0at > /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/= fs/zfs/zio.c:1847 > #12 0xc08d2532 in zio_execute (zio=3D0xd420a000) > =A0 =A0at > /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/= fs/zfs/zio.c:998 > #13 0xc0862aa0 in taskq_thread (arg=3D0xc44a10cc) > =A0 =A0at > /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/= os/taskq.c:854 > #14 0xc0506816 in fork_exit (callout=3D0xc0862960 <taskq_thread>, > arg=3D0xc44a10cc, frame=3D0xe6dbcd38) > =A0 =A0at /root/stable-7/sys/kern/kern_fork.c:811 > #15 0xc06d4670 in fork_trampoline () at > /root/stable-7/sys/i386/i386/exception.s:264 > (kgdb) print panicstr > $1 =3D 0xc0792320 "kmem_malloc(131072): kmem_map too small: 325656576 tot= al > allocated" > > > The arc size is now auto-tuned as: > > =A0 =A0 =A0 =A0kstat.zfs.misc.arcstats.c_min: 26214400 > =A0 =A0 =A0 =A0kstat.zfs.misc.arcstats.c_max: 209715200 > > and while monitoring kstat.zfs.misc.arcstats.size I noticed it fluctuated > between around 140 and 150 million. =A0Interestingly enough, immediately > before the last panic, It (arcstats.size) had just been reduced from > 150038268 to 128790020. > > I'm going to continue to poke around in my core dumps and see if I can > figure out what's going on. =A0Any hints as to what to look for or monito= r > while the system is running would be appreciated. > > =A0 =A0 =A0 =A0Thanks, > =A0 =A0 =A0 =A0Tim > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > --=20 When bad men combine, the good must associate; else they will fall one by one, an unpitied sacrifice in a contemptible struggle. Edmund Burke
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3c1674c90906041413n15e2ce88q9f279818641515e7>