Date: Wed, 13 Jan 2010 20:57:46 +0100 From: Ivan Voras <ivoras@freebsd.org> To: Doug Poland <doug@polands.org> Cc: freebsd-questions@freebsd.org Subject: Re: 8.0-R-p2 ZFS: unixbench causing kmem exhaustion panic Message-ID: <9bbcef731001131157h256c4d14mbb241bc4326405f8@mail.gmail.com> In-Reply-To: <b41ca31fbeacf104143509e8cba2fe66.squirrel@email.polands.org> References: <8418112cdfada93d83ca0cb5307c1d21.squirrel@email.polands.org> <hil1e8$ebs$1@ger.gmane.org> <b78f9b16683331ad0f574ecfc1b7f995.squirrel@email.polands.org> <9bbcef731001131035x604cdea1t81b14589cb10ad25@mail.gmail.com> <b41ca31fbeacf104143509e8cba2fe66.squirrel@email.polands.org>
next in thread | previous in thread | raw e-mail | index | archive | help
2010/1/13 Doug Poland <doug@polands.org>: > > On Wed, January 13, 2010 12:35, Ivan Voras wrote: >>>> >>>> Try adding vfs.zfs.arc_max=3D512M to /boot/loader.conf. >>>> >>> Would you suggest tweaking the vm.kmem_size tunables in addition to >>> arc_max? >> >> No, unless they auto-tune to something lesser than approximately >> arc_max*3. >> >> I try to set arc_max to be a third (or a quarter) the kmem_size, and >> tune kmem_size ad_hoc to suit the machine and its purpose. >> >> The reason for this is that "arc_max" is just a guideline, not a hard >> limit... the ZFS ARC usage can and will spike to much larger values, >> usually in the most inopportune moment. >> > This is the state of the machine when it panicked this time: > > panic: kmem_malloc(131072): kmem_map too small: 1296957440 total > allocated > cpuid =3D 1 > > /boot/loader.conf: vfs.zfs.arc_max=3D512M > vfs.numvnodes: 660 > vfs.zfs.arc_max: 536870912 > vfs.zfs.arc_meta_limit: 134217728 > vfs.zfs.arc_meta_used: 7006136 > vfs.zfs.arc_min: 67108864 > vfs.zfs.zil_disable: 0 > vm.kmem_size: 1327202304 > vm.kmem_size_max: 329853485875 (from the size of arc_max I assume you did remember to reboot after changing loader.conf and before testing again but just checking - did you?) Can you monitor and record kstat.zfs.misc.arcstats.size sysctl while the test is running (and crashing)? This looks curious - your kmem_max is ~~ 1.2 GB, arc_max is 0.5 GB and you are still having panics. Is there anything unusual about your system? Like unusually slow CPU, unusually fast or slow drives? I don't have any ideas smarter than reducing arc_max by half then try again and continue reducing it until it works. It would be very helpful if you could monitor the kstat.zfs.misc.arcstats.size sysctl while you are doing the tests to document what is happening to the system. If it by any chance stays the same you should probably monitor "vmstat -m". > Using a handy little script I found posted in several places, I was > monitoring memory: > > TEXT =C2=A0 =C2=A0 15373968 =C2=A0 =C2=A0 =C2=A0 14.66 =C2=A0 MiB > DATA =C2=A0 1536957440 =C2=A0 =C2=A0 =C2=A0 1465.76 MiB > TOTAL =C2=A01552331408 =C2=A0 =C2=A0 =C2=A0 1480.42 MiB > > Where TEXT =3D a sum of kldstat memory values > and =C2=A0 DATA =3D a sum of vmstat -m values > > Is there a next step to try, or is this chasing a wild goose? > > > -- > Regards, > Doug > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9bbcef731001131157h256c4d14mbb241bc4326405f8>