Date: Wed, 13 Jan 2010 14:26:41 -0600 From: "Doug Poland" <doug@polands.org> To: "Ivan Voras" <ivoras@freebsd.org> Cc: questions@freebsd.org Subject: Re: 8.0-R-p2 ZFS: unixbench causing kmem exhaustion panic Message-ID: <158692d18f40556dc57008f1661afd48.squirrel@email.polands.org> In-Reply-To: <9bbcef731001131157h256c4d14mbb241bc4326405f8@mail.gmail.com> References: <8418112cdfada93d83ca0cb5307c1d21.squirrel@email.polands.org> <hil1e8$ebs$1@ger.gmane.org> <b78f9b16683331ad0f574ecfc1b7f995.squirrel@email.polands.org> <9bbcef731001131035x604cdea1t81b14589cb10ad25@mail.gmail.com> <b41ca31fbeacf104143509e8cba2fe66.squirrel@email.polands.org> <9bbcef731001131157h256c4d14mbb241bc4326405f8@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, January 13, 2010 13:57, Ivan Voras wrote: > 2010/1/13 Doug Poland <doug@polands.org>: >> This is the state of the machine when it panicked this time: >> >> panic: kmem_malloc(131072): kmem_map too small: 1296957440 total >> allocated >> cpuid = 1 >> >> /boot/loader.conf: vfs.zfs.arc_max=512M >> vfs.numvnodes: 660 >> vfs.zfs.arc_max: 536870912 >> vfs.zfs.arc_meta_limit: 134217728 >> vfs.zfs.arc_meta_used: 7006136 >> vfs.zfs.arc_min: 67108864 >> vfs.zfs.zil_disable: 0 >> vm.kmem_size: 1327202304 >> vm.kmem_size_max: 329853485875 > > (from the size of arc_max I assume you did remember to reboot after > changing loader.conf and before testing again but just checking - did > you?) > Yes, I did reboot > Can you monitor and record kstat.zfs.misc.arcstats.size sysctl while > the test is running (and crashing)? > Certainly > This looks curious - your kmem_max is ~~ 1.2 GB, arc_max is 0.5 GB and > you are still having panics. Is there anything unusual about your > system? Like unusually slow CPU, unusually fast or slow drives? > Don't think there is anything unusual. This is 5 year old HP DL385. It has two 2.6GHz Opteron 252 CPUs. The disks are 6x36GB P-SCSI. There are behind an HP Smart Array 6i controller. I had to configure each drive as "RAID0" in order make it visible to the OS. Kinda hokey if you ask me. dmesg | grep -i CPU CPU: AMD Opteron(tm) Processor 252 (2605.92-MHz K8-class CPU) FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs smartctl -a /dev/da0 Device: COMPAQ RAID 0 VOLUME Version: OK Device type: disk Local Time is: Wed Jan 13 14:21:44 2010 CST Device does not support SMART dmesg | grep -i smart ciss0: <HP Smart Array 6i> port 0x5000-0x50ff mem 0xf7ef0000-0xf7ef1fff,0xf7e80000-0xf7ebffff irq 24 at device 4.0 on pci2 > I don't have any ideas smarter than reducing arc_max by half then try > again and continue reducing it until it works. It would be very > helpful if you could monitor the kstat.zfs.misc.arcstats.size sysctl > while you are doing the tests to document what is happening to the > system. If it by any chance stays the same you should probably monitor > "vmstat -m". > OK, will do monitor on the next run. Thanks for your help so far. -- Regards, Doug
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?158692d18f40556dc57008f1661afd48.squirrel>