From owner-freebsd-questions@FreeBSD.ORG Wed Jan 13 21:39:18 2010 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C9B82106566C; Wed, 13 Jan 2010 21:39:18 +0000 (UTC) (envelope-from doug@polands.org) Received: from hrndva-omtalb.mail.rr.com (hrndva-omtalb.mail.rr.com [71.74.56.122]) by mx1.freebsd.org (Postfix) with ESMTP id 687CE8FC17; Wed, 13 Jan 2010 21:39:18 +0000 (UTC) X-Authority-Analysis: v=1.0 c=1 a=sVhNVL3m-NYA:10 a=tH3TCQxNhl/vWwTfLVWeHw==:17 a=bqq2Vc5EAAAA:8 a=9G5cZbBhCoHlf84lHYQA:9 a=6simL32w41UMB-q6hA-ce_G0r0sA:4 a=5ERLOmoKdHQA:10 X-Cloudmark-Score: 0 X-Originating-IP: 75.87.219.217 Received: from [75.87.219.217] ([75.87.219.217:64009] helo=haran.polands.org) by hrndva-oedge04.mail.rr.com (envelope-from ) (ecelerity 2.2.2.39 r()) with ESMTP id 5D/82-18377-58D3E4B4; Wed, 13 Jan 2010 21:39:17 +0000 Received: from email.polands.org (ammon.polands.org [172.16.1.7]) by haran.polands.org (8.14.3/8.14.3) with ESMTP id o0DLdGSD088404; Wed, 13 Jan 2010 15:39:16 -0600 (CST) (envelope-from doug@polands.org) Received: from 209.103.214.35 (SquirrelMail authenticated user djp) by email.polands.org with HTTP; Wed, 13 Jan 2010 15:39:16 -0600 Message-ID: <3aa09fd8723749d1fa65f1b9a6faac60.squirrel@email.polands.org> In-Reply-To: <9bbcef731001131157h256c4d14mbb241bc4326405f8@mail.gmail.com> References: <8418112cdfada93d83ca0cb5307c1d21.squirrel@email.polands.org> <9bbcef731001131035x604cdea1t81b14589cb10ad25@mail.gmail.com> <9bbcef731001131157h256c4d14mbb241bc4326405f8@mail.gmail.com> Date: Wed, 13 Jan 2010 15:39:16 -0600 From: "Doug Poland" To: "Ivan Voras" User-Agent: SquirrelMail/1.4.20-RC2 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal Cc: freebsd-questions@freebsd.org Subject: Re: 8.0-R-p2 ZFS: unixbench causing kmem exhaustion panic X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jan 2010 21:39:19 -0000 On Wed, January 13, 2010 13:57, Ivan Voras wrote: > 2010/1/13 Doug Poland : >> > > Can you monitor and record kstat.zfs.misc.arcstats.size sysctl while > the test is running (and crashing)? > > This looks curious - your kmem_max is ~~ 1.2 GB, arc_max is 0.5 GB and > you are still having panics. Is there anything unusual about your > system? Like unusually slow CPU, unusually fast or slow drives? > > I don't have any ideas smarter than reducing arc_max by half then try > again and continue reducing it until it works. It would be very > helpful if you could monitor the kstat.zfs.misc.arcstats.size sysctl > while you are doing the tests to document what is happening to the > system. If it by any chance stays the same you should probably monitor > "vmstat -m". > > Ok, I re-ran with same config, but this time monitoring the sysctls you requested* ( and the rest I was watching ): panic: kmem_malloc(131072): kmem_map too small: 1292869632 total allocated cpuid = 0 * kstat.zfs.misc.arcstats.size: 166228176 vfs.numvnodes: 2848 vfs.zfs.arc_max: 536870912 vfs.zfs.arc_meta_limit: 134217728 vfs.zfs.arc_meta_used: 132890832 vfs.zfs.arc_min: 67108864 vfs.zfs.cache_flush_disable: 0 vfs.zfs.debug: 0 vfs.zfs.mdcomp_disable: 0 vfs.zfs.prefetch_disable: 1 vfs.zfs.recover: 0 vfs.zfs.scrub_limit: 10 vfs.zfs.super_owner: 0 vfs.zfs.txg.synctime: 5 vfs.zfs.txg.timeout: 30 vfs.zfs.vdev.aggregation_limit: 131072 vfs.zfs.vdev.cache.bshift: 16 vfs.zfs.vdev.cache.max: 16384 vfs.zfs.vdev.cache.size: 10485760 vfs.zfs.vdev.max_pending: 35 vfs.zfs.vdev.min_pending: 4 vfs.zfs.vdev.ramp_rate: 2 vfs.zfs.vdev.time_shift: 6 vfs.zfs.version.acl: 1 vfs.zfs.version.dmu_backup_header: 2 vfs.zfs.version.dmu_backup_stream: 1 vfs.zfs.version.spa: 13 vfs.zfs.version.vdev_boot: 1 vfs.zfs.version.zpl: 3 vfs.zfs.zfetch.array_rd_sz: 1048576 vfs.zfs.zfetch.block_cap: 256 vfs.zfs.zfetch.max_streams: 8 vfs.zfs.zfetch.min_sec_reap: 2 vfs.zfs.zil_disable: 0 vm.kmem_size: 1327202304 vm.kmem_size_max: 329853485875 vm.kmem_size_min: 0 vm.kmem_size_scale: 3 * vmstat -m | grep solaris: 1496232960 -- Regards, Doug