From owner-freebsd-questions@FreeBSD.ORG Wed Jan 13 19:58:16 2010 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 057851065670 for ; Wed, 13 Jan 2010 19:58:16 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from mail-ew0-f211.google.com (mail-ew0-f211.google.com [209.85.219.211]) by mx1.freebsd.org (Postfix) with ESMTP id 8AC208FC18 for ; Wed, 13 Jan 2010 19:58:15 +0000 (UTC) Received: by ewy3 with SMTP id 3so10834ewy.13 for ; Wed, 13 Jan 2010 11:58:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:from:date:x-google-sender-auth:message-id:subject:to:cc :content-type:content-transfer-encoding; bh=Ys9RrHvFR/IF3BvbVSSggLDPlvYH1lYZy1qa1GppuAo=; b=f1jHUcuA9lnJH/5wcJAiMAG8C5nmP5aG+qSsO16d8Vl1W8KQxLnuuP5Y4cV4c18QSe FXEhhPExCAaI6BxMOFmgz7QbBqbbdcVkxz7TqVlebw0JfJJo8uwhIq6E7kQx1dTssd3H qnFU45V2xb1S/8ijTFjBfAFwzMv9JGfp2ckUY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type :content-transfer-encoding; b=k4LDY/2bvQx2Jbp7DbBe6XRhdhpgOiILIhx/7Bn9Q2WRv5Fum8K6+uIkRzUJ6+piKb VIMorgG7EmrhTLEG8I+WFa/qBhd8o7N5jVHKqTJwC5IvBuH4+wI2FlzcXSpDeXq/DbgQ pGjhmPjLhs4h1m/la12IMATK3f/Wj5DI8GC94= MIME-Version: 1.0 Sender: ivoras@gmail.com Received: by 10.216.88.143 with SMTP id a15mr2614578wef.206.1263412688471; Wed, 13 Jan 2010 11:58:08 -0800 (PST) In-Reply-To: References: <8418112cdfada93d83ca0cb5307c1d21.squirrel@email.polands.org> <9bbcef731001131035x604cdea1t81b14589cb10ad25@mail.gmail.com> From: Ivan Voras Date: Wed, 13 Jan 2010 20:57:46 +0100 X-Google-Sender-Auth: 45c2598874e7c3cb Message-ID: <9bbcef731001131157h256c4d14mbb241bc4326405f8@mail.gmail.com> To: Doug Poland Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-questions@freebsd.org Subject: Re: 8.0-R-p2 ZFS: unixbench causing kmem exhaustion panic X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jan 2010 19:58:16 -0000 2010/1/13 Doug Poland : > > On Wed, January 13, 2010 12:35, Ivan Voras wrote: >>>> >>>> Try adding vfs.zfs.arc_max=3D512M to /boot/loader.conf. >>>> >>> Would you suggest tweaking the vm.kmem_size tunables in addition to >>> arc_max? >> >> No, unless they auto-tune to something lesser than approximately >> arc_max*3. >> >> I try to set arc_max to be a third (or a quarter) the kmem_size, and >> tune kmem_size ad_hoc to suit the machine and its purpose. >> >> The reason for this is that "arc_max" is just a guideline, not a hard >> limit... the ZFS ARC usage can and will spike to much larger values, >> usually in the most inopportune moment. >> > This is the state of the machine when it panicked this time: > > panic: kmem_malloc(131072): kmem_map too small: 1296957440 total > allocated > cpuid =3D 1 > > /boot/loader.conf: vfs.zfs.arc_max=3D512M > vfs.numvnodes: 660 > vfs.zfs.arc_max: 536870912 > vfs.zfs.arc_meta_limit: 134217728 > vfs.zfs.arc_meta_used: 7006136 > vfs.zfs.arc_min: 67108864 > vfs.zfs.zil_disable: 0 > vm.kmem_size: 1327202304 > vm.kmem_size_max: 329853485875 (from the size of arc_max I assume you did remember to reboot after changing loader.conf and before testing again but just checking - did you?) Can you monitor and record kstat.zfs.misc.arcstats.size sysctl while the test is running (and crashing)? This looks curious - your kmem_max is ~~ 1.2 GB, arc_max is 0.5 GB and you are still having panics. Is there anything unusual about your system? Like unusually slow CPU, unusually fast or slow drives? I don't have any ideas smarter than reducing arc_max by half then try again and continue reducing it until it works. It would be very helpful if you could monitor the kstat.zfs.misc.arcstats.size sysctl while you are doing the tests to document what is happening to the system. If it by any chance stays the same you should probably monitor "vmstat -m". > Using a handy little script I found posted in several places, I was > monitoring memory: > > TEXT =C2=A0 =C2=A0 15373968 =C2=A0 =C2=A0 =C2=A0 14.66 =C2=A0 MiB > DATA =C2=A0 1536957440 =C2=A0 =C2=A0 =C2=A0 1465.76 MiB > TOTAL =C2=A01552331408 =C2=A0 =C2=A0 =C2=A0 1480.42 MiB > > Where TEXT =3D a sum of kldstat memory values > and =C2=A0 DATA =3D a sum of vmstat -m values > > Is there a next step to try, or is this chasing a wild goose? > > > -- > Regards, > Doug > >