From owner-freebsd-stable@FreeBSD.ORG Thu Sep 25 14:51:56 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4143A1065692 for ; Thu, 25 Sep 2008 14:51:56 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from QMTA01.emeryville.ca.mail.comcast.net (qmta01.emeryville.ca.mail.comcast.net [76.96.30.16]) by mx1.freebsd.org (Postfix) with ESMTP id 25DCF8FC1F for ; Thu, 25 Sep 2008 14:51:55 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from OMTA13.emeryville.ca.mail.comcast.net ([76.96.30.52]) by QMTA01.emeryville.ca.mail.comcast.net with comcast id JzRW1a00J17UAYkA12rdgk; Thu, 25 Sep 2008 14:51:37 +0000 Received: from koitsu.dyndns.org ([67.180.253.227]) by OMTA13.emeryville.ca.mail.comcast.net with comcast id K2ru1a0084v8bD78Z2ruVK; Thu, 25 Sep 2008 14:51:55 +0000 X-Authority-Analysis: v=1.0 c=1 a=pl5uMaLHwrUA:10 a=20ZdTaP2r60A:10 a=QycZ5dHgAAAA:8 a=E0mws_dX3nLwywMrpwMA:9 a=Nb8y8zYARSa-ScBy8pEoeovNRCEA:4 a=EoioJ0NPDVgA:10 a=LY0hPdMaydYA:10 Received: by icarus.home.lan (Postfix, from userid 1000) id 226E9C9419; Thu, 25 Sep 2008 07:51:54 -0700 (PDT) Date: Thu, 25 Sep 2008 07:51:54 -0700 From: Jeremy Chadwick To: Bartosz Stec Message-ID: <20080925145154.GA15486@icarus.home.lan> References: <48DB6772.1060400@kkip.pl> <20080925130227.GA13497@icarus.home.lan> <48DB9CAA.9060807@kkip.pl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <48DB9CAA.9060807@kkip.pl> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-stable@freebsd.org Subject: Re: vm.kmem_size settings doesn't affect loader? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Sep 2008 14:51:56 -0000 On Thu, Sep 25, 2008 at 04:14:02PM +0200, Bartosz Stec wrote: >> Your options are: >> >> 1) Consider increasing it from 512M to something like 1.5GB; do not >> increase it past that on RELENG_7, as there isn't support for more than >> 2GB total. For example, on a 1GB memory machine, I often recommend >> 768M. On 2GB machines, 1536M. You will need to run -CURRENT if you >> want more. >> >> 2) Tune ZFS aggressively. Start by setting vfs.zfs.arc_min="16M" >> and vfs.zfs.arc_max="64M". >> >> If your machine has some small amount of memory (768MB, 1GB, etc.), >> then you probably shouldn't be using ZFS. >> >> > Problem occured on i386 machine with 1GB of memory and 7.1-pre (3HDD, > 40GB, RAIDZ1). I know that i386 is highly unrecommended for ZFS, but > it's just a home box for testing and learning purposes - I just want to > know what I'm doing and what should I expect when I decide to put ZFS on > server machines :) Currently, from posts on freebsd-fs, I conclude that > even with a gigs of kmem and using AMD64, we still can experience panic > from kmem_malloc. The i386 vs. amd64 argument is bogus, if you ask me. ZFS works on both. amd64 is recommended because ZFS contains code that makes heavy use of 64-bit values, and because amd64 offers large amounts of addressed memory without disgusting hacks like PAE. That said -- yes, even with "gigs of kmem and using AMD64", you can still panic due to kmem exhaustion. I have fairly decent experience with this problem, because it haunted me for quite some time. A large portion of the problem is that kmem_max, on i386 and amd64 (yes, you read that right) has a 2GB limit on RELENG_7. I repeat: a 2GB limit, regardless of i386 or amd64. This limit has been increased to 512GB on CURRENT, but there are no plans to MFC those changes, as they are too major. Let me tell you something I did this weekend. I had to copy literally 200GB of data from a ZFS raidz1 pool (spread across 3 disks) to two different places: 1) a UFS2 filesystem on a different disk, and 2) across a gigE network to a Windows machine. I had to do this because I was adding a disk to the vdev, which cannot be done without re-creating the pool (this is a known problem with ZFS, and has nothing to do with FreeBSD). The machine hosting the data runs RELENG_7 with amd64, and contains 4GB of memory. However, I've accomplished the same task with only 2GB of memory as well. These are the tuning settings I use: vm.kmem_size="1536M" vm.kmem_size_max="1536M" vfs.zfs.arc_min="16M" vfs.zfs.arc_max="64M" The entire copying process took almost 2 hours. Not once did I experience kmem exhaustion. I can *guarantee* that I would have crashed the box numerous times had I not tuned the machine with the values above. > Manual tuning is hard for me because I'm not familiar > with BSD kernel code nor kernel memory management. I'm just an end-user > who love concepts of ZFS and wait for it to be (more) stable. Of course > I've followed tuning guide carefully. I'm an "experienced" end-user who has very little experience with BSD kernel code and absolutely no experience with kernel memory management. Proper tuning is all that's needed, regardless of your knowledge set. Please try installing 2GB of memory in your i386 box, and then use the exact loader.conf values I specified above. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |