From owner-freebsd-stable@FreeBSD.ORG Fri Sep 26 08:43:31 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 87C81106568A for ; Fri, 26 Sep 2008 08:43:31 +0000 (UTC) (envelope-from admin@kkip.pl) Received: from mainframe.kkip.pl (kkip.pl [87.105.164.78]) by mx1.freebsd.org (Postfix) with ESMTP id 354998FC21 for ; Fri, 26 Sep 2008 08:43:31 +0000 (UTC) (envelope-from admin@kkip.pl) Received: from [192.168.0.10] by mainframe.kkip.pl with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1Kj8v9-000HSQ-PB; Fri, 26 Sep 2008 10:43:29 +0200 Message-ID: <48DCA0AF.5050000@kkip.pl> Date: Fri, 26 Sep 2008 10:43:27 +0200 From: Bartosz Stec User-Agent: Thunderbird 2.0.0.16 (Windows/20080708) MIME-Version: 1.0 To: Jeremy Chadwick , freebsd-stable@freebsd.org References: <48DB6772.1060400@kkip.pl> <20080925130227.GA13497@icarus.home.lan> <48DB9CAA.9060807@kkip.pl> <20080925145154.GA15486@icarus.home.lan> In-Reply-To: <20080925145154.GA15486@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Authenticated-User: admin@kkip.pl X-Authenticator: plain X-Sender-Verify: SUCCEEDED (sender exists & accepts mail) X-Spam-Score: -8.9 X-Spam-Score-Int: -88 X-Exim-Version: 4.69 (build at 26-Jun-2008 18:19:28) X-Date: 2008-09-26 10:43:29 X-Connected-IP: 192.168.0.10:3134 X-Message-Linecount: 111 X-Body-Linecount: 98 X-Message-Size: 4858 X-Body-Size: 4196 X-Received-Count: 1 X-Recipient-Count: 2 X-Local-Recipient-Count: 2 X-Local-Recipient-Defer-Count: 0 X-Local-Recipient-Fail-Count: 0 Cc: Subject: Re: vm.kmem_size settings doesn't affect loader? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Sep 2008 08:43:31 -0000 Jeremy Chadwick wrote: > On Thu, Sep 25, 2008 at 04:14:02PM +0200, Bartosz Stec wrote: > >>> Your options are: >>> >>> 1) Consider increasing it from 512M to something like 1.5GB; do not >>> increase it past that on RELENG_7, as there isn't support for more than >>> 2GB total. For example, on a 1GB memory machine, I often recommend >>> 768M. On 2GB machines, 1536M. You will need to run -CURRENT if you >>> want more. >>> >>> 2) Tune ZFS aggressively. Start by setting vfs.zfs.arc_min="16M" >>> and vfs.zfs.arc_max="64M". >>> >>> If your machine has some small amount of memory (768MB, 1GB, etc.), >>> then you probably shouldn't be using ZFS. >>> >>> >>> >> Problem occured on i386 machine with 1GB of memory and 7.1-pre (3HDD, >> 40GB, RAIDZ1). I know that i386 is highly unrecommended for ZFS, but >> it's just a home box for testing and learning purposes - I just want to >> know what I'm doing and what should I expect when I decide to put ZFS on >> server machines :) Currently, from posts on freebsd-fs, I conclude that >> even with a gigs of kmem and using AMD64, we still can experience panic >> from kmem_malloc. >> > > The i386 vs. amd64 argument is bogus, if you ask me. ZFS works on both. > amd64 is recommended because ZFS contains code that makes heavy use of > 64-bit values, and because amd64 offers large amounts of addressed > memory without disgusting hacks like PAE. > > That said -- yes, even with "gigs of kmem and using AMD64", you can > still panic due to kmem exhaustion. I have fairly decent experience > with this problem, because it haunted me for quite some time. > > A large portion of the problem is that kmem_max, on i386 and amd64 (yes, > you read that right) has a 2GB limit on RELENG_7. I repeat: a 2GB > limit, regardless of i386 or amd64. > > This limit has been increased to 512GB on CURRENT, but there are no > plans to MFC those changes, as they are too major. > > Let me tell you something I did this weekend. I had to copy literally > 200GB of data from a ZFS raidz1 pool (spread across 3 disks) to two > different places: 1) a UFS2 filesystem on a different disk, and 2) > across a gigE network to a Windows machine. I had to do this because I > was adding a disk to the vdev, which cannot be done without re-creating > the pool (this is a known problem with ZFS, and has nothing to do with > FreeBSD). > > The machine hosting the data runs RELENG_7 with amd64, and contains 4GB > of memory. However, I've accomplished the same task with only 2GB of > memory as well. > > These are the tuning settings I use: > > vm.kmem_size="1536M" > vm.kmem_size_max="1536M" > vfs.zfs.arc_min="16M" > vfs.zfs.arc_max="64M" > > The entire copying process took almost 2 hours. Not once did I > experience kmem exhaustion. I can *guarantee* that I would have crashed > the box numerous times had I not tuned the machine with the values > above. > > >> Manual tuning is hard for me because I'm not familiar >> with BSD kernel code nor kernel memory management. I'm just an end-user >> who love concepts of ZFS and wait for it to be (more) stable. Of course >> I've followed tuning guide carefully. >> > > I'm an "experienced" end-user who has very little experience with BSD > kernel code and absolutely no experience with kernel memory management. > Proper tuning is all that's needed, regardless of your knowledge set. > > Please try installing 2GB of memory in your i386 box, and then use > the exact loader.conf values I specified above. > > Thank you for hints. Yesterday I've added 512 MB memory to box (sum 1,5GB), and set vm.kmem_size and vm.kmem_size to "1024M". With pieces of 1024MB, 512MB, 256MB, 256MB available and 3 memory slots it is hard to have 2GB RAM ;) Until now it survived world cleaning/building/installing/bonnie++ benchmarkink/fs scrubing and general usage. Memory usage seems stable. If unfortunately kmem exhaustion will happen again I will experiment with ARC settings. IMHO you've explained gently a lot of zfs tuning concerns in this thread and they should be added to tuning guide - espacially explanation of ARC and prefetch settings. Thanks again! -- Bartosz Stec