From owner-freebsd-stable@FreeBSD.ORG Wed May 5 14:46:25 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1EED91065748 for ; Wed, 5 May 2010 14:46:25 +0000 (UTC) (envelope-from korvus@comcast.net) Received: from qmta15.westchester.pa.mail.comcast.net (qmta15.westchester.pa.mail.comcast.net [76.96.59.228]) by mx1.freebsd.org (Postfix) with ESMTP id BABA78FC13 for ; Wed, 5 May 2010 14:46:24 +0000 (UTC) Received: from omta22.westchester.pa.mail.comcast.net ([76.96.62.73]) by qmta15.westchester.pa.mail.comcast.net with comcast id Doqw1e00B1ap0As5FqmRh2; Wed, 05 May 2010 14:46:25 +0000 Received: from [192.168.2.164] ([206.210.89.202]) by omta22.westchester.pa.mail.comcast.net with comcast id Dqm81e00H4Mx3R23iqmCdQ; Wed, 05 May 2010 14:46:23 +0000 Message-ID: <4BE184AE.7070307@comcast.net> Date: Wed, 05 May 2010 10:46:06 -0400 From: Steve Polyack User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.7) Gecko/20100311 Thunderbird/3.0.1 MIME-Version: 1.0 To: Ben Kelly References: <4BDEA86E.3050109@zirakzigil.org> <20100503110100.GA93137@icarus.home.lan> <4BDEC106.3040807@zirakzigil.org> <4BE110E3.8040902@zirakzigil.org> <20100505075242.GA57550@icarus.home.lan> <4BE13067.1060606@zirakzigil.org> <20100505133302.GB1626@garage.freebsd.pl> <533152DA-3B2F-4994-9206-727A2B0010AD@wanderview.com> In-Reply-To: <533152DA-3B2F-4994-9206-727A2B0010AD@wanderview.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, FreeBSD Stable , Pawel Jakub Dawidek Subject: Re: Freebsd 8.0 kmem map too small X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 May 2010 14:46:25 -0000 On 05/05/10 10:12, Ben Kelly wrote: > On May 5, 2010, at 9:33 AM, Pawel Jakub Dawidek wrote: > > >> On Wed, May 05, 2010 at 10:46:31AM +0200, Giulio Ferro wrote: >> >>> On 05.05.2010 09:52, Jeremy Chadwick wrote: >>> >>> Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G... >>> >>> >>> >>>> Did you set both vm.kmem_size and vfs.zfs.arc_max, setting the latter to >>>> something *less* than vm.kmem_size? >>>> >>>> >>>> >>> Yes. >>> After your suggestion, I set >>> vfs.zfs.arc_max: 3758096384 >>> vm.kmem_size: 4G >>> >>> Now: >>> vfs.zfs.arc_max: 3758096384 >>> vm.kmem_size: 6392119296 >>> >> Could you try to track down the commit that is causing your problems? >> Could you try 8-STABLE kernel from before r206815? >> > > Are others generally able to run ARC values so close to kmem size? My experience has been that you really need the ARC to be much smaller than the kmem limit (like 25% or less) due to fragmentation of kmem_map. > > On a system here with 8GB of RAM and a very large zpool consisting of multiple zdevs, little tuning was needed. The system is running 8-STABLE(amd64) as of a month or two ago. The only things I have set in /boot/loader.conf are: vm.kmem_size="12G" vfs.zfs.arc_max="4G" Setting kmem_size to 12G came from a combination of recommendations I saw in various mailing list posts (you can probably dig them up). Setting it to the physical memory size was the initial recommendation, while others recommended 1.5x physical memory size to help prevent fragmentation/wasted space in kmem. Regardless, this has served us quite well for the ~6 months the system has been in use. It has never crashed, even under intensive multi-threaded benchmarking. -Steve