From owner-freebsd-performance@FreeBSD.ORG Fri Aug 17 03:15:47 2012 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7F4C2106566B for ; Fri, 17 Aug 2012 03:15:47 +0000 (UTC) (envelope-from gezeala@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 4BB698FC08 for ; Fri, 17 Aug 2012 03:15:47 +0000 (UTC) Received: by pbbrp2 with SMTP id rp2so2781967pbb.13 for ; Thu, 16 Aug 2012 20:15:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=references:in-reply-to:mime-version:content-type :content-transfer-encoding:message-id:cc:x-mailer:from:subject:date :to; bh=V82TycksMa6Xc59BS/W0KEcMGZgG4L9jVc5gsuB0cHM=; b=Hb+30HV1n9y7mCvwORHVI3BrJFfALER+8JXQLHRgmub2dfMLSDAQQ5xPl+/iSzxQBX ddDcoAb3RJCqKG95R9Njeh0aPYee1bpahdFeraoWeXvHfYwQDvYqMnogFCCPUFrbk/Hl 9M/+WM+MfdLnW6vF8p+B3d4MM9FfXRXiosQPREwonyzXoj0XPJqTIuYRh/MjtcmtLrkB GFf9VYJDtFA5AqACa53KJcnyj05jm8ZItYx+L1DWVDKd3MJeaSclmIej2gmeU1w7mHOp qOuJ5DAcVbV3x3Kp+6qmIzqoiWBPGvjvz8A1L3uGlZCyJRhNtUmJwVUrMufoH+j7E7O9 ovNw== Received: by 10.66.74.100 with SMTP id s4mr6436401pav.27.1345173346628; Thu, 16 Aug 2012 20:15:46 -0700 (PDT) Received: from [192.168.1.69] (pool-96-251-115-86.lsanca.dsl-w.verizon.net. [96.251.115.86]) by mx.google.com with ESMTPS id or1sm3943274pbb.10.2012.08.16.20.15.44 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 16 Aug 2012 20:15:45 -0700 (PDT) References: In-Reply-To: Mime-Version: 1.0 (iPod Mail 8C148) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Message-Id: X-Mailer: iPod Mail (8C148) From: Marie Bacuno II Date: Thu, 16 Aug 2012 20:15:48 -0700 To: Garrett Cooper Cc: "freebsd-performance@freebsd.org" Subject: Re: vm.kmem_size_max and vm.kmem_size capped at 329853485875 (~307GB) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Aug 2012 03:15:47 -0000 On Aug 16, 2012, at 18:47, Garrett Cooper wrote: > On Thu, Aug 16, 2012 at 6:44 PM, Garrett Cooper wrote= : >> On Thu, Aug 16, 2012 at 5:46 PM, Gezeala M. Bacu=C3=B1o II wrote: >>> Hello fellow listers, >>>=20 >>> On a server with 512GB RAM it appears that vm.kmem_size_max is not >>> being auto-tuned to use >329853485875 (~307GB). >>>=20 >>> On this machine vm.kmem_size is equal to vm.kmem_size_max >>>=20 >>> # from sysctl >>> vm.kmem_size_max: 329853485875 >>> vm.kmem_size: 329853485875 >>>=20 >>> On a machine with 1GB of RAM, I have successfully set vm.kmem_size_max >>> to 330GB and vm.kmem_size automatically adjusts to 1GB even if I >>> manually set it in /boot/loader.conf. >>>=20 >>> But on the machine with 512GB of RAM it just resets. For the machine >>> to boot, we need to go to the loader prompt and issue: >>>=20 >>> OK set vm.kmem_size_max=3D"300G" >>> OK boot >>>=20 >>> On all PCBSD (8,9) or FreeBSD (8.1,8.2,9) machines we have, >>> vm.kmem_size_max is always set to 329853485875. >>>=20 >>> How can I increase vm.kmem_size_max to use at least 500GB? And how is >>> 329853485875 determined (formula)? I need to increase vm.kmem_size_max >>> and vm.kmem_size so I can set vfs.zfs.arc_max (ZFS ARC) to use say >>> 490GB. >>>=20 >>> I'm browsing thru the source code at >>> http://fxr.watson.org/fxr/search?v=3DFREEBSD9&string=3Dvm.kmem_size_max >>> and I'm still trying to make sense of how vm.kmem_size_max is >>> computed. >>>=20 >>> I have posted the same topic on forums.freebsd.org but I'm not getting >>> any recommendations. >>>=20 >>> Please see the link for additional details: >>> http://forums.freebsd.org/showthread.php?t=3D33977 >>=20 >> Have you tried defining VM_KMEM_SIZE_MAX to your target value? >>=20 >> Its architecture specific BTW... see >> sys//include/vmparam.h -- look for `VM_KMEM_SIZE_MAX`. >=20 > Also, it's a tunable, not a sysctl... so you need to set the value in > /boot/loader.conf . > -Garrett Thanks for the quick reply. Yes, had it set on /boot/loader.conf and by trial and error on the loader pr= ompt. We were able to bump it to 400G successfully. Tried 500G and 450G and the ma= chine just spews out garbage in the screen. The latest output from "zfs-stats -a" with vm.kmem_size_max=3D"400G" is in t= he forum: http://forums.freebsd.org/showthread.php?t=3D33977 About the code, I am looking into amd64 arch. Still checking the values of t= he variables.. Can't just retrieve them using getconf. If you can point me t= o a doxygen like documentation appreciate it a lot. Where does the constant value 329853485875 came from? Thanks!