From owner-freebsd-performance@FreeBSD.ORG Tue Aug 21 01:27:19 2012 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 65B24106566C; Tue, 21 Aug 2012 01:27:19 +0000 (UTC) (envelope-from gezeala@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 1D4278FC08; Tue, 21 Aug 2012 01:27:18 +0000 (UTC) Received: by pbbrp2 with SMTP id rp2so8196829pbb.13 for ; Mon, 20 Aug 2012 18:27:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:content-transfer-encoding; bh=cySjppLIqnuggfSvYjQN1mA9Nfm1hRGER/v4QzDFiSg=; b=EtRfA2sDans47q2ceuiC5ZSpJGwjC8KoqTpV1lW7WRF4900Zij3GBaj7+3iVvjNGx6 0bEqg90/OBNx8hzGIyZpr4zn8DThh7aLQbtMyVoK3YIX6INNb6WehuWdaSgrSZiNJotx Ex5BdCcGMWo9HDljnxFBFoKcr/bfKV6I53eudOio7dA3+LjWKda/O26SQPiT3P3D0KBJ vvifFfl38UkksiAZMHyla+j4IK8tVAlOoS5ZSazjsAmo/653CXR0ZwHr/muDwMwhcxAI FUsjseHiQERpi/BnqgknRqE1YD0AwZjlSlnN5rbWAKFO9JvS33uvaiyPyz7h3pkG00nj Alzw== Received: by 10.68.237.38 with SMTP id uz6mr33921625pbc.23.1345512438429; Mon, 20 Aug 2012 18:27:18 -0700 (PDT) MIME-Version: 1.0 Received: by 10.68.117.145 with HTTP; Mon, 20 Aug 2012 18:26:58 -0700 (PDT) In-Reply-To: References: <502DEAD9.6050304@zonov.org> <502EB081.3030801@rice.edu> <502FE98E.40807@rice.edu> <50325634.7090904@rice.edu> From: =?ISO-8859-1?Q?Gezeala_M=2E_Bacu=F1o_II?= Date: Mon, 20 Aug 2012 18:26:58 -0700 Message-ID: To: Alan Cox Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: alc@freebsd.org, freebsd-performance@freebsd.org, Andrey Zonov , kib@freebsd.org Subject: Re: vm.kmem_size_max and vm.kmem_size capped at 329853485875 (~307GB) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Aug 2012 01:27:19 -0000 On Mon, Aug 20, 2012 at 9:07 AM, Gezeala M. Bacu=F1o II = wrote: > On Mon, Aug 20, 2012 at 8:22 AM, Alan Cox wrote: >> On 08/18/2012 19:57, Gezeala M. Bacu=F1o II wrote: >>> >>> On Sat, Aug 18, 2012 at 12:14 PM, Alan Cox wrote: >>>> >>>> On 08/17/2012 17:08, Gezeala M. Bacu=F1o II wrote: >>>>> >>>>> On Fri, Aug 17, 2012 at 1:58 PM, Alan Cox wrote: >>>>>> >>>>>> vm.kmem_size controls the maximum size of the kernel's heap, i.e., t= he >>>>>> region where the kernel's slab and malloc()-like memory allocators >>>>>> obtain >>>>>> their memory. While this heap may occupy the largest portion of the >>>>>> kernel's virtual address space, it cannot occupy the entirety of the >>>>>> address >>>>>> space. There are other things that must be given space within the >>>>>> kernel's >>>>>> address space, for example, the file system buffer map. >>>>>> >>>>>> ZFS does not, however, use the regular file system buffer cache. The >>>>>> ARC >>>>>> takes its place, and the ARC abuses the kernel's heap like nothing >>>>>> else. >>>>>> So, if you are running a machine that only makes trivial use of a >>>>>> non-ZFS >>>>>> file system, like you boot from UFS, but store all of your data in Z= FS, >>>>>> then >>>>>> you can dramatically reduce the size of the buffer map via boot load= er >>>>>> tuneables and proportionately increase vm.kmem_size. >>>>>> >>>>>> Any further increases in the kernel virtual address space size will, >>>>>> however, require code changes. Small changes, but changes nonethele= ss. >>>>>> >>>>>> Alan >>>>>> > <> >>>> Your objective should be to reduce the value of "sysctl vfs.maxbufspac= e". >>>> You can do this by setting the loader.conf tuneable "kern.maxbcache" t= o >>>> the >>>> desired value. >>>> >>>> What does your machine currently report for "sysctl vfs.maxbufspace"? >>>> >>> Here you go: >>> vfs.maxbufspace: 54967025664 >>> kern.maxbcache: 0 >> >> >> Try setting kern.maxbcache to two billion and adding 50 billion to the >> setting of vm.kmem_size{,_max}. >> 2 : 50 =3D=3D>> is this the ratio for further tuning kern.maxbcache:vm.kmem_size? Is kern.maxbcache also in bytes? > > Thank you. We'll try this and post back results. > >>> Other (probably) relevant values: >>> vfs.hirunningspace: 16777216 >>> vfs.lorunningspace: 11206656 >>> vfs.bufdefragcnt: 0 >>> vfs.buffreekvacnt: 2 >>> vfs.bufreusecnt: 320149 >>> vfs.hibufspace: 54966370304 >>> vfs.lobufspace: 54966304768 >>> vfs.maxmallocbufspace: 2748318515 >>> vfs.bufmallocspace: 0 >>> vfs.bufspace: 10490478592 >>> vfs.runningbufspace: 0 >>> >>> Let me know if you need other tuneables or sysctl values. Thanks a lot >>> for looking into this. >>> >> Setting the following as instructed, machine started successfully with 446GB for vm.kmem_size/_max. kern.maxbcache: 2000000000 vm.kmem_size_max: 479496729600 vm.kmem_size: 479496729600 ## auto-tuned vfs.maxbufspace: 1999994880 ... vfs.hirunningspace: 16777216 vfs.lorunningspace: 11206656 vfs.bufdefragcnt: 0 vfs.buffreekvacnt: 2 vfs.bufreusecnt: 11511 vfs.hibufspace: 1999339520 vfs.lobufspace: 1999273984 vfs.maxmallocbufspace: 99966976 vfs.bufmallocspace: 0 vfs.bufspace: 377028608 vfs.runningbufspace: 0 ## additional manual tuning vfs.zfs.arc_max: 455521893120 vfs.zfs.arc_min: 227760946560 kern.ipc.semmni: 256 kern.ipc.semmns: 512 kern.ipc.semmnu: 256 kern.ipc.shm_use_phys: 1 kern.ipc.shmmax: 24000000000 kern.ipc.shmall: 5859375 kern.ipc.nmbclusters: 2560000 kern.maxfiles: 197248 We'll do some further tests and report back if there are any issues. Thanks a lot!!