From owner-freebsd-performance@FreeBSD.ORG Fri Aug 17 20:16:35 2012 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4B6B7106566C; Fri, 17 Aug 2012 20:16:35 +0000 (UTC) (envelope-from gezeala@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 0C5878FC16; Fri, 17 Aug 2012 20:16:34 +0000 (UTC) Received: by pbbrp2 with SMTP id rp2so4142874pbb.13 for ; Fri, 17 Aug 2012 13:16:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:content-transfer-encoding; bh=IvCbIb3NSLZMJAkOkvetpxOzf2GxHLxqle6b7IRngeg=; b=LT0AL7qo2XtaksZWjlsDHAjwxlbYtrOyb4xdLi9O95F7i0ENbwS3FSB2rBAIxPAA3p AU7GYBeFtVr2W6YEcBo79nu2ca+FtJ7q2TnhxVTkCzTzaZkeqA7lde1kq8vvBhuVObx3 YcvHQ1Kgurz8qf22ZktJSdufpfC3Aw6jCb5K112Jpx+qJkPuGZTyo2f6JS6E9DnOjboo 8AVXtdYIu30qA2fVoVCgaYUd1+SEiRZALx1QEgy3EytW+aC8z0QFdNSSGrAhEntHTji0 9Y9MqQ6Fawn3xgvUeSaIfei2tGyeFQdZ/Obby0SYGoAc+1zLASiBel9beaZZqCoXt5/d TlIA== Received: by 10.68.230.232 with SMTP id tb8mr14352500pbc.19.1345234594421; Fri, 17 Aug 2012 13:16:34 -0700 (PDT) MIME-Version: 1.0 Received: by 10.68.190.71 with HTTP; Fri, 17 Aug 2012 13:16:13 -0700 (PDT) In-Reply-To: References: <502DEAD9.6050304@zonov.org> From: =?ISO-8859-1?Q?Gezeala_M=2E_Bacu=F1o_II?= Date: Fri, 17 Aug 2012 13:16:13 -0700 Message-ID: To: Andrey Zonov , alc@freebsd.org, kib@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-performance@freebsd.org Subject: Re: vm.kmem_size_max and vm.kmem_size capped at 329853485875 (~307GB) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Aug 2012 20:16:35 -0000 On Fri, Aug 17, 2012 at 7:38 AM, Gezeala M. Bacu=F1o II = wrote: > On Thu, Aug 16, 2012 at 11:55 PM, Andrey Zonov wrote: >> On 8/17/12 7:15 AM, Marie Bacuno II wrote: >>> >>> >>> On Aug 16, 2012, at 18:47, Garrett Cooper wrote: >>> >>>> On Thu, Aug 16, 2012 at 6:44 PM, Garrett Cooper >>>> wrote: >>>>> >>>>> On Thu, Aug 16, 2012 at 5:46 PM, Gezeala M. Bacu=F1o II >>>>> wrote: >>>>>> >>>>>> Hello fellow listers, >>>>>> >>>>>> On a server with 512GB RAM it appears that vm.kmem_size_max is not >>>>>> being auto-tuned to use >329853485875 (~307GB). >>>>>> >>>>>> On this machine vm.kmem_size is equal to vm.kmem_size_max >>>>>> >>>>>> # from sysctl >>>>>> vm.kmem_size_max: 329853485875 >>>>>> vm.kmem_size: 329853485875 >>>>>> >>>>>> On a machine with 1GB of RAM, I have successfully set vm.kmem_size_m= ax >>>>>> to 330GB and vm.kmem_size automatically adjusts to 1GB even if I >>>>>> manually set it in /boot/loader.conf. >>>>>> >>>>>> But on the machine with 512GB of RAM it just resets. For the machine >>>>>> to boot, we need to go to the loader prompt and issue: >>>>>> >>>>>> OK set vm.kmem_size_max=3D"300G" >>>>>> OK boot >>>>>> >>>>>> On all PCBSD (8,9) or FreeBSD (8.1,8.2,9) machines we have, >>>>>> vm.kmem_size_max is always set to 329853485875. >>>>>> >>>>>> How can I increase vm.kmem_size_max to use at least 500GB? And how i= s >>>>>> 329853485875 determined (formula)? I need to increase vm.kmem_size_m= ax >>>>>> and vm.kmem_size so I can set vfs.zfs.arc_max (ZFS ARC) to use say >>>>>> 490GB. >>>>>> >>>>>> I'm browsing thru the source code at >>>>>> http://fxr.watson.org/fxr/search?v=3DFREEBSD9&string=3Dvm.kmem_size_= max >>>>>> and I'm still trying to make sense of how vm.kmem_size_max is >>>>>> computed. >>>>>> >>>>>> I have posted the same topic on forums.freebsd.org but I'm not getti= ng >>>>>> any recommendations. >>>>>> >>>>>> Please see the link for additional details: >>>>>> http://forums.freebsd.org/showthread.php?t=3D33977 >>>>> >>>>> >>>>> Have you tried defining VM_KMEM_SIZE_MAX to your target value? >>>>> >>>>> Its architecture specific BTW... see >>>>> sys//include/vmparam.h -- look for `VM_KMEM_SIZE_MAX`. >>>> >>>> >>>> Also, it's a tunable, not a sysctl... so you need to set the value in >>>> /boot/loader.conf . >>>> -Garrett >>> >>> >>> Thanks for the quick reply. >>> >>> Yes, had it set on /boot/loader.conf and by trial and error on the load= er >>> prompt. >>> >>> We were able to bump it to 400G successfully. Tried 500G and 450G and t= he >>> machine just spews out garbage in the screen. >>> >>> The latest output from "zfs-stats -a" with vm.kmem_size_max=3D"400G" is= in >>> the forum: http://forums.freebsd.org/showthread.php?t=3D33977 >>> >>> About the code, I am looking into amd64 arch. Still checking the values= of >>> the variables.. Can't just retrieve them using getconf. If you can poin= t me >>> to a doxygen like documentation appreciate it a lot. >>> >>> Where does the constant value 329853485875 came from? >>> >> >> It comes from this macro: >> >> #define VM_KMEM_SIZE_MAX ((VM_MAX_KERNEL_ADDRESS - \ >> VM_MIN_KERNEL_ADDRESS + 1) * 3 / 5) >> >> ((1<<39) * 3 / 5) =3D 329853488332 >> >> AFAIK, VM_MAX_KERNEL_ADDRESS is limited to 512Gb. May be it's time to >> increase it again. I would asked kib@ or alc@ about that. >> >> -- >> Andrey Zonov > Thanks! That's great (great for deriving 512GB) but looks like bad news for us, we've really hit some limits there (FreeBSD auto-tuning wise). As I've stated above, we have tried setting vm.kmem_size_max to 500GB/450GB unsuccessfully so there may be some part of the code that's breaking. Is there any thread or discussion where you can point me as to why they used only 60%(hard coded) of 512GB? Some relevant codes I've gathered on this machine: /usr/src/sys/amd64/include/vmparam.h /* * Virtual addresses of things. Derived from the page directory and * page table indexes from pmap.h for precision. * * 0x0000000000000000 - 0x00007fffffffffff user map * 0x0000800000000000 - 0xffff7fffffffffff does not exist (hole) * 0xffff800000000000 - 0xffff804020100fff recursive page table (512GB sl= ot) * 0xffff804020101000 - 0xfffffdffffffffff unused * 0xfffffe0000000000 - 0xfffffeffffffffff 1TB direct map * 0xffffff0000000000 - 0xffffff7fffffffff unused * 0xffffff8000000000 - 0xffffffffffffffff 512GB kernel map * * Within the kernel map: * * 0xffffffff80000000 KERNBASE */ #define VM_MAX_KERNEL_ADDRESS KVADDR(KPML4I, NPDPEPG-1, NPDEPG-1, NPTEPG-= 1) #define VM_MIN_KERNEL_ADDRESS KVADDR(KPML4I, NPDPEPG-512, 0, 0) /usr/src/sys/amd64/include/pmap.h /* * Pte related macros. This is complicated by having to deal with * the sign extension of the 48th bit. */ #define KVADDR(l4, l3, l2, l1) ( \ ((unsigned long)-1 << 47) | \ ((unsigned long)(l4) << PML4SHIFT) | \ ((unsigned long)(l3) << PDPSHIFT) | \ ((unsigned long)(l2) << PDRSHIFT) | \ ((unsigned long)(l1) << PAGE_SHIFT)) /usr/src/sys/amd64/include/param.h #define PAGE_SHIFT 12 /* LOG2(PAGE_SIZE) */ #define PDRSHIFT 21 /* LOG2(NBPDR) */ #define PDPSHIFT 30 /* LOG2(NBPDP) */ #define PML4SHIFT 39 /* LOG2(NBPML4) */ Yet to derive: KPML4I, NPDPEPG-1, NPDEPG-1, NPTEPG-1. Checking pmap.c, vm_machdep.c etc. Additional Info: 1] Installed using PCBSD-9 Release amd64. 2] uname -a FreeBSD fmt-iscsi-stg1.musicreports.com 9.0-RELEASE FreeBSD 9.0-RELEASE #3: Tue Dec 27 14:14:29 PST 2011 root@build9x64.pcbsd.org:/usr/obj/builds/amd64/pcbsd-build90/fbsd-source/9.= 0/sys/GENERIC amd64 3] first few lines from /var/run/dmesg.boot: FreeBSD 9.0-RELEASE #3: Tue Dec 27 14:14:29 PST 2011 root@build9x64.pcbsd.org:/usr/obj/builds/amd64/pcbsd-build90/fbsd-sourc= e/9.0/sys/GENERIC amd64 CPU: Intel(R) Xeon(R) CPU E7- 8837 @ 2.67GHz (2666.82-MHz K8-class CPU) Origin =3D "GenuineIntel" Id =3D 0x206f2 Family =3D 6 Model =3D 2f St= epping =3D 2 Features=3D0xbfebfbff Features2=3D0x29ee3ff AMD Features=3D0x2c100800 AMD Features2=3D0x1 TSC: P-state invariant, performance statistics real memory =3D 549755813888 (524288 MB) avail memory =3D 530339893248 (505771 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 64 CPUs FreeBSD/SMP: 8 package(s) x 8 core(s) 4] relevant sysctl's with manual tuning: kern.maxusers: 384 kern.maxvnodes: 8222162 vfs.numvnodes: 675740 vfs.freevnodes: 417524 kern.ipc.somaxconn: 128 kern.openfiles: 5238 vfs.zfs.arc_max: 428422987776 vfs.zfs.arc_min: 53552873472 vfs.zfs.arc_meta_used: 3167391088 vfs.zfs.arc_meta_limit: 107105746944 vm.kmem_size_max: 429496729600 =3D=3D>> manually tuned vm.kmem_size: 429496729600 =3D=3D>> manually tuned vm.kmem_map_free: 107374727168 vm.kmem_map_size: 144625156096 vfs.wantfreevnodes: 2055540 kern.minvnodes: 2055540 kern.maxfiles: 197248 =3D=3D>> manually tuned vm.vmtotal: System wide totals computed every five seconds: (values in kilobytes) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Processes: (RUNQ: 1 Disk Wait: 1 Page Wait: 0 Sleep: 150) Virtual Memory: (Total: 1086325716K Active: 12377876K) Real Memory: (Total: 144143408K Active: 803432K) Shared Virtual Memory: (Total: 81384K Active: 37560K) Shared Real Memory: (Total: 32224K Active: 27548K) Free Memory Pages: 365565564K hw.availpages: 134170294 hw.physmem: 549561524224 hw.usermem: 391395241984 hw.realmem: 551836188672 vm.kmem_size_scale: 1 kern.ipc.nmbclusters: 2560000 =3D=3D>> manually tuned kern.ipc.maxsockbuf: 2097152 net.inet.tcp.sendbuf_max: 2097152 net.inet.tcp.recvbuf_max: 2097152 kern.maxfilesperproc: 18000 net.inet.ip.intr_queue_maxlen: 256 kern.maxswzone: 33554432 kern.ipc.shmmax: 10737418240 =3D=3D>> manually tuned kern.ipc.shmall: 2621440 =3D=3D>> manually tuned vfs.zfs.write_limit_override: 0 vfs.zfs.prefetch_disable: 0 hw.pagesize: 4096 hw.availpages: 134170294 kern.ipc.maxpipekva: 8586895360 kern.ipc.shm_use_phys: 1 =3D=3D>> manually tuned vfs.vmiodirenable: 1 debug.numcache: 632148 vfs.ncsizefactor: 2 vm.kvm_size: 549755809792 vm.kvm_free: 54456741888 kern.ipc.semmni: 256 kern.ipc.semmns: 512 kern.ipc.semmnu: 256 Thanks!