Date: Fri, 27 May 2005 17:49:12 -0400 From: Sven Willenberger <sven@dmv.com> To: Bruce Evans <bde@zeta.org.au> Cc: freebsd-amd64@freebsd.org Subject: Re: BKVASIZE for large block-size filesystems Message-ID: <1117230552.16920.9.camel@lanshark.dmv.com> In-Reply-To: <1117112579.15065.30.camel@lanshark.dmv.com> References: <1117055183.13183.57.camel@lanshark.dmv.com> <20050526090743.S75084@delplex.bde.org> <1117112579.15065.30.camel@lanshark.dmv.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 2005-05-26 at 09:02 -0400, Sven Willenberger wrote: > On Thu, 2005-05-26 at 10:38 +1000, Bruce Evans wrote: > > On Wed, 25 May 2005, Sven Willenberger wrote: > > <<SNIP>> > > > > > My question is are there any caveats about increasing BKVASIZE to 65536? > > > The system has 8G of RAM and I understand that nbufs decreases with > > > increasing BKVASIZE; > > > > The decrease in nbufs is a bug. It defeats half of the point of increasing > > BKVASIZE: if most buffers have size 64K, then increasing BKVASIZE from 16K > > to 64K gives approximately nbuf/4 buffers all of size 64K instead of nbuf > > buffers, with nbuf/4 of them of size 64K and 3*nbuf/4 of them unusable. > > Thus it avoids some resource wastage at a cost of possibly not using enough > > resources for effective caching. However, little is lost if most buffers > > have size 64K. Then the reduced nbuf consumes all of the kva resources that > > we are willing to allocate. The problem is when file systems are mixed and > > ones with a block size of 64K are not used much or at all. The worst case > > is when all blocks have size 512, which can happen for msdosfs. Then up > > to (BKVASIZE - 512) / BKVASIZE of the kva resource is wasted (> 99% for > > BKVASIZE = 65536 but only 97% for BKVASIZE = 16384). > > > > To fix the bug, change BKVASIZE in kern_vfs_bio_buffer_alloc() to 16384 > > and consider adjusting the machbcache tunable (see below). > > > Alas this did not work ... I replaced BKVASIZE in that function (and only that function ) with 16384 and adjusted BKVASIZE to 65536 in sys/param.h. After make buildworld, make buildkernel, make installkernel I rebooted (attempted to into single user mode). However, after the post and the BSD splash screen (where I am prompted to choose single user) the machine gets almost nowhere into the boot process and gets as far as: Copyright (c) 1992-2005 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 5.4-STABLE #1: Sat May 14 17:37:43 EDT 2005 root@myserver:/usr/obj/usr/src/sys/NORACCT Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Opteron(tm) Processor 248 (2190.75-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0xf5a Stepping = 10 Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2> AMD Features=0xe0500800<SYSCALL,NX,MMX+,LM,3DNow+,3DNow> real memory = 8589934592 (8192 MB) avail memory = 8255700992 (7873 MB) then the console completely locks up and there is no further response. Booting kernel.old does work fine and it proceeds to the next section (ACPI APIC table). > Ahh, so this is literal replace the word "BKVASIZE" in that function > with the word "16384". I am assuming that I can leave other instances of > BKVASIZE and BKVAMASK in that file (vfs_bio.c) alone then? > > > > how can I either determine if the resulting nbufs > > > will be sufficient or calculate what is needed based on RAM and system > > > usage? > > > > nbuf is not directly visible except using a debugger, but vfs.maxbufspace > > gives it indirectly -- divide the latter by BKVASIZE to get nbuf. A few > > thousand for it is plenty. > > > > I used to use BKVASIZE = 65536, and fixed the bug as above, and also doubled > > nbuf in kern_vfs_bio_buffer_alloc(), and also configured VM_BCACHE_SIZE_MAX > > to 512M so that the elevated nbuf was actually used, but the need for > > significantly increasing the default nbuf (at least with BKVASIZE = 16384) > > went away many years ago when memory sizes started exceeding 256M or so. > > My doubling of nbuf broke a few years later when memory sizes started > > exceeding 1GB. i386's just don't have enough virtual address space to use > > a really large nbuf, so when there is enough physical memory the default > > nbuf is as large as possible. I was only tuning BKVASIZE and > > VM_BCACHE_SIZE_MAX to benchmark file systems with large block sizes, but > > the performance with large block sizes was poor even with this tuning so > > I lost interest in it. Now I just use the defaults and the bug fix > > reduces to a spelling change. nbuf defeaults to about 7000 on my machines > > with 1GB of memory. This is plenty. With BKVASIZE = 64K and without the > > fix, it would be 1/4 as much, which seems a little low. > > I may have to try some of these other tricks and see what I can do .. first I need to build a regular kernel again so I can have a "safe" kernel.old after trying some further changes. Sven
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1117230552.16920.9.camel>