Date: Fri, 29 Oct 1999 22:50:58 +1000 (EST) From: Bruce Evans <bde@zeta.org.au> To: peter.jeremy@alcatel.com.au Cc: freebsd-current@FreeBSD.ORG Subject: Re: Deadlock in nbufkv Message-ID: <Pine.BSF.4.10.9910292208100.6950-100000@alphplex.bde.org> In-Reply-To: <99Oct25.071534est.40323@border.alcanet.com.au>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, 25 Oct 1999, Peter Jeremy wrote: > I've recently upgraded a system from 3.2-RELEASE to -CURRENT as of > 30-Sept (just before the signal changes). I now find that when > I try to do a CVS checkout, the system hangs, with cvs in `nbufkv'. > > The CVSROOT is on a filesystem with standard 8K/1K blocks. The > target FS is 32k/4k. Both FS are running softupdates. This worked > without problem under 3.2. The kernel config files are basically Filesystems with a block size > BKVASIZE can cause fragmentation of buffer virtual memory, so that there is no space for large buffers allthough there is plently of space for small buffers. 3.x handles the fragmentation more aggressively by discarding lots of potentially useful buffers. -current sometimes waits for fragmentation to be reduced instead, but sometimes seems to deadlock. I think it doesn't always deadlock; it just makes very slow progress. Matt's recent `kvawakeup' changes in vfs_bio.c may have fixed this. Filesystems with different block sizes also cause pessimal behaviour for reallocating buffer virtual memory. Allocation time scales with O(nbuf^2) or worse. For nbuf = 10000, I've observed the allocation time reducing the maximum transfer speed to 4MB/sec on a system that can copy buffers at 300MB/sec. Fragmentation of buffer virtual memory is easy to avoid except on systems with lots of memory (more than about 1GB) by setting BKVASIZE large: diff -c2 kern/vfs_bio.c~ kern/vfs_bio.c *** kern/vfs_bio.c~ Thu Oct 28 19:54:40 1999 --- kern/vfs_bio.c Thu Oct 28 20:17:49 1999 *************** *** 329,332 **** --- 369,373 ---- /* + * XXX out of date: * maxbufspace is currently calculated to be maximally efficient * when the filesystem block size is DFLTBSIZE or DFLTBSIZE*2 *************** *** 341,346 **** * buffer cache operation. */ ! maxbufspace = (nbuf + 8) * DFLTBSIZE; hibufspace = imax(3 * maxbufspace / 4, maxbufspace - MAXBSIZE * 5); /* * Limit the amount of malloc memory since it is wired permanently into --- 382,392 ---- * buffer cache operation. */ ! #if BKVASIZE >= MAXBSIZE ! maxbufspace = nbuf * BKVASIZE; ! #else ! maxbufspace = (nbuf + 8) * BKVASIZE / 2; ! #endif hibufspace = imax(3 * maxbufspace / 4, maxbufspace - MAXBSIZE * 5); + /* * Limit the amount of malloc memory since it is wired permanently into diff -c2 sys/param.h~ sys/param.h *** sys/param.h~ Thu Sep 30 07:04:18 1999 --- sys/param.h Thu Sep 30 07:04:20 1999 *************** *** 147,166 **** /* ! * File system parameters and macros. * ! * The file system is made out of blocks of at most MAXBSIZE units, with ! * smaller units (fragments) only in the last direct block. MAXBSIZE ! * primarily determines the size of buffers in the buffer pool. It may be ! * made larger without any effect on existing file systems; however making ! * it smaller make make some file systems unmountable. Also, MAXBSIZE ! * must be less than MAXPHYS!!! DFLTBSIZE is the average amount of ! * memory allocated by vfs_bio per nbuf. BKVASIZE is the average amount ! * of kernel virtual space allocated per nbuf. BKVASIZE should be >= ! * DFLTBSIZE. If it is significantly bigger than DFLTBSIZE, then ! * kva fragmentation causes fewer performance problems. */ #define MAXBSIZE 65536 ! #define BKVASIZE 8192 ! #define DFLTBSIZE 4096 #define MAXFRAG 8 --- 143,171 ---- /* ! * MAXBSIZE is the maximum size of buffers in the buffer pool. File systems ! * must not request buffers with a larger size. If their block size is ! * larger, then they must split up their blocks. MAXBSIZE should not be ! * reduced without changing all file systems that depend on it being large. ! * Also, MAXBSIZE must be less than MAXPHYS. * ! * BKVASIZE is the amount of kernel VM allocated per buffer. It should be ! * at least as large as the block size of the most heavily used file system, ! * but no larger than MAXBSIZE. Increasing it towards MAXBSIZE reduces ! * kernel VM fragmentation so it may improve performance, but it may waste ! * kernel VM. */ + #define BKVASIZE 65536 #define MAXBSIZE 65536 ! ! /* ! * File system parameters and macros. ! */ ! ! /* ! * Misplaced file system parameter/macro. XXX. ! * ! * The ufs file system is made out of blocks, with smaller units ! * (fragments) only in the last direct block. ! */ #define MAXFRAG 8 Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-current" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.4.10.9910292208100.6950-100000>