Date: Tue, 28 Mar 2006 13:01:35 -0500 From: Mikhail Teterin <mi+mx@aldan.algebra.com> To: Peter Jeremy <peterjeremy@optushome.com.au> Cc: stable@freebsd.org Subject: Re: Reading via mmap stinks (Re: weird bugs with mmap-ing via NFS) Message-ID: <200603281301.36087.mi%2Bmx@aldan.algebra.com> In-Reply-To: <20060328102721.GA2352@turion.vk2pj.dyndns.org> References: <200603232352.k2NNqPS8018729@gate.bitblocks.com> <20060325103927.GE703@turion.vk2pj.dyndns.org> <20060328102721.GA2352@turion.vk2pj.dyndns.org>
next in thread | previous in thread | raw e-mail | index | archive | help
в╕второк 28 березень 2006 05:27, Peter Jeremy написав: > I'd suggest that each mmap be capable of storing several hundred msec of > data as a minumum (maybe 10MB input and 5MB output, preferably more). > Synchronisation can be done by writing tokens into pipes shared with the > mmap's, optimised by sharing read/write pointers (so you only really need > the tokens when the shared buffer is full/empty). Thank you very much, Peter, for your suggestions. Unfortunately, I have no control whatsoever over the dump-ing part of the process. The dump is done by Sybase database servers -- old, clunky, and closed-source software, running on slow CPU (but good I/O) Sun hardware. You are right, of course, that my application (mzip being only part of it) needs to keep the dumper and the compressor in sync. Without any cooperation from the former, however, I see no other way but to temporarily throttle the NFS-bandwidth via firewall, when the compressor falls behind (as can be detected by the increased proportion of sys-time, I guess). Much as I apreciate the (past and future) help and suggestions, I'm not asking you, nor the mailing list to solve my particular problem here :-) I only gave the details of my need and application to illustrate a missed general optimization opportunity in FreeBSD -- reading large files via mmap need not be slower than via read. If anything, it should be (slightly) faster. After many days Matt has finally stated (admitted? ;-): read() uses a different heuristic then mmap() to implement the read-ahead. There is also code in there which depresses the page priority of 'old' already-read pages in the sequential case. There is no reason not to implement similar smarts in the mmap-handling code to similarly depress the priority of the in-memory pages in the MADV_SEQUENTIAL case, thus freeing more RAM for aggressive read-ahead. As I admitted before, actually implementing this far exceeds my own capabilities, so all I can do is pester, whoever cares, to do it instead :-) C'mon, guys... -mi
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200603281301.36087.mi%2Bmx>