Date: Sat, 25 Mar 2006 09:20:13 -0500 From: Mikhail Teterin <mi+kde@aldan.algebra.com> To: Peter Jeremy <peterjeremy@optushome.com.au> Cc: alc@freebsd.org, Mikhail Teterin <mi+mx@aldan.algebra.com>, stable@freebsd.org Subject: Re: Reading via mmap stinks (Re: weird bugs with mmap-ing via NFS) Message-ID: <200603250920.14208@aldan> In-Reply-To: <20060325103927.GE703@turion.vk2pj.dyndns.org> References: <200603232352.k2NNqPS8018729@gate.bitblocks.com> <200603241518.01027.mi%2Bmx@aldan.algebra.com> <20060325103927.GE703@turion.vk2pj.dyndns.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Saturday 25 March 2006 05:39 am, Peter Jeremy wrote: = On Fri, 2006-Mar-24 15:18:00 -0500, Mikhail Teterin wrote: = >which there is not with the read. Read also requires fairly large = >buffers in the user space to be efficient -- *in addition* to the = >buffers in the kernel. = = I disagree. With a filesystem read, the kernel is solely responsible = for handling physical I/O with an efficient buffer size. The userland = buffers simply amortise the cost of the system call and copyout = overheads. I don't see a disagreement in the above :-) Mmap API can be slightly faster than read -- kernel is still "responsible for handling physical I/O with an efficient buffer size". But instead of copying the data out after reading, it can read it directly into the process' memory. = >I'm also quite certain, that fulfulling my "demands" would add quite a = >bit of complexity to the mmap support in kernel, but hey, that's what the = > kernel is there for :-) = = Unfortunately, your patches to implement this seem to have become detached = from your e-mail. :-) If I manage to *convince* someone, that there is a problem to solve, I'll consider it a good contribution to the project... = mmap can lend itself to cleaner implementatione because there's no = need to have a nested loop to read buffers and then process them. You = can mmap then entire file and process it. The downside is that on a = 32-bit architecture, this limits you to processing files that are = somewhat less than 2GB. First, only one of our architectures is 32-bit :-) On 64-bit systems, the addressable memory (kind of) matches the maximum file size. Second even with the loop reading/processing chunks at a time, the implementation is cleaner, because it does not need to allocate any memory nor try to guess, which buffer size to pick for optimal performance, nor align the buffers on pages (which grep is doing, for example, rather hairily). = The downside is that touching an uncached page triggers a trap which may = not be as efficient as reading a block of data through the filesystem = interface, and I/O errors are delivered via signals (which may not be as = easy to handle). My point exactly. It does seem to be less efficient *at the moment* and I am trying to have the kernel support for this cleaner method of reading *improved*. By convincing someone with a clue to do it, that is... :-) = >Would you care to look at my program instead? Thanks: = > = > http://aldan.algebra.com/mzip.c I'm sorry, that should be http://aldan.algebra.com/~mi/mzip.c -- I checked this time :-( = I tried writing a program that just mmap'd my entire (2GB) test file = and summed all the longwords in it. The files I'm dealing with are database dumps -- 10-80Gb :-) Maybe, that's, what triggers some pessimal case?.. Thanks! Yours, -mi
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200603250920.14208>