Date: Sun, 17 Feb 2013 10:09:59 +1100 (EST) From: Bruce Evans <brde@optusnet.com.au> To: Jilles Tjoelker <jilles@stack.nl> Cc: svn-src-head@FreeBSD.org, svn-src-all@FreeBSD.org, src-committers@FreeBSD.org, Bruce Evans <brde@optusnet.com.au> Subject: Re: svn commit: r246824 - head/lib/libc/stdio Message-ID: <20130217093002.V1114@besplex.bde.org> In-Reply-To: <20130216132623.GA25333@stack.nl> References: <201302151044.r1FAi8rs010160@svn.freebsd.org> <20130216024443.N1029@besplex.bde.org> <20130216132623.GA25333@stack.nl>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, 16 Feb 2013, Jilles Tjoelker wrote: > On Sat, Feb 16, 2013 at 04:08:06AM +1100, Bruce Evans wrote: >> On Fri, 15 Feb 2013, Jilles Tjoelker wrote: > >>> Log: >>> setbuf(3): Remove bugs section about ancient versions of BSD. > >>> Modified: >>> head/lib/libc/stdio/setbuf.3 >>> ... >>> -On >>> -.Bx 4.2 >>> -and >>> -.Bx 4.3 >>> -systems, >>> -.Fn setbuf >>> -always uses a suboptimal buffer size and should be avoided. > >> This also removes the bugs section about setbuf() being unusable. It >> uses a buffer of size BUFSIZ, but that is unusably small. This bug >> is older than the old versions of BSD (setbuf() was broken as designed) >> but still applies. > > So I should put back a BUGS section with the below? > > ] .Fn setbuf > ] always uses a suboptimal buffer size and should be avoided. Yes, but change "always" to "usually".x > The risk is not very high because the lazy thing to do (not calling > setbuf() at all) gives better results than calling setbuf(). > >> BTW, I recently noticed many utilities using too-small stdio buffers: >> - md5(1) and libmd use raw BUFSIZ to get the same slowness as setbuf(). >> They even try to get misaligned buffers by allocating them as char >> arrays on they stack. > > md5(1) calls fread() for BUFSIZ bytes at a time. This causes more memory > copies than necessary but does not affect the stdio buffer size passed > to read(2). I saw 1K read(s) in truss output. I debugged it a bit further. With no options, md5 uses the library MD5FileChunk(). This uses raw read()s of 1K on /etc/passwd, and is broken for almost all non-regular files, including all device files except /dev/null. The brokeness can be worked around using stdin or pipes. E.g., "md5 < /dev/da1" or "cat /dev/da1 | md5". But not /dev/stdin or /dev/tty, since these are device files. This confuses md5 into using stdio, which uses st_blksize, which is broken in -current except for regular files. >> - cmp(1) in the non-regular file case naively believe that stdio chooses >> a good buffer size, and stdio naively believes that stat()'s >> st_blksize is a good buffer size. The latter is still broken in the >> kernel (it was unbroken a couple of years ago for regular files). > > Any way to fix stdio? Not really. Only the kernel or the user can reasonably know the correct size. For disks I try dd with all reasonable power of 2 sizes (or track sizes if the hardware has real tracks) and keep doubling the size until the speed saturates. The kernel knows less, but for disks it internally uses big sizes like 64K and 128K which work OK but are a bit wasteful for general use. >> So cmp [-lx] on disks runs very slowly. cmp's internal algorithms >> are also very slow (starting with using stdio at all), but can keep >> up with disks provided the disks are slowed down like this. The >> non-regular file case is little better. It uses mmap() and a >> different slow internal algorithm (not quite as slow since it doesn't >> use stdio). mmap() is especially suitable for disk files, but cmp >> "knows" that it only works on regular files. cp(1) and install(1) use >> mmap() more reasonably. > > mmap() is generally worse than read()/write() if you only access the > data once. VM manipulations are not free. In the interest of simplicity > it may be good to remove the extra code and only use read()/write(). I agree, but I haven't been able to find a clear win from avoiding mmap() even in cases where it is worse in theory. It requires less copying and the savings from this seem to compensate for VM overheads. Both methods bust caches for large data. Bruce
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130217093002.V1114>