From owner-freebsd-hackers@FreeBSD.ORG Sat Oct 18 22:05:23 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 18B96106568C for ; Sat, 18 Oct 2008 22:05:23 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 907E18FC22 for ; Sat, 18 Oct 2008 22:05:22 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1KrJvE-0004Wr-Vz for freebsd-hackers@freebsd.org; Sat, 18 Oct 2008 22:05:21 +0000 Received: from 89-172-60-70.adsl.net.t-com.hr ([89.172.60.70]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 18 Oct 2008 22:05:20 +0000 Received: from ivoras by 89-172-60-70.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 18 Oct 2008 22:05:20 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Ivan Voras Date: Sun, 19 Oct 2008 00:04:51 +0200 Lines: 53 Message-ID: References: <20081018213502.GL99270@dan.emsphone.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig0322905A6EC536F1489FB784" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 89-172-60-70.adsl.net.t-com.hr User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) In-Reply-To: <20081018213502.GL99270@dan.emsphone.com> X-Enigmail-Version: 0.95.7 Sender: news Subject: Re: Pipes, cat buffer size X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Oct 2008 22:05:23 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig0322905A6EC536F1489FB784 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Dan Nelson wrote: > In the last episode (Oct 18), Ivan Voras said: >> I'm working on a program that's intended to be used as a "filter", as >> in "something | myprogram > file". I'm trying it with cat and I'm >> seeing my read()s return small blocks, 64 kB in size. I suppose this >> is because cat writes in 64 kB blocks. So: >> >> a) Is there a way to programatically, per-process, set the pipe buffer= >> size? The program in question is a compressor and it's particularly >> inefficient when given small blocks and I'm wondering if the system ca= n >> buffer enough data for it. >=20 > Why not keep reading until you reach your desired compression block > size? Bzip2's default blocksize is 900k, for example. Of course. But that's not the point :) From what I see (didn't look at the code), Linux for example does some kind of internal buffering that decouples how the reader and the writer interact. I think that with FreeBSD's current behaviour the writer could write 1-byte buffers and the reader will be forced to read each byte individually. I don't know if there's some ulterior reason for this. >> b) Is there any objection to the following patch to cat: >=20 > It might be simpler to just use "dd if=3Dmyfile obs=3D1m" instead of > patching cat. I believe patching cat to bring its block size into the century of the fruitbat has its own benefits. --------------enig0322905A6EC536F1489FB784 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkj6XYkACgkQldnAQVacBchVhACeLs1vVbYU6sYy/6yKGAyvEAV7 WFwAn2rzKZxmt1rRJIjJcMI9KHFQoVaT =ncjE -----END PGP SIGNATURE----- --------------enig0322905A6EC536F1489FB784--