From owner-freebsd-hackers@FreeBSD.ORG  Sat Oct 18 21:35:08 2008
Return-Path: <owner-freebsd-hackers@FreeBSD.ORG>
Delivered-To: freebsd-hackers@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 6B3B6106568D
	for <freebsd-hackers@freebsd.org>; Sat, 18 Oct 2008 21:35:08 +0000 (UTC)
	(envelope-from dan@dan.emsphone.com)
Received: from dan.emsphone.com (dan.emsphone.com [199.67.51.101])
	by mx1.freebsd.org (Postfix) with ESMTP id 2F3B28FC15
	for <freebsd-hackers@freebsd.org>; Sat, 18 Oct 2008 21:35:07 +0000 (UTC)
	(envelope-from dan@dan.emsphone.com)
Received: from dan.emsphone.com (smmsp@localhost [127.0.0.1])
	by dan.emsphone.com (8.14.3/8.14.3) with ESMTP id m9ILZ6BH064611
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <freebsd-hackers@freebsd.org>; Sat, 18 Oct 2008 16:35:06 -0500 (CDT)
	(envelope-from dan@dan.emsphone.com)
Received: (from dan@localhost)
	by dan.emsphone.com (8.14.3/8.14.3/Submit) id m9ILZ2CB064609;
	Sat, 18 Oct 2008 16:35:02 -0500 (CDT) (envelope-from dan)
Date: Sat, 18 Oct 2008 16:35:02 -0500
From: Dan Nelson <dnelson@allantgroup.com>
To: Ivan Voras <ivoras@freebsd.org>
Message-ID: <20081018213502.GL99270@dan.emsphone.com>
References: <gddjoj$apg$1@ger.gmane.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <gddjoj$apg$1@ger.gmane.org>
X-OS: FreeBSD 7.1-PRERELEASE
User-Agent: Mutt/1.5.18 (2008-05-17)
Cc: freebsd-hackers@freebsd.org
Subject: Re: Pipes, cat buffer size
X-BeenThere: freebsd-hackers@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Technical Discussions relating to FreeBSD
	<freebsd-hackers.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>, 
	<mailto:freebsd-hackers-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-hackers>
List-Post: <mailto:freebsd-hackers@freebsd.org>
List-Help: <mailto:freebsd-hackers-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>,
	<mailto:freebsd-hackers-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sat, 18 Oct 2008 21:35:08 -0000

In the last episode (Oct 18), Ivan Voras said:
> I'm working on a program that's intended to be used as a "filter", as
> in "something | myprogram > file". I'm trying it with cat and I'm
> seeing my read()s return small blocks, 64 kB in size. I suppose this
> is because cat writes in 64 kB blocks. So:
> 
> a) Is there a way to programatically, per-process, set the pipe buffer
> size? The program in question is a compressor and it's particularly
> inefficient when given small blocks and I'm wondering if the system can
> buffer enough data for it.

Why not keep reading until you reach your desired compression block
size?  Bzip2's default blocksize is 900k, for example.

> b) Is there any objection to the following patch to cat:

It might be simpler to just use "dd if=myfile obs=1m" instead of
patching cat.

-- 
	Dan Nelson
	dnelson@allantgroup.com