From owner-freebsd-stable@FreeBSD.ORG Tue Mar 28 10:27:48 2006 Return-Path: X-Original-To: stable@freebsd.org Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3E24116A424 for ; Tue, 28 Mar 2006 10:27:48 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from mail08.syd.optusnet.com.au (mail08.syd.optusnet.com.au [211.29.132.189]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0C78643D48 for ; Tue, 28 Mar 2006 10:27:46 +0000 (GMT) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (c220-239-19-236.belrs4.nsw.optusnet.com.au [220.239.19.236]) by mail08.syd.optusnet.com.au (8.12.11/8.12.11) with ESMTP id k2SARNHp019070 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Tue, 28 Mar 2006 21:27:24 +1100 Received: from turion.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by turion.vk2pj.dyndns.org (8.13.4/8.13.4) with ESMTP id k2SARMcn002412; Tue, 28 Mar 2006 21:27:23 +1100 (EST) (envelope-from peter@turion.vk2pj.dyndns.org) Received: (from peter@localhost) by turion.vk2pj.dyndns.org (8.13.4/8.13.4/Submit) id k2SARMsQ002411; Tue, 28 Mar 2006 21:27:22 +1100 (EST) (envelope-from peter) Date: Tue, 28 Mar 2006 21:27:21 +1100 From: Peter Jeremy To: Mikhail Teterin Message-ID: <20060328102721.GA2352@turion.vk2pj.dyndns.org> References: <200603232352.k2NNqPS8018729@gate.bitblocks.com> <200603241518.01027.mi+mx@aldan.algebra.com> <20060325103927.GE703@turion.vk2pj.dyndns.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20060325103927.GE703@turion.vk2pj.dyndns.org> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.11 Cc: stable@freebsd.org Subject: Re: Reading via mmap stinks (Re: weird bugs with mmap-ing via NFS) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2006 10:27:48 -0000 On Sat, 2006-Mar-25 21:39:27 +1100, Peter Jeremy wrote: >What happens if you simulate read-ahead yourself? Have your main >program fork and the child access pages slightly ahead of the parent >but do nothing else. I suspect something like this may be the best approach for your application. My suggestion would be to split the backup into 3 processes that share memory. I wrote a program that is designed to buffer data in what looks like a big FIFO and "dump | myfifo | gzip > file.gz" is significantly faster than "dump | gzip > file.gz" so I suspect it will help you as well. Process 1 reads the input file into mmap A. Process 2 {b,gz}ips's mmap A into mmap B. Process 3 writes mmap B into the output file. Process 3 and mmap B may be optional, depending on your target's write performance. mmap A could be the real file with process 1 just accessing pages to force them into RAM. I'd suggest that each mmap be capable of storing several hundred msec of data as a minumum (maybe 10MB input and 5MB output, preferably more). Synchronisation can be done by writing tokens into pipes shared with the mmap's, optimised by sharing read/write pointers (so you only really need the tokens when the shared buffer is full/empty). -- Peter Jeremy