From owner-cvs-all Fri Nov 6 00:19:22 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id AAA02814 for cvs-all-outgoing; Fri, 6 Nov 1998 00:19:22 -0800 (PST) (envelope-from owner-cvs-all@FreeBSD.ORG) Received: from scanner.worldgate.com (scanner.worldgate.com [198.161.84.3]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id AAA02801 for ; Fri, 6 Nov 1998 00:19:19 -0800 (PST) (envelope-from marcs@znep.com) Received: from znep.com (uucp@localhost) by scanner.worldgate.com (8.9.1a/8.9.1) with UUCP id BAA00395; Fri, 6 Nov 1998 01:19:05 -0700 (MST) Received: from localhost (marcs@localhost) by alive.znep.com (8.7.5/8.7.3) with ESMTP id AAA21125; Fri, 6 Nov 1998 00:14:54 -0800 (PST) Date: Fri, 6 Nov 1998 00:14:54 -0800 (PST) From: Marc Slemko To: Matthew Dillon cc: David Greenman , cvs-committers@FreeBSD.ORG Subject: Re: sendfile() In-Reply-To: <199811060723.XAA01903@apollo.backplane.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-cvs-all@FreeBSD.ORG Precedence: bulk On Thu, 5 Nov 1998, Matthew Dillon wrote: > > :sendfile() certainly helped wcarchive, which is a good example of this. > :Zero copy technology helped IBM push numbers on their quad CPU system > :until they actually ran out of bandwidth on the bus with a fair amount of And, I should also note, that when you are running out of bandwidth on your bus, reducing the number of copies helps there too. > :... > > Heh. Well, I wrote a little program to test the overhead of > sendfile() vs write() in the 100% cached case. The program > makes a tcp connection to another program that just sinks the > data. > > Over localhost (with the sink program running on the same > host), it's 9x more efficient to use sendfile(). Over > a 10BaseT Lan (with the sink program running on the > remote host) it's 2x and 2.8x more efficient using > sendfile(). > > But sendfile() does not apparently handle O_NONBLOCK > descriptors right. This is very important, I think. Correct. > Dave? We can't use sendfile() in multi-threaded > select() cores without that. Right now it operates > as if O_NONBLOCK hadn't been specified. The concept of a sendfile() in nonblocking mode takes a bit to get your head around when you actually want to use it. A nonblocking write() has pretty well defined behaviour in what is indicated by successful completion of the nonblocking write. A nonblocking write does the same thing a write would, only it returns with a partial write if it doesn't all fit into whatever buffer there is. Should a nonblocking sendfile() do partial "writes" as well? How does that impact the overhead? For what sized files do multiple nonblocking sendfile()s take more overhead than just dedicating a kernel thread to it and exactly what is the tradeoff in L1/L2 locality between them? An aio_sendfile is what you really want. This is one area where NT has the concept right (ignore the API) with completion ports. The API of the current sendfile(), however, has what it needs to let a nonblocking sendfile() be useful (for some definition of useful) in that it allows multiple threads to use the same descriptor to send different parts of the same file. Some other ones (eg. Linux's) don't. > (send to localhost, sink program running on localhost. Note > user + system time ... sendfile() is 9x more efficient) I think there may be various cache interactions that can actually boost this advantage with a large number of sendfile()s vs. a large number of write()s. This is just speculation though, and would be implementation specific. I haven't looked at this implementation yet. > (send to host on 10BaseT lan, sink program running on remote. > 2x more efficient in this case.) > > (Using 16K send/recvpipes via route command) I think looking at a kernel profile in this situation, before and after, would show some very useful things. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe cvs-all" in the body of the message