Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 6 Nov 1998 00:14:54 -0800 (PST)
From:      Marc Slemko <marcs@znep.com>
To:        Matthew Dillon <dillon@apollo.backplane.com>
Cc:        David Greenman <dg@root.com>, cvs-committers@FreeBSD.ORG
Subject:   Re: sendfile()
Message-ID:  <Pine.BSF.4.03.9811052356190.11835-100000@alive.znep.com>
In-Reply-To: <199811060723.XAA01903@apollo.backplane.com>

next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 5 Nov 1998, Matthew Dillon wrote:

> 
> :sendfile() certainly helped wcarchive, which is a good example of this.
> :Zero copy technology helped IBM push numbers on their quad CPU system
> :until they actually ran out of bandwidth on the bus with a fair amount of

And, I should also note, that when you are running out of bandwidth on
your bus, reducing the number of copies helps there too.

> :...
> 
>     Heh.  Well, I wrote a little program to test the overhead of
>     sendfile() vs write() in the 100% cached case.  The program
>     makes a tcp connection to another program that just sinks the
>     data.
> 
>     Over localhost (with the sink program running on the same
>     host), it's 9x more efficient to use sendfile().  Over
>     a 10BaseT Lan (with the sink program running on the
>     remote host) it's 2x and 2.8x more efficient using
>     sendfile().
> 
>     But sendfile() does not apparently handle O_NONBLOCK
>     descriptors right.  This is very important, I think.

Correct.

>     Dave?  We can't use sendfile() in multi-threaded
>     select() cores without that.  Right now it operates
>     as if O_NONBLOCK hadn't been specified.

The concept of a sendfile() in nonblocking mode takes a bit to get your
head around when you actually want to use it.  A nonblocking write() has
pretty well defined behaviour in what is indicated by successful
completion of the nonblocking write.  

A nonblocking write does the same thing a write would, only it
returns with a partial write if it doesn't all fit into whatever
buffer there is.  Should a nonblocking sendfile() do partial "writes"
as well?  How does that impact the overhead?  For what sized files
do multiple nonblocking sendfile()s take more overhead than just
dedicating a kernel thread to it and exactly what is the tradeoff
in L1/L2 locality between them?

An aio_sendfile is what you really want.  This is one area where NT
has the concept right (ignore the API) with completion ports.

The API of the current sendfile(), however, has what it needs to
let a nonblocking sendfile() be useful (for some definition of useful)
in that it allows multiple threads to use the same descriptor to
send different parts of the same file.  Some other ones (eg. Linux's)
don't.

> 	(send to localhost, sink program running on localhost.  Note
> 	user + system time ... sendfile() is 9x more efficient)

I think there may be various cache interactions that can actually
boost this advantage with a large number of sendfile()s vs. a large
number of write()s.  This is just speculation though, and would be
implementation specific.  I haven't looked at this implementation
yet.

> 	(send to host on 10BaseT lan, sink program running on remote.  
> 	2x more efficient in this case.)
> 
> 	(Using 16K send/recvpipes via route command)

I think looking at a kernel profile in this situation, before and
after, would show some very useful things.


To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe cvs-all" in the body of the message



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.4.03.9811052356190.11835-100000>