Date: Tue, 27 May 2003 02:48:55 -0700 From: Terry Lambert <tlambert2@mindspring.com> To: Igor Sysoev <is@rambler-co.ru> Cc: arch@freebsd.org Subject: Re: sendfile(2) SF_NOPUSH flag proposal Message-ID: <3ED33487.59C03DA9@mindspring.com> References: <Pine.BSF.4.21.0305271157480.46491-100000@is>
next in thread | previous in thread | raw e-mail | index | archive | help
Igor Sysoev wrote: > > I don't see this as being terrifically useful; small files > > should probably just be mapped and written; the copy expense > > is still there for the headers and trailers, no matter what, > > and the file size itself is very small overhead, relatively > > speaking, for files small enough for this to be an issue. > > FreeBSD 4.x has no zero_copy(9) so mmap()ed files would be copied. > sendfile() allows to avoid this copy. It's actually "one copy", from an external mbuf reference to a VM buffer to the card, vs. two copies, from the mapped file in the process address space to mbufs, to the card. My point, though was that fr files small enough to fit in a packet, you are probably going to spend more on the headers and trailers copyin and the mbuf list assembly and the system call overhead, than you'll end up spending one the extra data copy for the very small file contents. I would be really surprised if you were able to demonstrate a measuarble performance difference which was above the noise. > By the way what do you call by small files ? 4K, 30K or 100K ? You were talking about the file and the header living in the same packet. Actually, according to the literature on the subject, the average web page returned by an HTTP server is ~8K. That would fit in a single jumbogram; but you said 14??b, which makes me believe you're using a standard MTU. Without knowing what protocol you're using, and without any knowledge of if you are using pipelining for server-to-client data, etc. (e.g. so back-to-back sendfiles are a possibility), I really don't know what to call "small". Certainly, I'd say that anything that went over 16K or 32K would take a stall in the socket send buffer, which is limited to around there, if you don't go out of your way to add RAM and increase the mbufs and use the sysctl to make the send buffer bigger. The stall would be a *much* bigger performance issue, IMO, since you will end up waiting for another NETISR (up to 100ms) plus a potential page fault on the file data, plus... etc.. So like I said: I don't think you are going to get a huge performance win out of doing it, no matter how you cook it. > > I also think your headers and trailers are very small, if > > they are fitting with the file contents in a single packet. > > I think this is atypical. > > If I ask Google: > ---- > HEAD /images/hp0.gif HTTP/1.0 > Host: www.google.com > > ---- > it will return me 230 bytes: The "HEAD" is atypical, compared to the "GET"; the full Google front page is larger than that, and consists of multiple files; assuming you support HTTP/1.1 and pipelining, it's going to be a back-to-back transfer involving multiple sendfile() calls. If you don't, then your major expense is going to be protection domain crossing and setting up and tearing down HTTP/1.0 connections, unless you set "Keep-Alive:". Even so, then if you get back something that's chunked, the only way to signal the end of it will be to drop the connection (no MIME size). > > BTW: if you go ahead with this, you should verify that it > > also works for the trailers, etc., and you should probably > > skip it if you headers > transmit queue depth, or file size > > > transmit queue depth, or trailers > transmit queue depth. > > Currently sendfile() can send the file in not full packets even > file is bigger then the transmit queue depth. It can send > it in 2 x 1460 + 1176 or 5 x 1460 + 892 packets. 3 packets vs. 6. And using HTTP/1.0, there's also the three handshake packets, SYN/SYN-ACK/ACK, and the tear-down three teardown packets, FIN/FIN-ACK/ACK (or 4), plus the ACK's for the packets you sent (should be one ACK, since that's below the TCP window size). Really: it's in the noise. Unless you are paying by packet count, you probably shouldn't care. -- Terry
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3ED33487.59C03DA9>