Date: Thu, 20 Jun 2002 22:07:12 -0500 From: Jonathan Lemon <jlemon@flugsvamp.com> To: Terry Lambert <tlambert2@mindspring.com> Cc: Gary Thorpe <gat7634@hotmail.com>, freebsd-arch@FreeBSD.ORG Subject: Re: multiple threads for interrupts Message-ID: <20020620220712.E70387@prism.flugsvamp.com> In-Reply-To: <3D1293AE.FDEC441D@mindspring.com> References: <F112l4rhYQYx3G5aYY6000252ae@hotmail.com> <3D1293AE.FDEC441D@mindspring.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Jun 20, 2002 at 07:47:10PM -0700, Terry Lambert wrote: > The way I would suggest doing this is to run the protocol processing > up to user space at interrupt time (LRP). This gets rid of NETISR. > > A lot of people complain that this won't allow you to receive as > many packets in a given period of time. They are missing the fact > that this only affect the burst rate until poll saturation occurs, > at which point in time the number of packets that you receive is in > fact clocked by buffer availability, and buffer availability is > clocked by the ability of NETISR to process the packets up to the > user space boundary, and that in turn is clocked by the ability to > process the packets out to the user space programs on the other end > of the sockets. > > What this all boils down to is that you should only permit receive > data interrupts to occur at the rate that you can move the data from > the wire, all the way through the system, to completion. > > The feedback process in the absence of available mbufs is to take > the interrupt, and then replace the contents of the mubuf receive > buffer ring with the new contents. The mbuf's only ever get pushed > up the stack if there is a replacement mbuf allocable from the > system in order to put on the ring in place of the received mbuf. > Effectively, we are talking about receive ring overflow here. > > If you trace the dependency graph on mbuf availability all the > way to user space, you will see that if you are receiving packets > faster than you can process them, then you end up spending all > your time servicing interrupts, and that takes away from your > time to actually push data through. > > Jeff Mogul of DEC Western Research Laboratories described this as > "receiver livelock" back early in the last decade. Yes, I believe anyone who has done work in this area is familiar with that technical report. > Luigi's and Jon Lemon's work only partially mitigates the problem. > Turning off interrupts doesn't deal with the NETISR triggering, > which only occurs when you splx() down from a hardware interrupt > level so that the SWL list is run. Running the packets on partially > up the stack doesn't resolve the problems up to the user/kernel > barrier. So both are only partial solutions. Yes. The patches (which I should really make an effort to find out where I put them; maybe to my left under some strawberry jam) were for -stable, and delivered the packet all the way to the socket buffer in interrupt context. They were more intended as a proof of concept than anything else, although I wouldn't have any qualms about committing them to -stable; the corresponding work has to be done in -current first though. I think that an interesting next step(s) would be to start collapsing some of the stack layering a bit. The design notes that VJ outlined for 4.4BSD (which were never completed) look interesting. -- Jonathan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20020620220712.E70387>