Date: Sat, 19 Jan 2013 07:47:30 -0800 (PST) From: Barney Cordoba <barney_cordoba@yahoo.com> To: freebsd-net@freebsd.org, John Baldwin <jhb@freebsd.org> Cc: Adrian Chadd <adrian@freebsd.org>, Luigi Rizzo <rizzo@iet.unipi.it> Subject: Re: two problems in dev/e1000/if_lem.c::lem_handle_rxtx() Message-ID: <1358610450.75691.YahooMailClassic@web121604.mail.ne1.yahoo.com> In-Reply-To: <201301181149.42277.jhb@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
=0A=0A--- On Fri, 1/18/13, John Baldwin <jhb@freebsd.org> wrote:=0A=0A> Fro= m: John Baldwin <jhb@freebsd.org>=0A> Subject: Re: two problems in dev/e100= 0/if_lem.c::lem_handle_rxtx()=0A> To: freebsd-net@freebsd.org=0A> Cc: "Barn= ey Cordoba" <barney_cordoba@yahoo.com>, "Adrian Chadd" <adrian@freebsd.org>= , "Luigi Rizzo" <rizzo@iet.unipi.it>=0A> Date: Friday, January 18, 2013, 11= :49 AM=0A> On Friday, January 18, 2013 9:30:40=0A> am Barney Cordoba wrote:= =0A> > =0A> > --- On Thu, 1/17/13, Adrian Chadd <adrian@freebsd.org>=0A> wr= ote:=0A> > =0A> > > From: Adrian Chadd <adrian@freebsd.org>=0A> > > Subject= : Re: two problems in=0A> dev/e1000/if_lem.c::lem_handle_rxtx()=0A> > > To:= "Barney Cordoba" <barney_cordoba@yahoo.com>=0A> > > Cc: "Luigi Rizzo" <riz= zo@iet.unipi.it>,=0A> freebsd-net@freebsd.org=0A> > > Date: Thursday, Janua= ry 17, 2013, 11:48 AM=0A> > > There's also the subtle race=0A> > > conditio= n in TX and RX handling that=0A> > > re-queuing the taskqueue gets around.= =0A> > > =0A> > > Which is:=0A> > > =0A> > > * The hardware is constantly r= eceiving frames ,=0A> right until=0A> > > you blow=0A> > > the FIFO away by= filling it up;=0A> > > * The RX thread receives a bunch of frames;=0A> > >= * .. and processes them;=0A> > > * .. once it's done processing, the hardw= are may=0A> have read=0A> > > some more=0A> > > frames in the meantime;=0A>= > > * .. and the hardware may have generated a=0A> mitigated=0A> > > inter= rupt which=0A> > > you're ignoring, since you're processing frames;=0A> > >= * So if your architecture isn't 100% paranoid, you=0A> may end=0A> > > up = having=0A> > > to wait for the next interrupt to handle what's=0A> currentl= y in=0A> > > the=0A> > > queue.=0A> > > =0A> > > Now if things are done cor= rect:=0A> > > =0A> > > * The hardware generates a mitigated interrupt=0A> >= > * The mask register has that bit disabled, so you=0A> don't end=0A> > > = up receiving it;=0A> > > * You finish your RX queue processing, and there's= =0A> more=0A> > > stuff that's=0A> > > appeared in the FIFO (hence why the = hardware has=0A> generated=0A> > > another=0A> > > mitigated interrupt);=0A= > > > * You unmask the interrupt;=0A> > > * .. and the hardware immediately= sends you the=0A> MSI or=0A> > > signals an interrupt;=0A> > > * .. thus y= ou re-enter the RX processing thread=0A> almost(!)=0A> > > immediately.=0A>= > > =0A> > > However as the poster(s) have said, the interrupt=0A> > > mas= k/unmask in the=0A> > > intel driver(s) may not be 100% correct, so you're= =0A> going to=0A> > > end up=0A> > > with situations where interrupts are m= issed.=0A> > > =0A> > > The reason why this wasn't a big deal in the=0A> de= ep/distant=0A> > > past is=0A> > > because we didn't used to have kernel pr= eemption,=0A> or=0A> > > multiple kernel=0A> > > threads running, or an ove= rly aggressive scheduler=0A> trying=0A> > > to=0A> > > parallelise things a= s much as possible. A lot of=0A> > > net80211/ath bugs=0A> > > have popped = out of the woodwork specifically=0A> because of the=0A> > > above=0A> > > c= hanges to the kernel. They were bugs before, but=0A> people=0A> > > didn't = hit=0A> > > them.=0A> > > =0A> > =0A> > I don't see the distinction between= the rx thread=0A> getting re-scheduled=0A> > "immediately" vs introducing = another thread. In fact=0A> you increase missed=0A> > interrupts by this me= thod. The entire point of=0A> interrupt moderation is=0A> > to tune the int= ervals where a driver is processed.=0A> > =0A> > You might as well just not= have a work limit and=0A> process until your done.=0A> > The idea that "ge= e, I've been taking up too much cpu,=0A> I'd better yield"=0A> > to just qu= eue a task and continue soon after doesn't=0A> make much sense to =0A> > me= .=0A> =0A> If there are multiple threads with the same priority then=0A> ba= tching the work up =0A> into chunks allows the scheduler to round-robin amo= ng=0A> them.=A0 However, when a =0A> task requeues itself that doesn't actu= ally work since the=0A> taskqueue thread =0A> will see the requeued task be= fore it yields the CPU.=A0=0A> Alternatively, if you =0A> force all the rel= evant interrupt handlers to use the same=0A> thread pool and =0A> instead o= f requeueing a separate task you requeue your=0A> handler in the ithread = =0A> pool then you can get the desired round-robin=0A> behavior.=A0 (I have= changes to =0A> the ithread stuff that get us part of the way there in tha= t=0A> handlers can =0A> reschedule themselves and much of the plumbing is i= n place=0A> for shared thread =0A> pools among different interrupts.)=0A=0A= I dont see any "round robin" effect here. You have:=0A=0ARepeat:=0A- Proces= s 100 frames=0Aif (more)=0A- Queue a Task=0A=0Athere's only 1 task at a tim= e. All its really doing is yielding and=0Arescheduling itself to resume the= loop.=0A=0ABC
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1358610450.75691.YahooMailClassic>