Date: Wed, 19 Aug 2009 11:29:30 -0700 (PDT) From: Barney Cordoba <barney_cordoba@yahoo.com> To: Manish Vachharajani <manishv@lineratesystems.com> Cc: freebsd-net@freebsd.org Subject: Re: Dropped vs. missed packets in the ixgbe driver Message-ID: <822688.18516.qm@web63903.mail.re1.yahoo.com> In-Reply-To: <5bc218350908181535o7c5275dfn2f6647454cfac804@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
=0A=0A--- On Tue, 8/18/09, Manish Vachharajani <manishv@lineratesystems.com= > wrote:=0A=0A> From: Manish Vachharajani <manishv@lineratesystems.com>=0A>= Subject: Re: Dropped vs. missed packets in the ixgbe driver=0A> To: "Barne= y Cordoba" <barney_cordoba@yahoo.com>=0A> Cc: freebsd-net@freebsd.org=0A> D= ate: Tuesday, August 18, 2009, 6:35 PM=0A> Indeed the debugging info is als= o=0A> interesting.=A0 However, I'd like to=0A> get some data from netstat w= hen the driver drops=0A> frames.=A0 When looking=0A> at the bge driver sour= ce, it appears that all input drops=0A> at the NIC=0A> are reported as inpu= t errors.=A0 It appears that the=0A> intel drivers=0A> (e1000, ixgb, and ix= gbe) don't report that number outside=0A> of the debug=0A> and stats printf= s at all, and this seems broken.=A0 What=0A> I want to know=0A> is if I hav= e just missed where these are reported.=0A> =0A> So, in a nutshell, the que= stion is:=A0 should these=0A> drivers be reporting=0A> miss events as input= errors in the ifnet struct as the bge=0A> driver=0A> does, or as drops in = the ifnet struct, was there some=0A> conscious=0A> decision not to report m= iss events anywhere outside the=0A> debug and=0A> stats info, or am I just = being silly and not seeing where=0A> the numbers=0A> are reported?=0A> =0A>= Also, don't worry on the performance front, we are also=0A> looking at the= =0A> driver in FreeBSD 8.0 :) which supports RSS to help=0A> performance=0A= > scaling,=A0 though we have some interesting data there=0A> that I'll post= =0A> about once I confirm that the numbers are indeed correct=0A> and not a= =0A> tuning or setup problem.=0A> =0A> Manish=0A> =0A> > --- On Mon, 8/17/0= 9, Manish Vachharajani <manishv@lineratesystems.com>=0A> wrote:=0A> >=0A> >= > From: Manish Vachharajani <manishv@lineratesystems.com>=0A> >> Subject: D= ropped vs. missed packets in the ixgbe=0A> driver=0A> >> To: freebsd-net@fr= eebsd.org=0A> >> Date: Monday, August 17, 2009, 6:24 PM=0A> >> I've been do= ing some performance=0A> >> testing on freebsd 7.2 and noticed=0A> >> that = the ixgbe driver does not report missed=0A> packets as=0A> >> dropped when= =0A> >> queried via netstat -id (the ixgbe driver in the=0A> 8.0=0A> >> bet= a has a=0A> >> similar issue).=A0 A missed packet is a packet that=0A> was= =0A> >> correctly=0A> >> received by the NIC but because it was out of=0A> = descriptors=0A> >> and internal=0A> >> memory, the packet had to be dropped= by the NIC=0A> itself -- a=0A> >> hardware=0A> >> register counts such eve= nts.=A0 Instead the driver=0A> only=0A> >> reports drops=0A> >> that are du= e to the driver itself, i.e., the NIC=0A> DMAed the=0A> >> packet to=0A> >>= memory but the driver had to drop something=0A> because it was=0A> >> out = of=0A> >> mbufs.=A0 Is the miss count reported elsewhere=0A> (besides=0A> >= > via a kernel=0A> >> printf from the driver)?=A0 At the end of the email= =0A> I=0A> >> give a stats dump=0A> >> from the driver and from netstat, no= te the number=0A> of=0A> >> packets dropped=0A> >> is 0 as reported by nets= tat but the Missed field=0A> in the=0A> >> dmesg output=0A> >> shows many m= issed packets.=0A> >>=0A> >> >From my perspective, it is disconcerting to= =0A> see=0A> >> performance=0A> >> degradation on the link, along with TCP = ack=0A> retransmits,=0A> >> packet=0A> >> reordering, etc. (on a point-to-p= oint link with no=0A> switch=0A> >> in between)=0A> >> but then see no drop= s reported by netstat because=0A> the=0A> >> driver didn't=0A> >> drop the = packet, the NIC did.=A0 The fix should be=0A> >> straight-forward and=0A> >= > I'll gladly make a patch assuming that it is=0A> indeed a bug=0A> >> and = not a=0A> >> conscious design choice.=0A> >>=0A> >> Here is the relevant ne= tstat output=0A> >>=0A> >> Name=A0 =A0 Mtu Network=0A> >> =A0=A0=A0Address= =0A> >> =A0 =A0 Ipkts Ierrs=A0 =A0 Opkts=0A> >> Oerrs=A0 Coll Drop=0A> >> i= x0=A0 =A0 1500 <Link#1>=0A> >> 00:30:48:94:60:ec=A0 =A0 =A0 =A0 0=0A> >> = =A0=A0=A00=A0 =A0 =A0 =A0 1=0A> >> =A00=A0 =A0=A0=A00=A0 =A0 0=0A> >> ix0= =A0 =A0 1500 192.168.105.0 192.168.105.2=0A> >> =A0 =A0 =A0 =A0 =A0 0=0A> >= > =A0=A0=A0-=A0 =A0 =A0 =A0 0=0A> >> =A0-=A0 =A0=A0=A0-=A0 =A0 -=0A=0A=0Aif= you look in ixgbe_update_stats_counters at the bottom:=0A=0A ifp->i= f_ierrors =3D missed_rx + adapter->stats.crcerrs +=0A adapte= r->stats.rlec;=0A=0Athe errors are added in. =0A=0ABC=0A> >> 00:30:48:94:60= :ed=A0 =A0 =A0 11M=0A> >> =A0=A0=A00=A0 =A0=A0=A06.1M=0A> >> =A00=A0 =A0=A0= =A00=A0 =A0 0=0A> >> ix1=A0 =A0 1500=0A> >> 192.168.5.0=A0=A0=A0192.168.5.2= =0A> >> =A0 =A0 =A0 10M=A0 =A0=A0=A0-=0A> >> =A0=A0=A06.1M=0A> >> =A0-=A0 = =A0=A0=A0-=A0 =A0 -=0A> >>=0A> >> And here is the dmesg output after doing = a sysctl=0A> >> dev.ix.1.stats=3D1=0A> >>=0A> >> ix1: Std Mbuf Failed =3D 0= =0A> >> ix1: Missed Packets =3D 413872=0A> >> ix1: Receive length errors = =3D 0=0A> >> ix1: Crc errors =3D 0=0A> >> ix1: Driver dropped packets =3D 0= =0A> >> ix1: watchdog timeouts =3D 0=0A> >> ix1: XON Rcvd =3D 616428212235= =0A> >> ix1: XON Xmtd =3D 0=0A> >> ix1: XOFF Rcvd =3D 616428212235=0A> >> i= x1: XOFF Xmtd =3D 0=0A> >> ix1: Total Packets Rcvd =3D 12424533=0A> >> ix1:= Good Packets Rcvd =3D 12010661=0A> >> ix1: Good Packets Xmtd =3D 6419128= =0A> >> ix1: TSO Transmissions =3D 0=0A> >>=0A> >> Manish=0A> >=0A> > the d= ebug sysctl show more interesting info. Don't get=0A> too excited=0A> > abo= ut doing performance testing with that driver. Its=0A> not designed=0A> > t= o be any higher in performance than any of the other=0A> intel drivers.=0A>= >=0A> > Barney=0A> >=0A> >=0A> >=0A> >=0A> =0A> =0A> =0A> -- =0A> Manish V= achharajani=0A> Founder=0A> LineRate Systems=0A> manishv@lineratesystems.co= m=0A> (609)635-9531 M=0A> _______________________________________________= =0A> freebsd-net@freebsd.org=0A> mailing list=0A> http://lists.freebsd.org/= mailman/listinfo/freebsd-net=0A> To unsubscribe, send any mail to "freebsd-= net-unsubscribe@freebsd.org"=0A> =0A=0A=0A
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?822688.18516.qm>