From owner-freebsd-net@freebsd.org Thu Jun 2 21:41:05 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6A693B6786E for ; Thu, 2 Jun 2016 21:41:05 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mail.strugglingcoder.info (strugglingcoder.info [104.236.146.68]) by mx1.freebsd.org (Postfix) with ESMTP id 489861B55; Thu, 2 Jun 2016 21:41:04 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from localhost (unknown [10.1.1.3]) (Authenticated sender: hiren@strugglingcoder.info) by mail.strugglingcoder.info (Postfix) with ESMTPA id 517D7177C3; Thu, 2 Jun 2016 14:41:04 -0700 (PDT) Date: Thu, 2 Jun 2016 14:41:04 -0700 From: hiren panchasara To: Sreekanth Rupavatharam Cc: "freebsd-net@freebsd.org" , "jfvogel@gmail.com" , sbruno@FreeBSD.org, erj@FreeBSD.org Subject: Re: Possible transmit/stats problem in igb driver. Message-ID: <20160602214104.GJ8994@strugglingcoder.info> References: <20160602202015.GG8994@strugglingcoder.info> <9A903EE5-3F2C-46C0-B563-1150F81E3507@juniper.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="DesjdUuHQDwS2t4N" Content-Disposition: inline In-Reply-To: <9A903EE5-3F2C-46C0-B563-1150F81E3507@juniper.net> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jun 2016 21:41:05 -0000 --DesjdUuHQDwS2t4N Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable + Sean, Eric On 06/02/16 at 09:11P, Sreekanth Rupavatharam wrote: > Inline >=20 > >Apart from stats, do you see anything else going wrong? i.e. do you > >actually see less packets (arp replies??) than expected? >=20 > [SR] The packets are not going out on the wire. The tool doesn?t receive = the packets. That?s how I started noticing the issue.=20 >=20 > >Taking your example, tx_packets is something we count in the drivers and > >total_pkts_txd is calculated in the card and we just read it off of it > >to report (E1000_TPT). >=20 > [SR] Correct. My main question would be under what circumstance would the= packet handed off to hardware will *not* be transmitted?. Especially consi= dering there are no transmit errors or pause frames received. There are no = dma tx failures either. That?s the baffling part. I tried another exercise = where I used ping of various sizes going out, but that doesn?t seem to trig= ger the problem. >=20 >=20 > >To understand your setup better, ixia is the sender and your box with > >igb(4) is the receiver and your are sending arp requests to it. >=20 > Yes, correct.=20 >=20 > >Can you post following for working (size <=3D 64bytes) and non-working > >(size > 64bytes) cases for before/after? > > > >sysctl dev.igb | grep tx_packets > >sysctl dev.igb | grep total_pkts_txd > >sysctl dev.igb | grep rx_packets > >sysctl dev.igb | grep total_pkts_recvd >=20 >=20 > Before(not working):=20 > dev.igb.1.queue0.tx_packets: 24907933 > dev.igb.1.queue0.rx_packets: 18086575 > dev.igb.1.mac_stats.total_pkts_recvd: 25057359 > dev.igb.1.mac_stats.total_pkts_txd: 16647169 >=20 > After(not working): > dev.igb.1.queue0.tx_packets: 24913324 > dev.igb.1.queue0.rx_packets: 18091832 > dev.igb.1.mac_stats.total_pkts_recvd: 25062618 > dev.igb.1.mac_stats.total_pkts_txd: 16647545 > >netstat -sp arp >=20 > The difference is 5391 for queue0.tx_packets but for mac_stats.total_pkt= s_txd is 376 > Everything else is matching up.=20 >=20 > Before (working) > dev.igb.1.queue0.tx_packets: 25359165 > dev.igb.1.queue0.rx_packets: 18526094 > dev.igb.1.mac_stats.total_pkts_recvd: 25508763 > dev.igb.1.mac_stats.total_pkts_txd: 16831587 >=20 >=20 > After(working) > dev.igb.1.queue0.tx_packets: 25364597 > dev.igb.1.queue0.rx_packets: 18531398 > dev.igb.1.mac_stats.total_pkts_recvd: 25514009 > dev.igb.1.mac_stats.total_pkts_txd: 16836833 >=20 >=20 > Another interesting stat is=20 > before_notworking:dev.igb.1.interrupts.tx_queue_empty: 16646890 > after_notworking:dev.igb.1.interrupts.tx_queue_empty: 16647266 >=20 > The difference here is exactly 376 which is the number of packets that th= e device actually claims to have transmitted. It?s as though it didn?t see = the other packets en-queued in the ring descriptor. >=20 Very interesting. Do you tune defaults at all? What does sysctl hw.igb say? Not sure if bumping up txd would help. Adding Sean and Eric to throw some light. >=20 > I can?t do netstat just for arp as these are coming in a tunnel(Packets d= on?t? show up as arp on the interface). However, I did see the packet rate = was about 500 packets/sec > Cheers, Hiren --DesjdUuHQDwS2t4N Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAABCgBmBQJXUKfrXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNEUyMEZBMUQ4Nzg4RjNGMTdFNjZGMDI4 QjkyNTBFMTU2M0VERkU1AAoJEIuSUOFWPt/ldLEIAKglyJHw+IRINg021FAF+3QN t5gjEuMkM4I3C/4VK25+BTouaOX3BZI/c6XGCL+uXSC+8HVKLZir7MQ0RRL6lq9H 3DGEP/qUZ7i3VOTA9IHXCc3HaLhuQ4Zs4x8sNw2YNhI29U6SS+IXSs+TiScisuHe ErfBLD7QEI9mq+JJoYLZdsHUosjVB7F/x9pJEddp0q63et2E5Hidu4xocW7JMIyI cvYYK0qEHn6k5hNNr9EjC0o2nvb3V5Z1q5pVdZppsmuSROtxU9FgtuT5LqMXuMMj FNGxPT6hIN8n7c2YXkjbfGpuVT7z1nT+80mQjiwSiykPYGL50IrKVKFZXuesNdU= =bO6U -----END PGP SIGNATURE----- --DesjdUuHQDwS2t4N--