From owner-freebsd-net@FreeBSD.ORG Wed Apr 1 02:01:59 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E608B1B7; Wed, 1 Apr 2015 02:01:58 +0000 (UTC) Received: from mail.strugglingcoder.info (strugglingcoder.info [65.19.130.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C4FE6D22; Wed, 1 Apr 2015 02:01:58 +0000 (UTC) Received: from localhost (unknown [10.1.1.3]) (Authenticated sender: hiren@strugglingcoder.info) by mail.strugglingcoder.info (Postfix) with ESMTPSA id 483EE106CDA; Tue, 31 Mar 2015 19:01:52 -0700 (PDT) Date: Tue, 31 Mar 2015 19:01:52 -0700 From: hiren panchasara To: freebsd-net@freebsd.org Subject: Re: em(4): difference between missed_packets and rx_overrun Message-ID: <20150401020152.GM10892@strugglingcoder.info> References: <20150326200853.GA19536@strugglingcoder.info> <20150327173334.GB39674@strugglingcoder.info> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="kunpHVz1op/+13PW" Content-Disposition: inline In-Reply-To: <20150327173334.GB39674@strugglingcoder.info> User-Agent: Mutt/1.5.23 (2014-03-12) Cc: jfv@freebsd.org, erj@freebsd.org, nitroboost@gmail.com X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Apr 2015 02:01:59 -0000 --kunpHVz1op/+13PW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 03/27/15 at 10:33P, hiren panchasara wrote: > + jfv, erj from Intel. >=20 > On 03/26/15 at 01:08P, hiren panchasara wrote: > > This is what we are seeing on em(4) 82574L chipset running stable/10: > >=20 > > dev.em.0.mac_stats.missed_packets: 1441927 > > dev.em.0.interrupts.rx_overrun: 153 > >=20 > > From the datasheet: > > http://www.intel.com/content/www/us/en/ethernet-controllers/82574l-gbe-= controller-datasheet.html > >=20 > > 10.2.7.4 Missed Packets Count - MPC (0x04010; R) > > Counts the number of missed packets. Packets are missed when the receive > > FIFO has insufficient space to store the incoming packet. This could be > > caused because of too few buffers allocated, or because there is > > insufficient bandwidth on the IO bus. Events setting this counter > > cause RXO, the receiver overrun interrupt, to be set. This register > > does not increment if receives are not enabled. > >=20 > > 10.2.4.1 Interrupt Cause Read Register - ICR (0x000C0; RC/WC) > > RXO Receiver Overrun > > Set on receive data FIFO overrun. Could be caused either because > > there are no available buffers or because PCIe receive bandwidth is > > inadequate. > >=20 > > So, first one is a count and another one is an interrupt. Are these 2 > > related? Both seem to be happen when on card FIFO gets full. We see no > > evidence of RX queue on the host being full based on > > dev.em.0.mac_stats.recv_no_buff. > >=20 > > Many a times we see missed_packets increasing without rx_overrun > > changing. > >=20 > > The spec says there is a 40KB buffer on card which seems to be used by > > both RX and TX? Is is split between them for 20KB each? OR is it > > possible that when we are doing high rate TX, we use up that buffer and > > RX suffers from that? So, I found some seemingly useful things in the spec: 10.2.12 Diagnostic Register Descriptions I tried to read on-chip FIFO head/tail to make a sense of what we have there when we see mpc (missed packet count) increase. In em_update_stats_counters() whenever I see mpc increasing, I go print following registers. fhead =3D E1000_READ_REG(&adapter->hw, E1000_RDFH); ftail =3D E1000_READ_REG(&adapter->hw, E1000_RDFT); fheads =3D E1000_READ_REG(&adapter->hw, E1000_RDFHS); ftails =3D E1000_READ_REG(&adapter->hw, E1000_RDFTS); fcount =3D E1000_READ_REG(&adapter->hw, E1000_RDFPC); But I do not see any pattern here. Some times I see a large differenec between head and tail and still no mpc increase and sometimes mpc++ when head/tail are following eachother closely. Also, fcount E1000_RDFPC which is supposed to be "The number of received packets currently in the RX FIFO." is always 0 OR a moderately low number. That doesn't make sense to me. If FIFO is not full why do we get so many missed packets? cheers, Hiren --kunpHVz1op/+13PW Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQF8BAEBCgBmBQJVG1GPXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNEUyMEZBMUQ4Nzg4RjNGMTdFNjZGMDI4 QjkyNTBFMTU2M0VERkU1AAoJEIuSUOFWPt/lLSYH/27M2X+5Kj3YCgPH5kEWMyts +kO7dOrPPLa1VOR1qC++M1SE3+0N9m7TSSAXe0NLretU1ln+v+2G94xAOoRh7IXl p+07fO/zMgqwsKVjzsahkSC7c8L/EdXgWd5kT5XKtuV6PyONgXUVTcaFFwDgJ74v g/w6CgsqVURKgH+n/uteG77VCZkrKzQ/HIvUMKOnjXw53HmbwdeAF3SmH7W2Hslq EeyYdH3lmxxWjiQW3Yb/JpXP6Ig2JiElsvjrsPLopKkVk9TKhigWQT5RDjOHtTmc HVz0kCRVhXUQbG5DDGlShnfig+TVJPmAmpmGxgkCgHdR38BEY+KX1peGICISZoU= =2bW7 -----END PGP SIGNATURE----- --kunpHVz1op/+13PW--