Date: Wed, 18 Apr 2012 09:27:18 -0700 From: Sean Bruno <seanbru@yahoo-inc.com> To: Luigi Rizzo <rizzo@iet.unipi.it> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, Jack Vogel <jfvogel@gmail.com> Subject: Re: igb(4) Raising IGB_MAX_TXD ?? Message-ID: <1334766438.3466.4.camel@powernoodle-l7.corp.yahoo.com> In-Reply-To: <20120418072818.GA58850@onelab2.iet.unipi.it> References: <1334705064.4486.23.camel@powernoodle-l7.corp.yahoo.com> <20120418072818.GA58850@onelab2.iet.unipi.it>
next in thread | previous in thread | raw e-mail | index | archive | help
--=-fZwHBsYnQWqPnYlo+ahK Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, 2012-04-18 at 00:28 -0700, Luigi Rizzo wrote: > On Tue, Apr 17, 2012 at 04:24:24PM -0700, Sean Bruno wrote: > > We're running a service with a 82576 configured with 4 queues and a > > maxed rxd/txd configuration: > >=20 > > http://people.freebsd.org/~sbruno/igb_stats.txt >=20 > these stats show that over half of your incoming traffic is > made of small packets (65..127 bytes) but especially, that > the "missed packets" count is very small (18k out of 40G packets) > none of them is reported as "no_desc_avail", and only 76 are > "recv_no_buffer". >=20 > Are you dropping packets in the ip interrupt handler by chance ? > what are your settings there ? >=20 nope, doesn't look like it. =20 http://people.freebsd.org/~sbruno/igb_ip_stats.txt > BTW it seems that there is only one global setting for the dispatch > policy, but for instance there are two netisr_dispatch() calls > in the incoming path, one for layer2 and one for layer3. > The former has relatively little work to do and so it might > make sense to have direct dispatch, the other can be expensive > so i wonder if it wouldn't be better to use deferred dispatch. > If not, perhaps you might try to reduce the rx_processing_limit > to bring down the load on the intr thread. I don't really see any issue with horsepower on this host at the moment with 4 queues. I mean it looks a little something like this under high load: http://people.freebsd.org/~sbruno/igb_top.txt I guess my question still stands though, since the ethernet controller is reporting that it doesn't have any more descriptors available is the hardcoded 4k max descriptors a limit that an be raised? > With your numbers i doubt that raising the queue size helps. Indeed, you're probably right and this is more than likely an application problem that will have to be resolved. However, I'm still curious if the MAX_RXD/TXD is really 4k or if the documentation is correct and we can raise it to 32k for testing? Sean --=-fZwHBsYnQWqPnYlo+ahK Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iQEcBAABAgAGBQJPjutmAAoJEL2UHwafTLtOqVAH/Re6EjwbReJdEXq2O9ZYM3LE kgWQH4XIYfG+UvlIYRjaOxL/8vydd8iFYQIdCUh2AZseZ19uJJveUSD0bp42V/9H mFR7Z6OOYUxWwI15WjoVh2I/q2J+qP3dX0VGHt5kvJLUcBm+Ys9asqHu54RBZeNB UtacUALDgKOO4nasTBZCyEYOl2R3czmN9BVpI8TNuhbekksXcJIJxERsfHJ71KD4 EnUi24/TVJAjUcp4lU/ECEhmXJmB0XRJ45AigyiM1Iii+N09WLss+yfQLuq2c8bA QMA6SKEORK2ymyua7tN/n/BsVOdpnYscyItzXB6DvvH3MWrVRMij6bqpf7Bhop4= =G5jY -----END PGP SIGNATURE----- --=-fZwHBsYnQWqPnYlo+ahK--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1334766438.3466.4.camel>