Date: Wed, 18 Apr 2012 09:28:18 +0200 From: Luigi Rizzo <rizzo@iet.unipi.it> To: sbruno@freebsd.org Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: igb(4) Raising IGB_MAX_TXD ?? Message-ID: <20120418072818.GA58850@onelab2.iet.unipi.it> In-Reply-To: <1334705064.4486.23.camel@powernoodle-l7.corp.yahoo.com> References: <1334705064.4486.23.camel@powernoodle-l7.corp.yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Apr 17, 2012 at 04:24:24PM -0700, Sean Bruno wrote: > We're running a service with a 82576 configured with 4 queues and a > maxed rxd/txd configuration: > > http://people.freebsd.org/~sbruno/igb_stats.txt these stats show that over half of your incoming traffic is made of small packets (65..127 bytes) but especially, that the "missed packets" count is very small (18k out of 40G packets) none of them is reported as "no_desc_avail", and only 76 are "recv_no_buffer". Are you dropping packets in the ip interrupt handler by chance ? what are your settings there ? BTW it seems that there is only one global setting for the dispatch policy, but for instance there are two netisr_dispatch() calls in the incoming path, one for layer2 and one for layer3. The former has relatively little work to do and so it might make sense to have direct dispatch, the other can be expensive so i wonder if it wouldn't be better to use deferred dispatch. If not, perhaps you might try to reduce the rx_processing_limit to bring down the load on the intr thread. > sysctl net | grep intr net.inet.ip.intr_queue_maxlen: 256 net.inet.ip.intr_queue_drops: 253 > sysctl net.isr net.isr.numthreads: 1 net.isr.maxprot: 16 net.isr.defaultqlimit: 256 net.isr.maxqlimit: 10240 net.isr.bindthreads: 0 net.isr.maxthreads: 1 net.isr.direct: 0 net.isr.direct_force: 0 net.isr.dispatch: direct > We still see, under higher load spikes, a tendency to drop packets (I > suspect an application issue at this point, but want to attempt to > alleviate some congestion). > > I note that IGB_MAX_RXD is set to 4k. Reviewing the Intel spec shell on > the 82576 I see that the maximum value for the Descriptor Ring Length > (8.10.8) is 32k descriptors. > > Does that mean I should be able to go higher that 4k? I suspect that > even if I can, that would merely make the traffic fill more buffer > space, but maybe its enough to make it work a bit better. With your numbers i doubt that raising the queue size helps. cheers luigi > Sean > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120418072818.GA58850>