From owner-freebsd-net@FreeBSD.ORG Wed Dec 1 17:54:10 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9276B10657AE for ; Wed, 1 Dec 2010 17:54:10 +0000 (UTC) (envelope-from john@traktor.dnepro.net) Received: from smtp-out.dnepro.net (smtp-out.dnepro.net [195.24.131.41]) by mx1.freebsd.org (Postfix) with ESMTP id 281448FC12 for ; Wed, 1 Dec 2010 17:54:09 +0000 (UTC) Received: from traktor.dnepro.net (localhost [127.0.0.1]) by traktor.dnepro.net (8.14.3/8.14.3) with ESMTP id oB1Hs1Vf076372 for ; Wed, 1 Dec 2010 19:54:02 +0200 (EET) (envelope-from john@traktor.dnepro.net) Received: (from john@localhost) by traktor.dnepro.net (8.14.3/8.14.3/Submit) id oB1Hs11T076368 for freebsd-net@freebsd.org; Wed, 1 Dec 2010 19:54:01 +0200 (EET) (envelope-from john) Date: Wed, 1 Dec 2010 19:54:01 +0200 From: Eugene Perevyazko To: freebsd-net@freebsd.org Message-ID: <20101201175401.GA69269@traktor.dnepro.net> Mail-Followup-To: freebsd-net@freebsd.org References: <20101110110428.GA3505@traktor.dnepro.net> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="k+w/mQv8wyuph6w0" Content-Disposition: inline In-Reply-To: <20101110110428.GA3505@traktor.dnepro.net> User-Agent: Mutt/1.4.2.3i Subject: Re: igb dual-port adapter 1200Mbps limit - what to tune? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Dec 2010 17:54:10 -0000 --k+w/mQv8wyuph6w0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Wed, Nov 10, 2010 at 01:04:28PM +0200, Eugene Perevyazko wrote: > > I have a router running RELENG_7 with two dual-port igbs - igb0 and igb1 are on > 82575 on intel s5520ur mb and igb2 and igb3 are on 82576 on ET dual-port card. > 82576 is in 8x slot. > Main traffic flows from igb0+igb1 to igb2+igb3, less traffic goes back. > There's no traffic flow in directions igb0 - igb1 and igb2 - igb3. > > There are vlans on all interfaces. > > igb0 and igb1 are outbound links. > igb2 and igb3 are connected to switch. > CPU is E5620@2.4GHz, 8 cores, irqs bound to different cores skipping HT ones. > Tried 2 queues and 1 queue per iface, neither hitting cpu limit. > > The problem is that traffic through igb2+igb3 is limited at around 1200Mbps Tx > while I was hoping for 1600-1800Mbps Tx. > I'd like to say that now this host is forwarding 1710 Mb/s @ 189 kpps in one direction + 411 Mb/s @ 140 kpps in reverse direction (30 minutes average). The following changes were made: - no lagg in use - only one vlan left instead of 5 vlans - added motherboard module 'HP NC360T PCIe DP Gigabit Server Adapter (n1e5132)' with 2 em interfaces (only one used) - 3 interfaces are connected to hosts with patchcords, not through switch - hyperthreading turned off in bios - igb driver patched to reduce irq rate and make it tunable (patch follows) - hw.igb.num_queues=4, dev.igb.*.enable_aim=0, dev.igb.*.default_intrrate=4000 the last sysctl is added in patch) - igb queues manually repinned to different cores - ipfw rules minimized igb driver used is 1.9.6/RELENG_7, igb.c version 1.3.2.12 Can't say for sure whether there is a key change or is it all changes summed up, sorry. -- Eugene Perevyazko --k+w/mQv8wyuph6w0 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="igb.intrrate.patch" --- if_igb.c.old 2010-09-27 21:34:04.000000000 +0300 +++ if_igb.c 2010-11-16 14:31:11.000000000 +0200 @@ -293,6 +293,17 @@ static int igb_enable_aim = TRUE; TUNABLE_INT("hw.igb.enable_aim", &igb_enable_aim); /* +** default interrupt rate in ints/sec +** applied to each queue separately +** used if enable_aim=0 +*/ +static int igb_default_intrrate = IGB_INTS_PER_SEC; +TUNABLE_INT("hw.igb.default_intrrate", &igb_default_intrrate); +#undef IGB_DEFAULT_ITR +/*#define IGB_DEFAULT_ITR 1000000000/(igb_default_intrrate * 256)*/ +#define IGB_DEFAULT_ITR ((1000000/igb_default_intrrate)<<2) + +/* * MSIX should be the default for best performance, * but this allows it to be forced off for testing. */ @@ -422,6 +433,11 @@ igb_attach(device_t dev) OID_AUTO, "enable_aim", CTLTYPE_INT|CTLFLAG_RW, &igb_enable_aim, 1, "Interrupt Moderation"); + SYSCTL_ADD_INT(device_get_sysctl_ctx(dev), + SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), + OID_AUTO, "default_intrrate", CTLTYPE_INT|CTLFLAG_RW, + &igb_default_intrrate, 1, "Default ints/sec (for each queue)"); + callout_init_mtx(&adapter->timer, &adapter->core_mtx, 0); /* Determine hardware and mac info */ @@ -2342,6 +2358,7 @@ igb_configure_queues(struct adapter *ada } /* Set the starting interrupt rate */ + newitr &= 0x7FFC; /* Mask invalid bits */ if (hw->mac.type == e1000_82575) newitr |= newitr << 16; else --k+w/mQv8wyuph6w0--