From owner-freebsd-net@FreeBSD.ORG Mon May 23 17:43:12 2011 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F1B81106566C for ; Mon, 23 May 2011 17:43:12 +0000 (UTC) (envelope-from bhutchings@solarflare.com) Received: from exchange.solarflare.com (mail.solarflare.com [216.237.3.220]) by mx1.freebsd.org (Postfix) with ESMTP id D4D868FC0A for ; Mon, 23 May 2011 17:43:12 +0000 (UTC) Received: from [10.26.194.132] ([12.11.238.122]) by exchange.solarflare.com with Microsoft SMTPSVC(6.0.3790.4675); Mon, 23 May 2011 10:43:12 -0700 From: Ben Hutchings To: Luigi Rizzo In-Reply-To: <20110523172010.GA540@onelab2.iet.unipi.it> References: <20110523172010.GA540@onelab2.iet.unipi.it> Content-Type: text/plain; charset="UTF-8" Organization: Solarflare Date: Mon, 23 May 2011 10:43:10 -0700 Message-ID: <1306172590.3456.75.camel@localhost> Mime-Version: 1.0 X-Mailer: Evolution 2.32.3 Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 23 May 2011 17:43:12.0364 (UTC) FILETIME=[E31382C0:01CC1970] X-TM-AS-Product-Ver: SMEX-8.0.0.1181-6.500.1024-18152.004 X-TM-AS-Result: No--13.339900-0.000000-31 X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No Cc: net@freebsd.org Subject: Re: strange performance receiving with ixgbe/82599 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 May 2011 17:43:13 -0000 On Mon, 2011-05-23 at 19:20 +0200, Luigi Rizzo wrote: > Jack, > i am doing some experiments with a modified ixgbe driver, and i am > seeing some strange performance numbers which I'd like to explain. > > My card uses the 82599 chip, monted on an x16 slot on an Asus > motherboard. CPU is i7-870 @ 2.93GHz. Both -head and RELENG_8 > exhibit the same behaviour. I am using a modified driver which > essentially accesses the rings from userspace, sending or > receiving frames as soon as there is room. > > On the transmit side, using 64 byte packets (60+4 CRC) i can saturate > the link (measured 14.87-14.88 Mpps) with no problem with just one > core at 1.33 GHz and 2 or 4 queues (one queue tops at 12Mpps, > but that's not a big issue). > > On the receive side, i can sustain 14.2Mpps (on 2-4 queues) if the > sender generates 68-byte packets (i.e. 64+4CRC), but as soon as i > go to 63+4 or less, the receive rate drops to 11.5-11.6Mpps. > CPU cycles abound, because the speed is unchanged down to 1.45GHz > > Any reason to explain that suddend drop with short (but legal) > packet sizes ? [...] With 64-bit wide RAM, a DMA write of 64 bytes with 8-byte alignment covers exactly 8 memory words. However a DMA write of 63 bytes with 8-byte alignment at the start covers 7 full and one partial memory word. So a read-modify-write operation is required for the last word. Ben. -- Ben Hutchings, Senior Software Engineer, Solarflare Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.