From owner-freebsd-net@FreeBSD.ORG Thu Apr 28 15:51:11 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 18863106564A for ; Thu, 28 Apr 2011 15:51:11 +0000 (UTC) (envelope-from kungfujesus06@gmail.com) Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id CCFE28FC12 for ; Thu, 28 Apr 2011 15:51:10 +0000 (UTC) Received: by iwn33 with SMTP id 33so3238733iwn.13 for ; Thu, 28 Apr 2011 08:51:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:date:from:to:cc:subject:message-id:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=wMeGT/h6izQBIrtA15dV6AwZGwmKcOO4Cmz/D3NG9qQ=; b=HTHXMjn0/IY7cg9AbOKYvj8v6NzdGIgMRW1FIAc2ih4t9RJLYzz9leKlpKU3yMtJXu IkAnaLkrZXX7dnD93Ry+iAod1hbVGRNSSv1VQSgDgIr3QJgw+oECwxobLkvdTSrjFZAw ttG1y3TtqVvNQp37503uMCeHI0Qvc4pF2RgKw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=AZzRUJlse4MjBtEqdWiBd/f8dWfR/i6QmK2r/jnp70uWQ47Yh39qKa+zTL8jkL8m4d J1bMBo2v/jOAjGWDhtnF6z19BKa/u0DcR2KcZG+3XpiiPuj7meSV4LVpvOpiFUoFibS5 gM0vJNEToVWcdbPRb95QpJ512ZLKETxKtB1Y4= Received: by 10.42.100.204 with SMTP id b12mr4423020ico.18.1304005869976; Thu, 28 Apr 2011 08:51:09 -0700 (PDT) Received: from ossumpossum.geop.uc.edu ([129.137.163.68]) by mx.google.com with ESMTPS id uf10sm685870icb.5.2011.04.28.08.51.08 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 28 Apr 2011 08:51:09 -0700 (PDT) Date: Thu, 28 Apr 2011 11:51:24 -0400 From: Adam Stylinski To: Pierre Lamy Message-ID: <20110428155124.GD19362@ossumpossum.geop.uc.edu> References: <20110428072946.GA11391@zephyr.adamsnet> <4DB961EA.4080407@userid.org> <20110428132513.GB2800@ossumpossum.geop.uc.edu> <4DB994E7.9060805@userid.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="FFoLq8A0u+X9iRU8" Content-Disposition: inline In-Reply-To: <4DB994E7.9060805@userid.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-net@freebsd.org Subject: Re: em0 performance subpar X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2011 15:51:11 -0000 --FFoLq8A0u+X9iRU8 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Apr 28, 2011 at 11:25:11AM -0500, Pierre Lamy wrote: > Someone mentioned on freebsd-current: >=20 > With the 7.2.2 driver you also will use different mbuf pools depending on > > the MTU you are using. If you use jumbo frames it will use 4K clusters, > > if you go to 9K jumbos it will use 9K mbuf clusters. *The number of the= se > > allocated by default is small (like 6400 small :) .* > > > > I would use 'netstat -m' to see what the pools look like. >=20 > Hope this helps. Check the perf with 1500 byte frames. >=20 > Adam Stylinski wrote: > > On Thu, Apr 28, 2011 at 08:47:38AM -0400, Pierre Lamy wrote: > > =20 > >> Try using netblast on FreeBSD instead of iperf, there have been a lot = of=20 > >> discussions about this on this list. > >> > >> Is it possible you're maxing out the system's PCI-xxx bus? Did you tun= e=20 > >> up the system buffers? Data doesn't just get queued up on the NIC=20 > >> driver, it also queues within the system's kernel buffers. Try these, = I=20 > >> have no idea if it will help: > >> > >> sysctl -w net.inet.tcp.sendspace=3D373760 > >> sysctl -w net.inet.tcp.recvspace=3D373760 > >> sysctl -w net.local.stream.sendspace=3D82320 > >> sysctl -w net.local.stream.recvspace=3D82320 > >> sysctl -w net.local.stream.recvspace=3D373760 > >> sysctl -w net.local.stream.sendspace=3D373760 > >> sysctl -w net.raw.recvspace=3D373760 > >> sysctl -w net.raw.sendspace=3D373760 > >> sysctl -w net.inet.tcp.local_slowstart_flightsize=3D10 > >> sysctl -a net.inet.tcp.delayed_ack=3D0 > >> sysctl -w kern.maxvnodes=3D600000 > >> sysctl -w net.local.dgram.recvspace=3D8192 > >> sysctl -w net.local.dgram.maxdgram=3D8192 > >> sysctl -w net.inet.tcp.slowstart_flightsize=3D10 > >> sysctl -w net.inet.tcp.path_mtu_discovery=3D0 > >> > >> They're all tunable while system is running. > >> > >> -Pierre > >> > >> On 4/28/2011 3:29 AM, Adam Stylinski wrote: > >> =20 > >>> Hello, > >>> > >>> I have an intel gigabit network adapter (the 1000 GT w/chipset 82541P= I) which performs poorly in Freebsd compared to the same card in Linux. I'= ve tried this card in two different freebsd boxes and for whatever reason I= get poor transmit performance. I've done all of the tweaking specified in= just about every guide out there (the usual TCP window scaling, larger nmb= clusters, delayed acks, etc) and still I get only around 600mbps. I'm usin= g jumbo frames, with an MTU of 9000. I'm testing this with iperf. While I= realize that this may not be the most realistic test, linux hosts with the= same card can achieve 995Mbit/s to another host running this. When the Fr= eebsd box is the server, Linux hosts can transmit to it at around 800 somet= hing Mbit/s. I've increased the transmit descriptors as specified in the i= f_em man page, and while that gave me 20 or 30 more mbit/s, my transmit per= formance is still below normal. > >>> > >>> sysctl stats report that the card is trigger a lot of tx_desc_fail2: > >>> dev.em.0.tx_desc_fail2: 3431 > >>> > >>> Looking at a comment in the source code this indicates that the card = was not able to obtain enough transmit descriptors (but I've given the card= the maximum of 4096 in my loader.conf tunable). Is this a bug or a perfor= mance regression of some kind? Does anybody have a fix for this? I tried = another card with the same chip in a different box on 8-STABLE to no avail = (the box I'm trying to improve performance on is on 8.2-RELEASE-p1). > >>> > >>> Anybody manage to make this card push above 600mbps in ideal network = benchmarks? Any help would be gladly appreciated. > >>> =20 > >> > >> =20 > > > > I doubt I'm saturating the PCI bus, the only other thing on the bus is = a really really crappy PCI video card. The same card on lesser powerful ma= chines with Linux (Pentium 4) are able to achieve much more throughput, so = it's not likely a bus limitation. > > > > I adjusted the listed sysctl live tunables which I hadn't already compe= nsated for and it didn't seem to have an effect. > > > > =20 >=20 >=20 >=20 I tweaked that, doesn't seem to help. I'd rather not go down to a 1500 byt= e MTU just yet. I may try this later, but most of my network is configured= for jumbo frames. [adam@nasbox ~]$ sysctl kern.ipc.nmbjumbo9 kern.ipc.nmbjumbo9: 12800 --=20 Adam Stylinski PGP Key: http://pohl.ececs.uc.edu/~adam/publickey.pub Blog: http://technicallyliving.blogspot.com --FFoLq8A0u+X9iRU8 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (FreeBSD) iQIcBAEBAgAGBQJNuYz7AAoJED6sRHE6TvmnoZUP/02iM1WELL0XtWFBu++CHH42 NCjLH3LUVsQW/ief/XoqXt3zvfaPIemRZEi50jAKPbXDcq9t+PCG8vbvE2d+Q2i+ h1zmwbdIgT/x0lqm9U9aaTNpcIXjFUqVS1Qs36HcA31DOMcP4bSakJt+iSZQp9tl TfwLpuH8NojyLDMvZkkEpzX8Rgvc3NmmjSysCLfhnSortjQhiyxCBP0u5IyDHC38 KTB3PPkTeFm3jvXzCRtwRWqeLG/Wzp8I9PdpmoU1cXuVlMqUukEqWjd0NL85dbHL T3QFQChe4KsZ7s/WLF654WZR9QDXDgDmi06i3mD5kWfJf/hhUn05aAbFy8etcYwf gizKccgip4TegIzuIVbkk5Kb/KCI0WsSFQkGBQeQv+YMVXww6N0nnK9GDgxIUtJu cbWNwrOt+RWMCBahZbFvAqB5ffDnD0MLjuRTlWYVlCkjsi6gL+a1sdyYtHqfB+J0 wF1eGmImVHrQqmQo8QTBXn6smaSktGrnByhkwS1TeYigT5RXYtk12tFUCZ5Z6hCV gDSUWsZG38DlXldocRvHv4OEpIA5eGXGr7X7V9n+52qRO6YG6QLumpzOPxpofmhy mImjUaGyzdRk7eHXM8V9DaG5J3DZQkEpMBjRbjftyw0AdlsXByodyDF6tCnrETzI +3QUGoIXfP5Pw0qUq016 =+gSG -----END PGP SIGNATURE----- --FFoLq8A0u+X9iRU8--