From owner-freebsd-net@FreeBSD.ORG Thu Apr 28 13:25:03 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 018E71065673 for ; Thu, 28 Apr 2011 13:25:02 +0000 (UTC) (envelope-from kungfujesus06@gmail.com) Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id ADE9F8FC0A for ; Thu, 28 Apr 2011 13:25:02 +0000 (UTC) Received: by iwn33 with SMTP id 33so3073421iwn.13 for ; Thu, 28 Apr 2011 06:25:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:date:from:to:cc:subject:message-id:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=LOveyBuNxhfGBHoDybw31HCHbqUMyxTaTv/KX2/be1s=; b=VbPzF2nduAnhoOIzxsGuxOIjjsXpYtdvuj0sc3/JEZT89ziHoQzoZGSOWJPuWTbpNH XY0jIP9nnnkHO8WTN1ELhT5PiIqBq5O7N2r4LUs4tP4u2bq7EN2gDwM4tshhWm0AWhhf 3x0uFwMYCI9l30sraxWOZ9BiFFKG0WuIKGK+E= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=OSKRMZk5IgrwnfqMWHa9cUDZU4g76sf+IOQvb2sgmpLQoh8ZXrijnXaynHYUJWmGOW N4sGLgh85HR2Hvzg9ViR5o0PnCAzDuK53x9IZaGmcK22e4Endy7xmdp7xgf0rwgOG0Ii JJvPBgK2GomXR87TwJLkcjIYjwDfJpUxfW3g4= Received: by 10.43.65.197 with SMTP id xn5mr4620668icb.87.1303997101907; Thu, 28 Apr 2011 06:25:01 -0700 (PDT) Received: from ossumpossum.geop.uc.edu ([129.137.163.68]) by mx.google.com with ESMTPS id y10sm720558iba.12.2011.04.28.06.24.57 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 28 Apr 2011 06:24:58 -0700 (PDT) Date: Thu, 28 Apr 2011 09:25:13 -0400 From: Adam Stylinski To: Pierre Lamy Message-ID: <20110428132513.GB2800@ossumpossum.geop.uc.edu> References: <20110428072946.GA11391@zephyr.adamsnet> <4DB961EA.4080407@userid.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="1LKvkjL3sHcu1TtY" Content-Disposition: inline In-Reply-To: <4DB961EA.4080407@userid.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-net@freebsd.org Subject: Re: em0 performance subpar X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2011 13:25:03 -0000 --1LKvkjL3sHcu1TtY Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Apr 28, 2011 at 08:47:38AM -0400, Pierre Lamy wrote: > Try using netblast on FreeBSD instead of iperf, there have been a lot of= =20 > discussions about this on this list. >=20 > Is it possible you're maxing out the system's PCI-xxx bus? Did you tune= =20 > up the system buffers? Data doesn't just get queued up on the NIC=20 > driver, it also queues within the system's kernel buffers. Try these, I= =20 > have no idea if it will help: >=20 > sysctl -w net.inet.tcp.sendspace=3D373760 > sysctl -w net.inet.tcp.recvspace=3D373760 > sysctl -w net.local.stream.sendspace=3D82320 > sysctl -w net.local.stream.recvspace=3D82320 > sysctl -w net.local.stream.recvspace=3D373760 > sysctl -w net.local.stream.sendspace=3D373760 > sysctl -w net.raw.recvspace=3D373760 > sysctl -w net.raw.sendspace=3D373760 > sysctl -w net.inet.tcp.local_slowstart_flightsize=3D10 > sysctl -a net.inet.tcp.delayed_ack=3D0 > sysctl -w kern.maxvnodes=3D600000 > sysctl -w net.local.dgram.recvspace=3D8192 > sysctl -w net.local.dgram.maxdgram=3D8192 > sysctl -w net.inet.tcp.slowstart_flightsize=3D10 > sysctl -w net.inet.tcp.path_mtu_discovery=3D0 >=20 > They're all tunable while system is running. >=20 > -Pierre >=20 > On 4/28/2011 3:29 AM, Adam Stylinski wrote: > > Hello, > > > > I have an intel gigabit network adapter (the 1000 GT w/chipset 82541PI)= which performs poorly in Freebsd compared to the same card in Linux. I've= tried this card in two different freebsd boxes and for whatever reason I g= et poor transmit performance. I've done all of the tweaking specified in j= ust about every guide out there (the usual TCP window scaling, larger nmbcl= usters, delayed acks, etc) and still I get only around 600mbps. I'm using = jumbo frames, with an MTU of 9000. I'm testing this with iperf. While I r= ealize that this may not be the most realistic test, linux hosts with the s= ame card can achieve 995Mbit/s to another host running this. When the Free= bsd box is the server, Linux hosts can transmit to it at around 800 somethi= ng Mbit/s. I've increased the transmit descriptors as specified in the if_= em man page, and while that gave me 20 or 30 more mbit/s, my transmit perfo= rmance is still below normal. > > > > sysctl stats report that the card is trigger a lot of tx_desc_fail2: > > dev.em.0.tx_desc_fail2: 3431 > > > > Looking at a comment in the source code this indicates that the card wa= s not able to obtain enough transmit descriptors (but I've given the card t= he maximum of 4096 in my loader.conf tunable). Is this a bug or a performa= nce regression of some kind? Does anybody have a fix for this? I tried an= other card with the same chip in a different box on 8-STABLE to no avail (t= he box I'm trying to improve performance on is on 8.2-RELEASE-p1). > > > > Anybody manage to make this card push above 600mbps in ideal network be= nchmarks? Any help would be gladly appreciated. >=20 >=20 >=20 I doubt I'm saturating the PCI bus, the only other thing on the bus is a re= ally really crappy PCI video card. The same card on lesser powerful machin= es with Linux (Pentium 4) are able to achieve much more throughput, so it's= not likely a bus limitation. I adjusted the listed sysctl live tunables which I hadn't already compensat= ed for and it didn't seem to have an effect. --=20 Adam Stylinski PGP Key: http://pohl.ececs.uc.edu/~adam/publickey.pub Blog: http://technicallyliving.blogspot.com --1LKvkjL3sHcu1TtY Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (FreeBSD) iQIcBAEBAgAGBQJNuWq5AAoJED6sRHE6Tvmnmm4P/19McIX8pZAIfBAvWU9WzIBi iEsbkyKMgTjRTxbWR1fvEw6XYqcqDkialypiQH66AdIppCFs4bli1APG7Urw+XAE LY01qgcWWWsFIxZgxADlMHmrVPtBH+rCsdNlnhFmyhS85Mq8sxfyBqbB5h6OjUJW mOBdacpfDb2BAkaMxni4B5yjJoLZu9R2iS+ttnNtp2x6ccfgtgEWW1G3jaUY6qpW SUM7MXh5hiQzpmHnl1VQoFcW31CeCwgeK6xsscl2rlQM5+FFD5+sLnBl5M0vsyGt 4gOe3DmzP7+v+Ul3TYW9VvIYPE+p6fFn1ZgM65ry4nTkABR0AvAyKfiP2ebTzCyR Y81Q6aXP5LPUWxwJPSMqFKZkGP+KwtgO3ZMkScz4EBGE8b163+DmJFT1rnKrX6io f55JYsoh2OeLFRstmSnu5HalhtcTSQZVAUFWpQIPogmgc76g3Sov9s51RCsU7RYo 5u0qTTmphACs4UvHc5NFV6znu/5MJcJdZSDa3kzmffhmBg8ViL2yVuW1e20KSKmG /8F4kJycvuJ+QesYC82umyR1wYRj6uWJOvSbA2Jkr4+RfNMiTw++bnJ+rWr49yBj Xo5lB5jLhaXU/tq3sIwaNBjm9MgglgIylCLMS4ZpuHxpG4dR3t/wwI6D3Y1NqUZ/ yYDQB0yb6hf6tOMZekg6 =/JUS -----END PGP SIGNATURE----- --1LKvkjL3sHcu1TtY--