Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 28 Apr 2011 11:51:24 -0400
From:      Adam Stylinski <kungfujesus06@gmail.com>
To:        Pierre Lamy <pierre@userid.org>
Cc:        freebsd-net@freebsd.org
Subject:   Re: em0 performance subpar
Message-ID:  <20110428155124.GD19362@ossumpossum.geop.uc.edu>
In-Reply-To: <4DB994E7.9060805@userid.org>
References:  <20110428072946.GA11391@zephyr.adamsnet> <4DB961EA.4080407@userid.org> <20110428132513.GB2800@ossumpossum.geop.uc.edu> <4DB994E7.9060805@userid.org>

next in thread | previous in thread | raw e-mail | index | archive | help

--FFoLq8A0u+X9iRU8
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Thu, Apr 28, 2011 at 11:25:11AM -0500, Pierre Lamy wrote:
> Someone mentioned on freebsd-current:
>=20
> With the 7.2.2 driver you also will use different mbuf pools depending on
> > the MTU you are using. If you use jumbo frames it will use 4K clusters,
> > if you go to 9K jumbos it will use 9K mbuf clusters. *The number of the=
se
> > allocated by default is small (like 6400 small  :) .*
> >
> > I would use 'netstat -m' to see what the pools look like.
>=20
> Hope this helps. Check the perf with 1500 byte frames.
>=20
> Adam Stylinski wrote:
> > On Thu, Apr 28, 2011 at 08:47:38AM -0400, Pierre Lamy wrote:
> >  =20
> >> Try using netblast on FreeBSD instead of iperf, there have been a lot =
of=20
> >> discussions about this on this list.
> >>
> >> Is it possible you're maxing out the system's PCI-xxx bus? Did you tun=
e=20
> >> up the system buffers? Data doesn't just get queued up on the NIC=20
> >> driver, it also queues within the system's kernel buffers. Try these, =
I=20
> >> have no idea if it will help:
> >>
> >> sysctl -w net.inet.tcp.sendspace=3D373760
> >> sysctl -w net.inet.tcp.recvspace=3D373760
> >> sysctl -w net.local.stream.sendspace=3D82320
> >> sysctl -w net.local.stream.recvspace=3D82320
> >> sysctl -w net.local.stream.recvspace=3D373760
> >> sysctl -w net.local.stream.sendspace=3D373760
> >> sysctl -w net.raw.recvspace=3D373760
> >> sysctl -w net.raw.sendspace=3D373760
> >> sysctl -w net.inet.tcp.local_slowstart_flightsize=3D10
> >> sysctl -a net.inet.tcp.delayed_ack=3D0
> >> sysctl -w kern.maxvnodes=3D600000
> >> sysctl -w net.local.dgram.recvspace=3D8192
> >> sysctl -w net.local.dgram.maxdgram=3D8192
> >> sysctl -w net.inet.tcp.slowstart_flightsize=3D10
> >> sysctl -w net.inet.tcp.path_mtu_discovery=3D0
> >>
> >> They're all tunable while system is running.
> >>
> >> -Pierre
> >>
> >> On 4/28/2011 3:29 AM, Adam Stylinski wrote:
> >>    =20
> >>> Hello,
> >>>
> >>> I have an intel gigabit network adapter (the 1000 GT w/chipset 82541P=
I) which performs poorly in Freebsd compared to the same card in Linux.  I'=
ve tried this card in two different freebsd boxes and for whatever reason I=
 get poor transmit performance.  I've done all of the tweaking specified in=
 just about every guide out there (the usual TCP window scaling, larger nmb=
clusters, delayed acks, etc) and still I get only around 600mbps.  I'm usin=
g jumbo frames, with an MTU of 9000.  I'm testing this with iperf.  While I=
 realize that this may not be the most realistic test, linux hosts with the=
 same card can achieve 995Mbit/s to another host running this.  When the Fr=
eebsd box is the server, Linux hosts can transmit to it at around 800 somet=
hing Mbit/s.  I've increased the transmit descriptors as specified in the i=
f_em man page, and while that gave me 20 or 30 more mbit/s, my transmit per=
formance is still below normal.
> >>>
> >>> sysctl stats report that the card is trigger a lot of tx_desc_fail2:
> >>> 	dev.em.0.tx_desc_fail2: 3431
> >>>
> >>> Looking at a comment in the source code this indicates that the card =
was not able to obtain enough transmit descriptors (but I've given the card=
 the maximum of 4096 in my loader.conf tunable).  Is this a bug or a perfor=
mance regression of some kind?  Does anybody have a fix for this?  I tried =
another card with the same chip in a different box on 8-STABLE to no avail =
(the box I'm trying to improve performance on is on 8.2-RELEASE-p1).
> >>>
> >>> Anybody manage to make this card push above 600mbps in ideal network =
benchmarks?  Any help would be gladly appreciated.
> >>>      =20
> >>
> >>    =20
> >
> > I doubt I'm saturating the PCI bus, the only other thing on the bus is =
a really really crappy PCI video card.  The same card on lesser powerful ma=
chines with Linux (Pentium 4) are able to achieve much more throughput, so =
it's not likely a bus limitation.
> >
> > I adjusted the listed sysctl live tunables which I hadn't already compe=
nsated for and it didn't seem to have an effect.
> >
> >  =20
>=20
>=20
>=20
I tweaked that, doesn't seem to help.  I'd rather not go down to a 1500 byt=
e MTU just yet.  I may try this later, but most of my network is configured=
 for jumbo frames.

[adam@nasbox ~]$ sysctl kern.ipc.nmbjumbo9
kern.ipc.nmbjumbo9: 12800

--=20
Adam Stylinski
PGP Key: http://pohl.ececs.uc.edu/~adam/publickey.pub
Blog: http://technicallyliving.blogspot.com

--FFoLq8A0u+X9iRU8
Content-Type: application/pgp-signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (FreeBSD)

iQIcBAEBAgAGBQJNuYz7AAoJED6sRHE6TvmnoZUP/02iM1WELL0XtWFBu++CHH42
NCjLH3LUVsQW/ief/XoqXt3zvfaPIemRZEi50jAKPbXDcq9t+PCG8vbvE2d+Q2i+
h1zmwbdIgT/x0lqm9U9aaTNpcIXjFUqVS1Qs36HcA31DOMcP4bSakJt+iSZQp9tl
TfwLpuH8NojyLDMvZkkEpzX8Rgvc3NmmjSysCLfhnSortjQhiyxCBP0u5IyDHC38
KTB3PPkTeFm3jvXzCRtwRWqeLG/Wzp8I9PdpmoU1cXuVlMqUukEqWjd0NL85dbHL
T3QFQChe4KsZ7s/WLF654WZR9QDXDgDmi06i3mD5kWfJf/hhUn05aAbFy8etcYwf
gizKccgip4TegIzuIVbkk5Kb/KCI0WsSFQkGBQeQv+YMVXww6N0nnK9GDgxIUtJu
cbWNwrOt+RWMCBahZbFvAqB5ffDnD0MLjuRTlWYVlCkjsi6gL+a1sdyYtHqfB+J0
wF1eGmImVHrQqmQo8QTBXn6smaSktGrnByhkwS1TeYigT5RXYtk12tFUCZ5Z6hCV
gDSUWsZG38DlXldocRvHv4OEpIA5eGXGr7X7V9n+52qRO6YG6QLumpzOPxpofmhy
mImjUaGyzdRk7eHXM8V9DaG5J3DZQkEpMBjRbjftyw0AdlsXByodyDF6tCnrETzI
+3QUGoIXfP5Pw0qUq016
=+gSG
-----END PGP SIGNATURE-----

--FFoLq8A0u+X9iRU8--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20110428155124.GD19362>