Date: Tue, 18 Aug 2015 08:53:01 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Daniel Braniss <danny@cs.huji.ac.il> Cc: FreeBSD Net <freebsd-net@freebsd.org>, Christopher Forgeron <csforgeron@gmail.com>, FreeBSD stable <freebsd-stable@freebsd.org>, Slawa Olhovchenkov <slw@zxy.spb.ru> Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance Message-ID: <1721122651.24481798.1439902381663.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <9D8B0503-E8FA-43CA-88F0-01F184F84D9B@cs.huji.ac.il> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <20150817094145.GB3158@zxy.spb.ru> <CAB2_NwBOLcL4EVjFN6=BvBC_YN=gmfZMweVbmb5ZPCsK4Hnx1g@mail.gmail.com> <17871443-E105-4434-80B1-6939306A865F@cs.huji.ac.il> <473274181.23263108.1439814072514.JavaMail.zimbra@uoguelph.ca> <7F892C70-9C04-4468-9514-EDBFE75CF2C6@cs.huji.ac.il> <805850043.24018217.1439848150695.JavaMail.zimbra@uoguelph.ca> <9D8B0503-E8FA-43CA-88F0-01F184F84D9B@cs.huji.ac.il>
next in thread | previous in thread | raw e-mail | index | archive | help
Daniel Braniss wrote:
>=20
> > On Aug 18, 2015, at 12:49 AM, Rick Macklem <rmacklem@uoguelph.ca> wrote=
:
> >=20
> > Daniel Braniss wrote:
> >>=20
> >>> On Aug 17, 2015, at 3:21 PM, Rick Macklem <rmacklem@uoguelph.ca> wrot=
e:
> >>>=20
> >>> Daniel Braniss wrote:
> >>>>=20
> >>>>> On Aug 17, 2015, at 1:41 PM, Christopher Forgeron
> >>>>> <csforgeron@gmail.com>
> >>>>> wrote:
> >>>>>=20
> >>>>> FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and Fre=
eBSD
> >>>>> 10.1. Before 10.1 it was less.
> >>>>>=20
> >>>>=20
> >>>> this is NOT iperf/3 where i do get close to wire speed,
> >>>> it=E2=80=99s NFS writes, i.e., almost real work :-)
> >>>>=20
> >>>>> I used to tweak the card settings, but now it's just stock. You may
> >>>>> want
> >>>>> to
> >>>>> check your settings, the Mellanox may just have better defaults for
> >>>>> your
> >>>>> switch.
> >>>>>=20
> >>> Have you tried disabling TSO for the Intel? With TSO enabled, it will=
be
> >>> copying
> >>> every transmitted mbuf chain to a new chain of mbuf clusters via.
> >>> m_defrag() when
> >>> TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to be t=
he
> >>> 82599 chip
> >>> these days?)
> >>>=20
> >>=20
> >> hi Rick
> >>=20
> >> how can i check the chip?
> >>=20
> > Haven't a clue. Does "dmesg" tell you? (To be honest, since disabling T=
SO
> > helped,
> > I'll bet you don't have a 82598.)
> >=20
> >>> This has been fixed in the driver very recently, but those fixes won'=
t be
> >>> in 10.1.
> >>>=20
> >>> rick
> >>> ps: If you could test with 10.2, it would be interesting to see how t=
he
> >>> ix
> >>> does with
> >>> the current driver fixes in it?
> >>=20
> >> I new TSO was involved!
> >> ok, firstly, it=E2=80=99s 10.2 stable.
> >> with TSO enabled, ix is bad, around 64MGB/s.
> >> disabling TSO it=E2=80=99s better, around 130
> >>=20
> > Hmm, could you check to see of these lines are in sys/dev/ixgbe/if_ix.c=
at
> > around
> > line#2500?
> > /* TSO parameters */
> > 2572 =09 =09 ifp->if_hw_tsomax =3D 65518;
> > 2573 =09 =09 ifp->if_hw_tsomaxsegcount =3D IXGBE_82599_SCATTER=
;
> > 2574 =09 =09 ifp->if_hw_tsomaxsegsize =3D 2048;
> >=20
> > They are in stable/10. I didn't look at releng/10.2. (And if they're in=
a
> > #ifdef
> > for FreeBSD11, take the #ifdef away.)
> > If they are there and not ifdef'd, I can't explain why disabling TSO wo=
uld
> > help.
> > Once TSO is fixed so that it handles the 64K transmit segments without
> > copying all
> > the mbufs, I suspect you might get better perf. with it enabled?
> >=20
>=20
> this is 10.2 :
> they are on lines 2509-2511 and I don=E2=80=99t see any #ifdefs around i=
t.
>=20
> the plot thickens :-)
>=20
If this is just a test machine, maybe you could test with these lines (at a=
bout #880)
in sys/netinet/tcp_output.c commented out? (It looks to me like this will d=
isable TSO
for almost all the NFS writes.)
- around line #880 in sys/netinet/tcp_output.c:
=09=09=09/*
=09=09=09 * In case there are too many small fragments
=09=09=09 * don't use TSO:
=09=09=09 */
=09=09=09if (len <=3D max_len) {
=09=09=09=09len =3D max_len;
=09=09=09=09sendalot =3D 1;
=09=09=09=09tso =3D 0;
=09=09=09}
This was added along with the other stuff that did the if_hw_tsomaxsegcount=
, etc and I
never noticed it until now (not my patch).
rick
> danny
>=20
> > Good luck with it, rick
> >=20
> >> still, mlxen0 is about 250! with and without TSO
> >>=20
> >>=20
> >>>=20
> >>>>> On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov <slw@zxy.spb.ru
> >>>>> <mailto:slw@zxy.spb.ru>> wrote:
> >>>>> On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote:
> >>>>>=20
> >>>>>> hi,
> >>>>>> I have a host (Dell R730) with both cards, connected to an HP8=
200
> >>>>>> switch at 10Gb.
> >>>>>> when writing to the same storage (netapp) this is what I get:
> >>>>>> ix0: ~130MGB/s
> >>>>>> mlxen0 ~330MGB/s
> >>>>>> this is via nfs/tcpv3
> >>>>>>=20
> >>>>>> I can get similar (bad) performance with the mellanox if I
> >>>>>> increase
> >>>>>> the file size
> >>>>>> to 512MGB.
> >>>>>=20
> >>>>> Look like mellanox have internal beffer for caching and do ACK
> >>>>> acclerating.
> >>>>>=20
> >>>>>> so at face value, it seems the mlxen does a better use of
> >>>>>> resources
> >>>>>> than the intel.
> >>>>>> Any ideas how to improve ix/intel's performance?
> >>>>>=20
> >>>>> Are you sure about netapp performance?
> >>>>> _______________________________________________
> >>>>> freebsd-net@freebsd.org <mailto:freebsd-net@freebsd.org> mailing li=
st
> >>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> >>>>> <https://lists.freebsd.org/mailman/listinfo/freebsd-net>;
> >>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.o=
rg
> >>>>> <mailto:freebsd-net-unsubscribe@freebsd.org>"
> >>>>>=20
> >>>>=20
> >>>> _______________________________________________
> >>>> freebsd-stable@freebsd.org mailing list
> >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> >>>> To unsubscribe, send any mail to
> >>>> "freebsd-stable-unsubscribe@freebsd.org"
> >>=20
> >> _______________________________________________
> >> freebsd-stable@freebsd.org mailing list
> >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.o=
rg"
>=20
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1721122651.24481798.1439902381663.JavaMail.zimbra>