Date: Mon, 19 Oct 2015 15:03:53 +0000 From: "Eggert, Lars" <lars@netapp.com> To: Luigi Rizzo <rizzo@iet.unipi.it> Cc: "jfv@FreeBSD.org" <jfv@freebsd.org>, "ricera10@gmail.com" <ricera10@gmail.com>, Giuseppe Lettieri <g.lettieri@iet.unipi.it>, "net@freebsd.org" <net@freebsd.org> Subject: Re: ixl 40G bad performance? Message-ID: <C81B629B-67D5-4E5B-A215-452E837BE673@netapp.com> In-Reply-To: <CA%2BhQ2%2Bj5PkmRWe-ydahExfvOj%2BzestFEn%2B5DZ9_acG2Z_v=8_w@mail.gmail.com> References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> <CA%2BhQ2%2Bj5PkmRWe-ydahExfvOj%2BzestFEn%2B5DZ9_acG2Z_v=8_w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_5D33675E-C592-4762-8246-EC3CEF5B4EE2 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, On 2015-10-19, at 16:20, Luigi Rizzo <rizzo@iet.unipi.it> wrote: >=20 > i would look at the following: > - c states and clock speed - make sure you never go below C1, > and fix the clock speed to max. > Sure these parameters also affect the 10G card, but there > may be strange interaction that trigger the power saving > modes in different ways I already have powerd_flags=3D"-a max -b max -n max" in rc.conf, which I = hope should be enough. > - interrupt moderation (may affect ping latency, > do not remember how it is set in ixl but probably a sysctl ixl(4) describes two sysctls that sound like they control AIM, and they = default to off: hw.ixl.dynamic_tx_itr: 0 hw.ixl.dynamic_rx_itr: 0 > - number of queues (32 is a lot i wouldn't use more than 4-8), > may affect cpu-socket affinity With hw.ixl.max_queues=3D4 in loader.conf, performance is still = unchanged. > - tso and flow director - i have seen bad effects of > accelerations so i would run the iperf test with > of these features disabled on both sides, and then enable > them one at a time No change with "ifconfig -tso4 -tso6 -rxcsum -txcsum -lro". How do I turn off flow director? > - queue sizes - the driver seems to use 1024 slots which is > about 1.5 MB queued, which in turn means you have 300us > (and possibly half of that) to drain the queue at 40Gbit/s. > 150-300us may seem an eternity, but if a couple of cores fall > into c7 your budget is gone and the loss will trigger a > retransmission and window halving etc. Also no change with "hw.ixl.ringsz=3D256" in loader.conf. This is really weird. Lars --Apple-Mail=_5D33675E-C592-4762-8246-EC3CEF5B4EE2 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="signature.asc" Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- iQCVAwUBViUGW9ZcnpRveo1xAQjoBAP9F6YSpjWoZXl+C4yVN3eDWuzw9lCHfM7k r87TQ0avA1puKOTE84chmNCuhonnCvFpGWAscgmufZ/PeYkyR5TynFjwpMwjkVXB 4I18y/i7sOZqV98JOUWoVIBY+hhYb9BeitX7s+Ip6zoBgMILg82/mkn45tGb2gtV 6gC8c7VtXz4= =pAzg -----END PGP SIGNATURE----- --Apple-Mail=_5D33675E-C592-4762-8246-EC3CEF5B4EE2--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?C81B629B-67D5-4E5B-A215-452E837BE673>