Date: Wed, 12 Aug 2015 01:28:29 +0000 (UTC) From: Barney Cordoba <barney_cordoba@yahoo.com> To: =?UTF-8?Q?Olivier_Cochard-Labb=C3=A9?= <olivier@cochard.me>, Maxim Sobolev <sobomax@freebsd.org> Cc: FreeBSD Net <freebsd-net@freebsd.org>, "freebsd@intel.com" <freebsd@intel.com>, =?UTF-8?Q?Jev_Bj=C3=B6rsell?= <jev@sippysoft.com> Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 Message-ID: <1931183673.3234634.1439342909940.JavaMail.yahoo@mail.yahoo.com> In-Reply-To: <CA%2Bq%2BTcqq1SGofhwA9PBmkcNrWsiyY32Ha9tfFM8ROwXmmvX%2BrA@mail.gmail.com> References: <CA%2Bq%2BTcqq1SGofhwA9PBmkcNrWsiyY32Ha9tfFM8ROwXmmvX%2BrA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Wow, this is really important! if this is a college project, I give you a D=
. Maybe a D- because it's almost useless information.
You ignore the most important aspect of "performance". Efficiency is arguab=
ly the most important aspect of performance.=C2=A0
1M pps at 20% cpu usage is much better "performance" than 1.2M pps at 85%.=
=C2=A0
Why don't any of you understand this simple thing? Why does spreading equal=
ity really matter, unless you are hitting a wall with your cpus? I don't ca=
re which cpu processes which packet. If you weren't doing moronic things li=
ke binding to a cpu, then you'd never have to care about distribution unles=
s it was extremely unbalanced.
BC=20
On Tuesday, August 11, 2015 7:15 PM, Olivier Cochard-Labb=C3=A9 <olivi=
er@cochard.me> wrote:
=20
On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev <sobomax@freebsd.org> wrot=
e:
> Hi folks,
>
> =E2=80=8BHi,
=E2=80=8B
> We've trying to migrate some of our high-PPS systems to a new hardware th=
at
> has four X540-AT2 10G NICs and observed that interrupt time goes through
> roof after we cross around 200K PPS in and 200K out (two ports in LACP).
> The previous hardware was stable up to about 350K PPS in and 350K out. I
> believe the old one was equipped with the I350 and had the identical LACP
> configuration. The new box also has better CPU with more cores (i.e. 24
> cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3.
>
=E2=80=8B200K PPS, and even 350K PPS are very low value indeed.
On a Intel Xeon L5630 (4 cores only) with one X540-AT2=E2=80=8B
=E2=80=8B(then 2 10Gigabit ports)=E2=80=8B I've reached about 1.8Mpps (fast=
forwarding
enabled) [1].
But my setup didn't use lagg(4): Can you disable lagg configuration and
re-measure your performance without lagg ?
Do you let Intel NIC drivers using 8 queues for port too?
In my use case (forwarding smallest UDP packet size), I obtain better
behaviour by limiting NIC queues to 4 (hw.ix.num_queues or
hw.ixgbe.num_queues, don't remember) if my system had 8 cores. And this
with Gigabit Intel[2] or Chelsio NIC [3].
Don't forget to disable TSO and LRO too.
=E2=80=8BRegards,
Olivier
[1]
http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_an_ib=
m_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs
[2]
http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_sup=
erserver_5018a-ftn4#graph1
[3]
http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_hp_=
proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr#reduci=
ng_nic_queues
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
From owner-freebsd-net@freebsd.org Wed Aug 12 01:40:17 2015
Return-Path: <owner-freebsd-net@freebsd.org>
Delivered-To: freebsd-net@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
[IPv6:2001:1900:2254:206a::19:1])
by mailman.ysv.freebsd.org (Postfix) with ESMTP id A2BA399E7B8
for <freebsd-net@mailman.ysv.freebsd.org>;
Wed, 12 Aug 2015 01:40:17 +0000 (UTC)
(envelope-from barney_cordoba@yahoo.com)
Received: from nm1-vm0.bullet.mail.ne1.yahoo.com
(nm1-vm0.bullet.mail.ne1.yahoo.com [98.138.91.74])
(using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits))
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id 6C5531CA
for <freebsd-net@freebsd.org>; Wed, 12 Aug 2015 01:40:17 +0000 (UTC)
(envelope-from barney_cordoba@yahoo.com)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048;
t=1439343609; bh=g0kHJWoBbWWQcX7i17mr+ATL4vVJs831Ef4Jrp2PDGM=;
h=Date:From:Reply-To:To:Cc:In-Reply-To:References:Subject:From:Subject;
b=KzoNi/BIo21zcbWtM9+t9ifIDGyIngCrf8aviCcyhAHyyS1vHq6Gk5bBW1m4cG7oipuWewJLCDsn3XR8gS5WRgsPCmhtUEtWKtfVdnJgLiQfVB3vx7zxYiWT89rkutAlWtbsDbofu2/TyDZi1rTiAdXkT31OauFieL5VbkupuI/uxZMbLPuLp9amcwrGHkjjUg+E7kd2afWkX3BD7uxHNPdq+1+UVqbHt1/OYlsH0X5aHcll1955WTJEy0UYcl4ZMEQctpFx3v9hEOBZURkWiUnubwqapE3/JWqST/Pw3TrU8ubYHrIhrjpl9h1Qg/bNtdkMR+kwne31oX/C3wxC9g==
Received: from [98.138.100.115] by nm1.bullet.mail.ne1.yahoo.com with NNFMP;
12 Aug 2015 01:40:09 -0000
Received: from [98.138.88.238] by tm106.bullet.mail.ne1.yahoo.com with NNFMP;
12 Aug 2015 01:40:09 -0000
Received: from [127.0.0.1] by omp1038.mail.ne1.yahoo.com with NNFMP;
12 Aug 2015 01:40:09 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 856165.77852.bm@omp1038.mail.ne1.yahoo.com
X-YMail-OSG: SpJcUVsVM1m_AJRXVKNA2xCb_5ie.ahITTyyihLhZWPsb8eYh1jqQ2f2RYlmQ_H
XVPV7CPkYUaZi8xBJoLxb3NqnG_siFNKw8NlobR3ZQjLnNtSPlyUnDxV88btcOaNStVKN80ce29p
w.5AWUsLzxKm1D7AbhYeEsN58xwvt7YE.8NR1WUapscewyleo_iMUsYm41YAPG1ye7tF3vFghpoS
kOHq4_hNDKXtGrCwsOMsodjwwdn8aBiHKu7M7l6VEsq1XqjW1Lb4bujECLPblAXLa3Zl7z7QDNRj
jYXsDM6w2PnBVvOZburgdL6rUto1Soem4xULXAUkEf9IukjC2cWvMBpgNOt0ZunZdFedQjt8IAJo
YXRWGlt1cTDMGGSIrDfh1QJ0anDtCofCNa9J2f4jnRIe3Cy_Xyq8r4PKxKqYBJcRjwf4vpvx_WTa
c8fQx20NirKd5QkZ6cUiRQtS_2x10dnPrxcHsiQz_Qdhwvo0NLD72sdP2pEbbHClGA1s4ipbgXHy
.sau.9H4MIDgqGw--
Received: by 98.138.101.170; Wed, 12 Aug 2015 01:40:09 +0000
Date: Wed, 12 Aug 2015 01:40:09 +0000 (UTC)
From: Barney Cordoba <barney_cordoba@yahoo.com>
Reply-To: Barney Cordoba <barney_cordoba@yahoo.com>
To: Barney Cordoba <barney_cordoba@yahoo.com>,
=?UTF-8?Q?Olivier_Cochard-Labb=C3=A9?= <olivier@cochard.me>,
Maxim Sobolev <sobomax@freebsd.org>
Cc: FreeBSD Net <freebsd-net@freebsd.org>,
"freebsd@intel.com" <freebsd@intel.com>,
=?UTF-8?Q?Jev_Bj=C3=B6rsell?= <jev@sippysoft.com>
Message-ID: <477444474.1722346.1439343609122.JavaMail.yahoo@mail.yahoo.com>
In-Reply-To: <1931183673.3234634.1439342909940.JavaMail.yahoo@mail.yahoo.com>
References: <1931183673.3234634.1439342909940.JavaMail.yahoo@mail.yahoo.com>
Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in
FreeBSD 10.1
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Content-Filtered-By: Mailman/MimeDel 2.1.20
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.20
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>,
<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>;
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 12 Aug 2015 01:40:17 -0000
Also, using a slow-ass cpu like the atom is completely absurd; first, no-on=
e would ever use them.=C2=A0
You have to test cpu usage under 60% cpu usage, because as you get to highe=
r cpu usage levels the lock contention increases exponentially. You're incr=
easing lock contention by having more queues; so more queues at higher cpu =
% usage will perform increasingly bad as usage increases.
You'd never run a system at 95% usage (ie totally hammering it) in real wor=
ld usage, so why would you benchmark at such a high usage? Everything chang=
es as cpu available become scarce.=C2=A0
"What is the pps at 50% cpu usage" is a better question to ask than the one=
you're asking.
BC=20
On Tuesday, August 11, 2015 9:29 PM, Barney Cordoba via freebsd-net <f=
reebsd-net@freebsd.org> wrote:
=20
Wow, this is really important! if this is a college project, I give you a =
D. Maybe a D- because it's almost useless information.
You ignore the most important aspect of "performance". Efficiency is arguab=
ly the most important aspect of performance.=C2=A0
1M pps at 20% cpu usage is much better "performance" than 1.2M pps at 85%.=
=C2=A0
Why don't any of you understand this simple thing? Why does spreading equal=
ity really matter, unless you are hitting a wall with your cpus? I don't ca=
re which cpu processes which packet. If you weren't doing moronic things li=
ke binding to a cpu, then you'd never have to care about distribution unles=
s it was extremely unbalanced.
BC=20
=C2=A0 =C2=A0 On Tuesday, August 11, 2015 7:15 PM, Olivier Cochard-Labb=C3=
=A9 <olivier@cochard.me> wrote:
=C2=A0=20
On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev <sobomax@freebsd.org> wrot=
e:
> Hi folks,
>
> =E2=80=8BHi,
=E2=80=8B
> We've trying to migrate some of our high-PPS systems to a new hardware th=
at
> has four X540-AT2 10G NICs and observed that interrupt time goes through
> roof after we cross around 200K PPS in and 200K out (two ports in LACP).
> The previous hardware was stable up to about 350K PPS in and 350K out. I
> believe the old one was equipped with the I350 and had the identical LACP
> configuration. The new box also has better CPU with more cores (i.e. 24
> cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3.
>
=E2=80=8B200K PPS, and even 350K PPS are very low value indeed.
On a Intel Xeon L5630 (4 cores only) with one X540-AT2=E2=80=8B
=E2=80=8B(then 2 10Gigabit ports)=E2=80=8B I've reached about 1.8Mpps (fast=
forwarding
enabled) [1].
But my setup didn't use lagg(4): Can you disable lagg configuration and
re-measure your performance without lagg ?
Do you let Intel NIC drivers using 8 queues for port too?
In my use case (forwarding smallest UDP packet size), I obtain better
behaviour by limiting NIC queues to 4 (hw.ix.num_queues or
hw.ixgbe.num_queues, don't remember) if my system had 8 cores. And this
with Gigabit Intel[2] or Chelsio NIC [3].
Don't forget to disable TSO and LRO too.
=E2=80=8BRegards,
Olivier
[1]
http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_an_ib=
m_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs
[2]
http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_sup=
erserver_5018a-ftn4#graph1
[3]
http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_hp_=
proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr#reduci=
ng_nic_queues
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
=C2=A0=20
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
From owner-freebsd-net@freebsd.org Wed Aug 12 01:46:45 2015
Return-Path: <owner-freebsd-net@freebsd.org>
Delivered-To: freebsd-net@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
[IPv6:2001:1900:2254:206a::19:1])
by mailman.ysv.freebsd.org (Postfix) with ESMTP id CE06499E92A
for <freebsd-net@mailman.ysv.freebsd.org>;
Wed, 12 Aug 2015 01:46:45 +0000 (UTC)
(envelope-from sobomax@sippysoft.com)
Received: from mail-wi0-f181.google.com (mail-wi0-f181.google.com
[209.85.212.181])
(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
(Client CN "smtp.gmail.com",
Issuer "Google Internet Authority G2" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 64E90A07
for <freebsd-net@freebsd.org>; Wed, 12 Aug 2015 01:46:44 +0000 (UTC)
(envelope-from sobomax@sippysoft.com)
Received: by wicne3 with SMTP id ne3so82163661wic.0
for <freebsd-net@freebsd.org>; Tue, 11 Aug 2015 18:46:42 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20130820;
h=x-gm-message-state:mime-version:sender:in-reply-to:references:date
:message-id:subject:from:to:cc:content-type;
bh=QnUF8NPF+tTwauLQQi4jIFxu/EM5dzbuE3bv602TNLA=;
b=IQ8EqxNP84p69cCmQf0TYp9Eu6SIO+xgXrpcKCvkNFOsC7HGCxFapbES6tsclhUlri
xhCXNAcVKm1i2H5Aa4STLwTMoq3qrt0Gbb3i+E3WgsRjKgUBZyEsr1fnjXKzjaQNF6Ws
0ALcvBOyGtDxtZRuIR7oGU6RIfPsCuI8auHhCNUPij/Ddl8if0kdRiIi4xtmUh1CrSgl
gLxaEgTIJYrlZ7uvyxv8L+qpcwaaPXjVHVMc3P6Qur0pT7AJ6KzqlZqQfeIP0U3zY+xX
fTSk11fZxds49Rozt8wZktf4f9d1kMA+8/CK4kQvXKODTFSWXjft+HRpzjxjTwgr0KRI
HVqw==
X-Gm-Message-State: ALoCoQn6BzD71Yqfk+Zxv5qXKPmBmIo21oSNzpCcMRTwYIeegAxqx08VRJ5W7JqfyDA5jJ+pYf1l
MIME-Version: 1.0
X-Received: by 10.180.78.166 with SMTP id c6mr21150419wix.8.1439344002764;
Tue, 11 Aug 2015 18:46:42 -0700 (PDT)
Sender: sobomax@sippysoft.com
Received: by 10.27.143.15 with HTTP; Tue, 11 Aug 2015 18:46:42 -0700 (PDT)
In-Reply-To: <1931183673.3234634.1439342909940.JavaMail.yahoo@mail.yahoo.com>
References: <CA+q+Tcqq1SGofhwA9PBmkcNrWsiyY32Ha9tfFM8ROwXmmvX+rA@mail.gmail.com>
<1931183673.3234634.1439342909940.JavaMail.yahoo@mail.yahoo.com>
Date: Tue, 11 Aug 2015 18:46:42 -0700
X-Google-Sender-Auth: vYWQutNBf8V6QTWySNnYWFiUeYc
Message-ID: <CAH7qZft_MCdGc=2yCj+9-kgZ9Hf6SW1T8NR71ktSnUaP9c6f9g@mail.gmail.com>
Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in
FreeBSD 10.1
From: Maxim Sobolev <sobomax@FreeBSD.org>
To: Barney Cordoba <barney_cordoba@yahoo.com>
Cc: =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= <olivier@cochard.me>,
FreeBSD Net <freebsd-net@freebsd.org>, "freebsd@intel.com" <freebsd@intel.com>,
=?UTF-8?Q?Jev_Bj=C3=B6rsell?= <jev@sippysoft.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Content-Filtered-By: Mailman/MimeDel 2.1.20
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.20
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>,
<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>;
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 12 Aug 2015 01:46:46 -0000
Thanks Barney for totally useless response and an attempted insult! And
yes, we are hitting CPU limit on 12 core E5-2620 v2 systems running I350,
so yes, we do know a little bit how to distribute our application at least
with the igb. For some reason this does not work with ixgb and we are
trying to understand why is it so.
http://sobomax.sippysoft.com/ScreenShot388.png
On Tue, Aug 11, 2015 at 6:28 PM, Barney Cordoba <barney_cordoba@yahoo.com>
wrote:
> Wow, this is really important! if this is a college project, I give you a
> D. Maybe a D- because it's almost useless information.
>
> You ignore the most important aspect of "performance". Efficiency is
> arguably the most important aspect of performance.
>
> 1M pps at 20% cpu usage is much better "performance" than 1.2M pps at 85%=
.
>
> Why don't any of you understand this simple thing? Why does spreading
> equality really matter, unless you are hitting a wall with your cpus? I
> don't care which cpu processes which packet. If you weren't doing moronic
> things like binding to a cpu, then you'd never have to care about
> distribution unless it was extremely unbalanced.
>
> BC
>
>
>
> On Tuesday, August 11, 2015 7:15 PM, Olivier Cochard-Labb=C3=A9 <
> olivier@cochard.me> wrote:
>
>
> On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev <sobomax@freebsd.org>
> wrote:
>
> > Hi folks,
> >
> > =E2=80=8BHi,
> =E2=80=8B
>
>
> > We've trying to migrate some of our high-PPS systems to a new hardware
> that
> > has four X540-AT2 10G NICs and observed that interrupt time goes throug=
h
> > roof after we cross around 200K PPS in and 200K out (two ports in LACP)=
.
> > The previous hardware was stable up to about 350K PPS in and 350K out. =
I
> > believe the old one was equipped with the I350 and had the identical LA=
CP
> > configuration. The new box also has better CPU with more cores (i.e. 24
> > cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3.
> >
>
> =E2=80=8B200K PPS, and even 350K PPS are very low value indeed.
> On a Intel Xeon L5630 (4 cores only) with one X540-AT2=E2=80=8B
>
> =E2=80=8B(then 2 10Gigabit ports)=E2=80=8B I've reached about 1.8Mpps (fa=
stforwarding
> enabled) [1].
> But my setup didn't use lagg(4): Can you disable lagg configuration and
> re-measure your performance without lagg ?
>
> Do you let Intel NIC drivers using 8 queues for port too?
> In my use case (forwarding smallest UDP packet size), I obtain better
> behaviour by limiting NIC queues to 4 (hw.ix.num_queues or
> hw.ixgbe.num_queues, don't remember) if my system had 8 cores. And this
> with Gigabit Intel[2] or Chelsio NIC [3].
>
> Don't forget to disable TSO and LRO too.
>
> =E2=80=8BRegards,
>
> Olivier
>
> [1]
>
> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_an_=
ibm_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs
> [2]
>
> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_s=
uperserver_5018a-ftn4#graph1
> [3]
>
> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_h=
p_proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr#redu=
cing_nic_queues
>
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
>
>
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1931183673.3234634.1439342909940.JavaMail.yahoo>