From owner-freebsd-performance@FreeBSD.ORG Mon Aug 3 08:46:27 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9CB76106564A for ; Mon, 3 Aug 2009 08:46:27 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Received: from GRFEDG701BA020.telecomitalia.it (grfedg701ba020.telecomitalia.it [156.54.233.200]) by mx1.freebsd.org (Postfix) with ESMTP id 259FB8FC24 for ; Mon, 3 Aug 2009 08:46:26 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Received: from GRFHUB703BA020.griffon.local (10.188.101.113) by GRFEDG701BA020.telecomitalia.it (10.188.45.100) with Microsoft SMTP Server (TLS) id 8.1.340.0; Mon, 3 Aug 2009 10:46:25 +0200 Received: from GRFMBX702BA020.griffon.local ([10.188.101.11]) by GRFHUB703BA020.griffon.local ([10.188.101.113]) with mapi; Mon, 3 Aug 2009 10:46:25 +0200 From: Invernizzi Fabrizio To: "freebsd-performance@freebsd.org" Date: Mon, 3 Aug 2009 10:46:23 +0200 Thread-Topic: Test on 10GBE Intel based network card Thread-Index: AcoUFuFzAEm0esyBRsKS9hPjzf2uPg== Message-ID: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> Accept-Language: it-IT Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: it-IT Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2009 08:46:28 -0000 Hi all I am doing some tests on a BSD system with a 10gbe Intel based network card= and I have some doubts about the configuration since the performance I am = experiencing looks (very) poor. This is the system I am doing test on: - HP 380 G5 (XEON X5420, CPU speed: 2.50GHz, BUS speed: 1333 MHz, L2 cache = size: 12 MB, L2 cache speed: 2,5 GHz) with 1 quad-core installed. - Network card: Silicom PE10G2i-LR - Dual Port Fiber (LR) 10 Gigabit Ethern= et PCI Express Server Adapter Intel=AE based (chip 82598EB). Driver ixgbe-1.8.6 - FreeBSD 7.2-RELEASE (64 bit) with this options compiled in the kernel options ZERO_COPY_SOCKETS # Turn on zero copy se= nd code options HZ=3D1000 options BPF_JITTER I worked on the driver settings in order to have big TX/RX rings and low in= terrupt rate (traffic latency is not an issue). In order to tune up the system i started with some echo request tests. These are the maximum Bandwidths I can send without loss: - 64 byte packets: 312 Mbps (1,64% CPU idle) - 512 byte packets: 2117 Mbps (1,63% CPU idle) - 1492 byte packets: 5525 Mbps (1,93% CPU idle) Am I right considering these figures lower than expected? The system is just managing network traffic! Now I have started with netgraph tests, in particular with ng_bpf and the o= verall system is going even worst. I sent some HTTP traffic (597 bytes-long packets) and I configured an ng_bp= f to filter TCP traffic out from the incoming interface (ix0). If I use the ngctl msg to see counters on the ng_bpf node, I see extremely = poor performance: - Sending 96Mbps of this traffic I figured out 0.1% packet loss. This looks= very strange. May be some counter bug? - Sending 5500Mbps, the netgraph (not the network card driver) is loosing 2= 1% of the number of sent packets. See below a snapshot of the CPU load unde= r traffic load CPU: 0.0% user, 0.0% nice, 87.0% system, 9.1% interrupt, 3.9% idle Mem: 16M Active, 317M Inact, 366M Wired, 108K Cache, 399M Buf, 7222M Free Swap: 2048M Total, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12 root 1 171 ki31 0K 16K RUN 2 20.2H 68.80% idle: c= pu2 11 root 1 171 ki31 0K 16K RUN 3 20.1H 64.70% idle: c= pu3 14 root 1 171 ki31 0K 16K RUN 0 20.2H 64.26% idle: c= pu0 13 root 1 171 ki31 0K 16K RUN 1 20.2H 63.67% idle: c= pu1 38 root 1 -68 - 0K 16K CPU1 1 1:28 34.67% ix0 rxq 40 root 1 -68 - 0K 16K CPU2 0 1:26 34.18% ix0 rxq 34 root 1 -68 - 0K 16K CPU3 3 1:27 34.08% ix0 rxq 36 root 1 -68 - 0K 16K RUN 2 1:26 34.08% ix0 rxq 33 root 1 -68 - 0K 16K WAIT 3 0:40 4.05% irq260:= ix0 39 root 1 -68 - 0K 16K WAIT 2 0:41 3.96% irq263:= ix0 35 root 1 -68 - 0K 16K WAIT 0 0:39 3.66% irq261:= ix0 37 root 1 -68 - 0K 16K WAIT 1 0:42 3.47% irq262:= ix0 16 root 1 -32 - 0K 16K WAIT 0 14:53 2.49% swi4: c= lock sio Am I missing something? Does someone know some (more) system tuning to have higher traffic rate sup= ported? Any help is greatly appreciated. Fabrizio ------------------------------------------------------------------ Telecom Italia Fabrizio INVERNIZZI Technology - TILAB Accesso Fisso e Trasporto Via Reiss Romoli, 274 10148 Torino Tel. +39 011 2285497 Mob. +39 3316001344 Fax +39 06 41867287 Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle per= sone indicate. La diffusione, copia o qualsiasi altra azione derivante dall= a conoscenza di queste informazioni sono rigorosamente vietate. Qualora abb= iate ricevuto questo documento per errore siete cortesemente pregati di dar= ne immediata comunicazione al mittente e di provvedere alla sua distruzione= , Grazie. This e-mail and any attachments is confidential and may contain privileged = information intended for the addressee(s) only. Dissemination, copying, pri= nting or use by anybody else is unauthorised. If you are not the intended r= ecipient, please delete this message and any attachments and advise the sen= der by return e-mail, Thanks. From owner-freebsd-performance@FreeBSD.ORG Mon Aug 3 09:41:42 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9645A106566B for ; Mon, 3 Aug 2009 09:41:42 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id E29C98FC16 for ; Mon, 3 Aug 2009 09:41:41 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id C84FE1B10F00; Mon, 3 Aug 2009 11:24:26 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on malcho.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.5 Received: from maylo.moneybookers.com (maylo.dev.moneybookers.net [192.168.3.20]) by blah.sun-fish.com (Postfix) with ESMTP id 27F8D1B10EDC; Mon, 3 Aug 2009 11:24:26 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by maylo.moneybookers.com (Postfix) with ESMTP id 1E3B637BF879; Mon, 3 Aug 2009 11:24:26 +0200 (CEST) Received: from maylo.moneybookers.com ([127.0.0.1]) by localhost (maylo.dev.moneybookers.net [127.0.0.1]) (amavisd-new, port 10024) with LMTP id XL9jvU-z85Zc; Mon, 3 Aug 2009 11:23:54 +0200 (CEST) Received: from hater.cmotd.com (hater.cmotd.com [192.168.3.125]) by maylo.moneybookers.com (Postfix) with ESMTP id 972F237BF84E; Mon, 3 Aug 2009 11:22:23 +0200 (CEST) Message-Id: <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> From: Stefan Lambrev To: Invernizzi Fabrizio In-Reply-To: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Apple Message framework v935.3) Date: Mon, 3 Aug 2009 12:22:23 +0300 References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> X-Mailer: Apple Mail (2.935.3) X-Virus-Scanned: ClamAV 0.94/9645/Mon Aug 3 00:11:36 2009 on blah.cmotd.com X-Virus-Status: Clean Cc: "freebsd-performance@freebsd.org" Subject: Re: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2009 09:41:42 -0000 Hi, The limitation that you see is about the max number of packets that =20 FreeBSD can handle - it looks like your best performance is reached at =20= 64 byte packets? Am I correct that the maximum you can reach is around 639,000 packets =20= per second? Also you are not routing the traffic, but instead the server handles =20 the requests itself and eat CPU to reply? On Aug 3, 2009, at 11:46 AM, Invernizzi Fabrizio wrote: > Hi all > > I am doing some tests on a BSD system with a 10gbe Intel based =20 > network card and I have some doubts about the configuration since =20 > the performance I am experiencing looks (very) poor. > > This is the system I am doing test on: > > > > - HP 380 G5 (XEON X5420, CPU speed: 2.50GHz, BUS speed: 1333 MHz, L2 =20= > cache size: 12 MB, L2 cache speed: 2,5 GHz) with 1 quad-core =20 > installed. > > - Network card: Silicom PE10G2i-LR - Dual Port Fiber (LR) 10 Gigabit =20= > Ethernet PCI Express Server Adapter Intel=AE based (chip 82598EB). > Driver ixgbe-1.8.6 > > - FreeBSD 7.2-RELEASE (64 bit) with this options compiled in the =20 > kernel > options ZERO_COPY_SOCKETS # Turn on zero =20 > copy send code > options HZ=3D1000 > options BPF_JITTER > > > > I worked on the driver settings in order to have big TX/RX rings and =20= > low interrupt rate (traffic latency is not an issue). > > > > In order to tune up the system i started with some echo request tests. > > These are the maximum Bandwidths I can send without loss: > > - 64 byte packets: 312 Mbps (1,64% CPU idle) > > - 512 byte packets: 2117 Mbps (1,63% CPU idle) > > - 1492 byte packets: 5525 Mbps (1,93% CPU idle) > > > > Am I right considering these figures lower than expected? > The system is just managing network traffic! > > > > Now I have started with netgraph tests, in particular with ng_bpf =20 > and the overall system is going even worst. > > I sent some HTTP traffic (597 bytes-long packets) and I configured =20 > an ng_bpf to filter TCP traffic out from the incoming interface (ix0). > > If I use the ngctl msg to see counters on the ng_bpf node, I see =20 > extremely poor performance: > > > > - Sending 96Mbps of this traffic I figured out 0.1% packet loss. =20 > This looks very strange. May be some counter bug? > > - Sending 5500Mbps, the netgraph (not the network card driver) is =20 > loosing 21% of the number of sent packets. See below a snapshot of =20 > the CPU load under traffic load > > > > CPU: 0.0% user, 0.0% nice, 87.0% system, 9.1% interrupt, 3.9% idle > > Mem: 16M Active, 317M Inact, 366M Wired, 108K Cache, 399M Buf, 7222M =20= > Free > > Swap: 2048M Total, 2048M Free > > > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU =20 > COMMAND > > 12 root 1 171 ki31 0K 16K RUN 2 20.2H 68.80% =20 > idle: cpu2 > > 11 root 1 171 ki31 0K 16K RUN 3 20.1H 64.70% =20 > idle: cpu3 > > 14 root 1 171 ki31 0K 16K RUN 0 20.2H 64.26% =20 > idle: cpu0 > > 13 root 1 171 ki31 0K 16K RUN 1 20.2H 63.67% =20 > idle: cpu1 > > 38 root 1 -68 - 0K 16K CPU1 1 1:28 34.67% =20 > ix0 rxq > > 40 root 1 -68 - 0K 16K CPU2 0 1:26 34.18% =20 > ix0 rxq > > 34 root 1 -68 - 0K 16K CPU3 3 1:27 34.08% =20 > ix0 rxq > > 36 root 1 -68 - 0K 16K RUN 2 1:26 34.08% =20 > ix0 rxq > > 33 root 1 -68 - 0K 16K WAIT 3 0:40 4.05% =20 > irq260: ix0 > > 39 root 1 -68 - 0K 16K WAIT 2 0:41 3.96% =20 > irq263: ix0 > > 35 root 1 -68 - 0K 16K WAIT 0 0:39 3.66% =20 > irq261: ix0 > > 37 root 1 -68 - 0K 16K WAIT 1 0:42 3.47% =20 > irq262: ix0 > > 16 root 1 -32 - 0K 16K WAIT 0 14:53 2.49% =20 > swi4: clock sio > > > > > > > > Am I missing something? > > Does someone know some (more) system tuning to have higher traffic =20 > rate supported? > > > > Any help is greatly appreciated. > > > > Fabrizio > > > > > > ------------------------------------------------------------------ > Telecom Italia > Fabrizio INVERNIZZI > Technology - TILAB > Accesso Fisso e Trasporto > Via Reiss Romoli, 274 10148 Torino > Tel. +39 011 2285497 > Mob. +39 3316001344 > Fax +39 06 41867287 > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente =20 > alle persone indicate. La diffusione, copia o qualsiasi altra azione =20= > derivante dalla conoscenza di queste informazioni sono rigorosamente =20= > vietate. Qualora abbiate ricevuto questo documento per errore siete =20= > cortesemente pregati di darne immediata comunicazione al mittente e =20= > di provvedere alla sua distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain =20 > privileged information intended for the addressee(s) only. =20 > Dissemination, copying, printing or use by anybody else is =20 > unauthorised. If you are not the intended recipient, please delete =20 > this message and any attachments and advise the sender by return e-=20 > mail, Thanks. > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to = "freebsd-performance-unsubscribe@freebsd.org=20 > " -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-performance@FreeBSD.ORG Mon Aug 3 09:53:57 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BB25D1065675 for ; Mon, 3 Aug 2009 09:53:57 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Received: from GRFEDG702BA020.telecomitalia.it (grfedg702ba020.telecomitalia.it [156.54.233.201]) by mx1.freebsd.org (Postfix) with ESMTP id F01348FC1C for ; Mon, 3 Aug 2009 09:53:56 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Received: from GRFHUB701BA020.griffon.local (10.188.101.111) by GRFEDG702BA020.telecomitalia.it (10.188.45.101) with Microsoft SMTP Server (TLS) id 8.1.340.0; Mon, 3 Aug 2009 11:53:54 +0200 Received: from GRFMBX702BA020.griffon.local ([10.188.101.11]) by grfhub701ba020.griffon.local ([10.188.101.111]) with mapi; Mon, 3 Aug 2009 11:53:53 +0200 From: Invernizzi Fabrizio To: Stefan Lambrev Date: Mon, 3 Aug 2009 11:53:52 +0200 Thread-Topic: Test on 10GBE Intel based network card Thread-Index: AcoUHDpuARTRxgSESvWTT7FcXadVNQAAwT5Q Message-ID: <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> In-Reply-To: <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> Accept-Language: it-IT, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: it-IT, en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "freebsd-performance@freebsd.org" Subject: RE: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2009 09:53:58 -0000 Hi > -----Original Message----- > From: Stefan Lambrev [mailto:stefan.lambrev@moneybookers.com] > Sent: luned=EC 3 agosto 2009 11.22 > To: Invernizzi Fabrizio > Cc: freebsd-performance@freebsd.org > Subject: Re: Test on 10GBE Intel based network card > > Hi, > > The limitation that you see is about the max number of packets that > FreeBSD can handle - it looks like your best performance is reached at > 64 byte packets? If you are meaning in term of Packet per second, you are right. These are t= he packet per second measured during tests: 64 byte: 610119 Pps 512 byte: 516917 Pps 1492 byte: 464962 Pps > Am I correct that the maximum you can reach is around 639,000 packets > per second? Yes, as you can see the maximum is 610119 Pps. Where does this limit come from? > Also you are not routing the traffic, but instead the server handles > the requests itself and eat CPU to reply? Correct. In these first tests I want to "tune" the system, so I am using th= e (let me say) worst scenario. > > _______________________________________________ > > freebsd-performance@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > > To unsubscribe, send any mail to "freebsd-performance- > unsubscribe@freebsd.org > > " > > -- > Best Wishes, > Stefan Lambrev > ICQ# 24134177 > > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle per= sone indicate. La diffusione, copia o qualsiasi altra azione derivante dall= a conoscenza di queste informazioni sono rigorosamente vietate. Qualora abb= iate ricevuto questo documento per errore siete cortesemente pregati di dar= ne immediata comunicazione al mittente e di provvedere alla sua distruzione= , Grazie. This e-mail and any attachments is confidential and may contain privileged = information intended for the addressee(s) only. Dissemination, copying, pri= nting or use by anybody else is unauthorised. If you are not the intended r= ecipient, please delete this message and any attachments and advise the sen= der by return e-mail, Thanks. From owner-freebsd-performance@FreeBSD.ORG Mon Aug 3 10:15:13 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 80B7D106566C for ; Mon, 3 Aug 2009 10:15:13 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id C895F8FC08 for ; Mon, 3 Aug 2009 10:15:12 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 90DA71B10EF1; Mon, 3 Aug 2009 12:15:11 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on malcho.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.5 Received: from maylo.moneybookers.com (maylo.dev.moneybookers.net [192.168.3.20]) by blah.sun-fish.com (Postfix) with ESMTP id 002621B10EDC; Mon, 3 Aug 2009 12:15:10 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by maylo.moneybookers.com (Postfix) with ESMTP id B199E37BF837; Mon, 3 Aug 2009 12:15:10 +0200 (CEST) Received: from maylo.moneybookers.com ([127.0.0.1]) by localhost (maylo.dev.moneybookers.net [127.0.0.1]) (amavisd-new, port 10024) with LMTP id OqREPfadvFHs; Mon, 3 Aug 2009 12:15:08 +0200 (CEST) Received: from hater.cmotd.com (hater.cmotd.com [192.168.3.125]) by maylo.moneybookers.com (Postfix) with ESMTP id 2CB6E37BF847; Mon, 3 Aug 2009 12:15:08 +0200 (CEST) Message-Id: <18AAC16B-3CC0-4C70-A009-00A325AB5932@moneybookers.com> From: Stefan Lambrev To: Invernizzi Fabrizio In-Reply-To: <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Apple Message framework v935.3) Date: Mon, 3 Aug 2009 13:15:08 +0300 References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> X-Mailer: Apple Mail (2.935.3) X-Virus-Scanned: ClamAV 0.94/9645/Mon Aug 3 00:11:36 2009 on blah.cmotd.com X-Virus-Status: Clean Cc: "freebsd-performance@freebsd.org" Subject: Re: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2009 10:15:13 -0000 Hi, On Aug 3, 2009, at 12:53 PM, Invernizzi Fabrizio wrote: > Hi > > >> -----Original Message----- >> From: Stefan Lambrev [mailto:stefan.lambrev@moneybookers.com] >> Sent: luned=EC 3 agosto 2009 11.22 >> To: Invernizzi Fabrizio >> Cc: freebsd-performance@freebsd.org >> Subject: Re: Test on 10GBE Intel based network card >> >> Hi, >> >> The limitation that you see is about the max number of packets that >> FreeBSD can handle - it looks like your best performance is reached =20= >> at >> 64 byte packets? > > If you are meaning in term of Packet per second, you are right. =20 > These are the packet per second measured during tests: > > 64 byte: 610119 Pps > 512 byte: 516917 Pps > 1492 byte: 464962 Pps > > >> Am I correct that the maximum you can reach is around 639,000 packets >> per second? > > Yes, as you can see the maximum is 610119 Pps. > Where does this limit come from? I duno - the tests I did before were with SYN packets (random source) =20= which was my worst scenario, and the server CPU were busy generating MD5 check sums for =20 "syncache" (around 35% of the time). If I have to compare my results with your, you beat me with factor =20 2.5, may be because you use ICMP for the test and your processor is better then my test stations :) Also my experience is only with gigabit cards (em driver) and FreeBSD =20= 7.something_before_1 where the em thread was eating 100% cpu. If you are lucky LOCK_PROFILING(9) will help you to see where the CPUs =20= spend their time, if not you will see kernel panic :) Once problematic locks identified they can be reworked, but I think =20 the first part is already done and work on the second already started. In my experience increasing hw.em.rxd and hw.em.txd yelled better =20 results, but I think ixgb already comes tuned by default as it still doesn't have to support such a large number of different =20 cards. Also at the time of my tests there were not support for multi queues =20 in the OS even if they were supported by the HW, which is changed in =20 7.2 (?) > >> Also you are not routing the traffic, but instead the server handles >> the requests itself and eat CPU to reply? > > Correct. In these first tests I want to "tune" the system, so I am =20 > using the (let me say) worst scenario. > > >>> _______________________________________________ >>> freebsd-performance@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >>> To unsubscribe, send any mail to "freebsd-performance- >> unsubscribe@freebsd.org >>> " >> >> -- >> Best Wishes, >> Stefan Lambrev >> ICQ# 24134177 >> >> >> >> > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente =20 > alle persone indicate. La diffusione, copia o qualsiasi altra azione =20= > derivante dalla conoscenza di queste informazioni sono rigorosamente =20= > vietate. Qualora abbiate ricevuto questo documento per errore siete =20= > cortesemente pregati di darne immediata comunicazione al mittente e =20= > di provvedere alla sua distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain =20 > privileged information intended for the addressee(s) only. =20 > Dissemination, copying, printing or use by anybody else is =20 > unauthorised. If you are not the intended recipient, please delete =20 > this message and any attachments and advise the sender by return e-=20 > mail, Thanks. > -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-performance@FreeBSD.ORG Mon Aug 3 10:17:52 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B9905106566C for ; Mon, 3 Aug 2009 10:17:52 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Received: from GRFEDG701BA020.telecomitalia.it (grfedg701ba020.telecomitalia.it [156.54.233.200]) by mx1.freebsd.org (Postfix) with ESMTP id 2BAC58FC16 for ; Mon, 3 Aug 2009 10:17:52 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Received: from GRFHUB702BA020.griffon.local (10.188.101.112) by GRFEDG701BA020.telecomitalia.it (10.188.45.100) with Microsoft SMTP Server (TLS) id 8.1.340.0; Mon, 3 Aug 2009 12:17:51 +0200 Received: from GRFMBX702BA020.griffon.local ([10.188.101.11]) by GRFHUB702BA020.griffon.local ([10.188.101.112]) with mapi; Mon, 3 Aug 2009 12:17:50 +0200 From: Invernizzi Fabrizio To: =?iso-8859-1?Q?Istv=E1n?= Date: Mon, 3 Aug 2009 12:17:48 +0200 Thread-Topic: Test on 10GBE Intel based network card Thread-Index: AcoUIWdeWsXwwxKeTc2y0JXhwyqdEQAAhwvQ Message-ID: <36A93B31228D3B49B691AD31652BCAE9A4560DF961@GRFMBX702BA020.griffon.local> References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> In-Reply-To: Accept-Language: it-IT Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: it-IT Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "freebsd-performance@freebsd.org" , Stefan Lambrev Subject: RE: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2009 10:17:53 -0000 HI These are the sysctl vars you pointed out kern.ipc.somaxconn: 128 net.inet.tcp.recvspace: 65536 net.inet.tcp.sendspace: 32768 kern.ipc.shmmax: 33554432 kern.ipc.shmmni: 192 kern.ipc.shmseg: 128 kern.ipc.semmni: 10 net.local.stream.sendspace: 8192 net.local.stream.recvspace: 8192 net.inet.tcp.local_slowstart_flightsize: 4 net.inet.tcp.nolocaltimewait: 0 net.inet.tcp.hostcache.expire: 3600 kern.maxusers: 384 kern.ipc.nmbclusters: 65635 kern.ipc.maxsockets: 65635 kern.ipc.maxsockbuf: 262144 net.inet.tcp.tcbhashsize: 512 net.inet.tcp.hostcache.hashsize: 512 Fabrizio ________________________________________ From: Istv=E1n [mailto:leccine@gmail.com] Sent: luned=EC 3 agosto 2009 12.02 To: Invernizzi Fabrizio Cc: Stefan Lambrev; freebsd-performance@freebsd.org Subject: Re: Test on 10GBE Intel based network card what about your sysctls? I would like to see what have you done yet. Here is a previous conversation about that: http://www.mail-archive.com/fre= ebsd-performance@freebsd.org/msg02293.html Actually I am not aware of the changes in 7.2 or what could be the best val= ue set for you, worth to try to fine tune these. Regards, Istvan On Mon, Aug 3, 2009 at 10:53 AM, Invernizzi Fabrizio wrote: Hi > -----Original Message----- > From: Stefan Lambrev [mailto:stefan.lambrev@moneybookers.com] > Sent: luned=EC 3 agosto 2009 11.22 > To: Invernizzi Fabrizio > Cc: freebsd-performance@freebsd.org > Subject: Re: Test on 10GBE Intel based network card > > Hi, > > The limitation that you see is about the max number of packets that > FreeBSD can handle - it looks like your best performance is reached at > 64 byte packets? If you are meaning in term of Packet per second, you are right. These are t= he packet per second measured during tests: 64 byte: 610119 Pps 512 byte: 516917 Pps 1492 byte: 464962 Pps > Am I correct that the maximum you can reach is around 639,000 packets > per second? Yes, as you can see the maximum is 610119 Pps. Where does this limit come from? > Also you are not routing the traffic, but instead the server handles > the requests itself and eat CPU to reply? Correct. In these first tests I want to "tune" the system, so I am using th= e (let me say) worst scenario. > > _______________________________________________ > > freebsd-performance@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > > To unsubscribe, send any mail to "freebsd-performance- > unsubscribe@freebsd.org > > " > > -- > Best Wishes, > Stefan Lambrev > ICQ# 24134177 > > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle per= sone indicate. La diffusione, copia o qualsiasi altra azione derivante dall= a conoscenza di queste informazioni sono rigorosamente vietate. Qualora abb= iate ricevuto questo documento per errore siete cortesemente pregati di dar= ne immediata comunicazione al mittente e di provvedere alla sua distruzione= , Grazie. This e-mail and any attachments is confidential and may contain privileged = information intended for the addressee(s) only. Dissemination, copying, pri= nting or use by anybody else is unauthorised. If you are not the intended r= ecipient, please delete this message and any attachments and advise the sen= der by return e-mail, Thanks. _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.o= rg" -- the sun shines for all Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle per= sone indicate. La diffusione, copia o qualsiasi altra azione derivante dall= a conoscenza di queste informazioni sono rigorosamente vietate. Qualora abb= iate ricevuto questo documento per errore siete cortesemente pregati di dar= ne immediata comunicazione al mittente e di provvedere alla sua distruzione= , Grazie. This e-mail and any attachments is confidential and may contain privileged = information intended for the addressee(s) only. Dissemination, copying, pri= nting or use by anybody else is unauthorised. If you are not the intended r= ecipient, please delete this message and any attachments and advise the sen= der by return e-mail, Thanks. From owner-freebsd-performance@FreeBSD.ORG Mon Aug 3 10:20:59 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0C6F51065672 for ; Mon, 3 Aug 2009 10:20:59 +0000 (UTC) (envelope-from leccine@gmail.com) Received: from mail-fx0-f210.google.com (mail-fx0-f210.google.com [209.85.220.210]) by mx1.freebsd.org (Postfix) with ESMTP id 5E6988FC1D for ; Mon, 3 Aug 2009 10:20:58 +0000 (UTC) (envelope-from leccine@gmail.com) Received: by fxm6 with SMTP id 6so1236211fxm.43 for ; Mon, 03 Aug 2009 03:20:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=w9x/WPFVvHBWERAGke7aGa71YFYAVjwxRB/swwlCsiw=; b=TrAa2Fwe4jgCqQmKl29jYC9dCeH5bvooBJZ9HaRNyyVKuwwrG+ExnpQODQvLzbHVNT kxwJ/nNOWs11e9WK3nD0VMjWWyFHA82NAjMQOkdI5jyZ4XB12W2C/xtPStnxlNZ6sobJ t3y6C60HK40zDJAGvXu7SM2Gx6Nwe6fFuh900= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=Z6LRNWVWtjluUIzMN+8fMkAyhI8k09idg12P/jMMrCcrpS2qi1l9DySyDk3T5UOYdy eSeDOa+PXzmRoVsFZApjTXBUzSckYgrK+f8Qgr6fdPVr+1Uuhbu5y+kop8NB3oMaQa7y IJ442f0Psenp3D+vXl38zSJkL2jmd97ARw+KA= MIME-Version: 1.0 Received: by 10.204.102.15 with SMTP id e15mr7990108bko.196.1249294857065; Mon, 03 Aug 2009 03:20:57 -0700 (PDT) In-Reply-To: <36A93B31228D3B49B691AD31652BCAE9A4560DF961@GRFMBX702BA020.griffon.local> References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> <36A93B31228D3B49B691AD31652BCAE9A4560DF961@GRFMBX702BA020.griffon.local> Date: Mon, 3 Aug 2009 11:20:57 +0100 Message-ID: From: =?ISO-8859-1?B?SXN0duFu?= To: Invernizzi Fabrizio Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: "freebsd-performance@freebsd.org" , Stefan Lambrev Subject: Re: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2009 10:20:59 -0000 Nice, I guess to analyse the bottlenecks you might want to use dtrace....If it is working with 7.2, I am not sure about that. Regards, Istvan On Mon, Aug 3, 2009 at 11:17 AM, Invernizzi Fabrizio < fabrizio.invernizzi@telecomitalia.it> wrote: > HI > > These are the sysctl vars you pointed out > > kern.ipc.somaxconn: 128 > net.inet.tcp.recvspace: 65536 > net.inet.tcp.sendspace: 32768 > kern.ipc.shmmax: 33554432 > kern.ipc.shmmni: 192 > kern.ipc.shmseg: 128 > kern.ipc.semmni: 10 > net.local.stream.sendspace: 8192 > net.local.stream.recvspace: 8192 > net.inet.tcp.local_slowstart_flightsize: 4 > net.inet.tcp.nolocaltimewait: 0 > net.inet.tcp.hostcache.expire: 3600 > > kern.maxusers: 384 > kern.ipc.nmbclusters: 65635 > kern.ipc.maxsockets: 65635 > kern.ipc.maxsockbuf: 262144 > net.inet.tcp.tcbhashsize: 512 > net.inet.tcp.hostcache.hashsize: 512 > > > Fabrizio > > ________________________________________ > From: Istv=E1n [mailto:leccine@gmail.com] > Sent: luned=EC 3 agosto 2009 12.02 > To: Invernizzi Fabrizio > Cc: Stefan Lambrev; freebsd-performance@freebsd.org > Subject: Re: Test on 10GBE Intel based network card > > what about your sysctls? > > I would like to see what have you done yet. > > Here is a previous conversation about that: > http://www.mail-archive.com/freebsd-performance@freebsd.org/msg02293.html > > Actually I am not aware of the changes in 7.2 or what could be the best > value set for you, worth to try to fine tune these. > > Regards, > Istvan > On Mon, Aug 3, 2009 at 10:53 AM, Invernizzi Fabrizio < > fabrizio.invernizzi@telecomitalia.it> wrote: > Hi > > > > -----Original Message----- > > From: Stefan Lambrev [mailto:stefan.lambrev@moneybookers.com] > > Sent: luned=EC 3 agosto 2009 11.22 > > To: Invernizzi Fabrizio > > Cc: freebsd-performance@freebsd.org > > Subject: Re: Test on 10GBE Intel based network card > > > > Hi, > > > > The limitation that you see is about the max number of packets that > > FreeBSD can handle - it looks like your best performance is reached at > > 64 byte packets? > If you are meaning in term of Packet per second, you are right. These are > the packet per second measured during tests: > > 64 byte: 610119 Pps > 512 byte: 516917 Pps > 1492 byte: 464962 Pps > > > > Am I correct that the maximum you can reach is around 639,000 packets > > per second? > Yes, as you can see the maximum is 610119 Pps. > Where does this limit come from? > > > Also you are not routing the traffic, but instead the server handles > > the requests itself and eat CPU to reply? > Correct. In these first tests I want to "tune" the system, so I am using > the (let me say) worst scenario. > > > > > _______________________________________________ > > > freebsd-performance@freebsd.org mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > > > To unsubscribe, send any mail to "freebsd-performance- > > unsubscribe@freebsd.org > > > " > > > > -- > > Best Wishes, > > Stefan Lambrev > > ICQ# 24134177 > > > > > > > > > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle > persone indicate. La diffusione, copia o qualsiasi altra azione derivante > dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualo= ra > abbiate ricevuto questo documento per errore siete cortesemente pregati d= i > darne immediata comunicazione al mittente e di provvedere alla sua > distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain privilege= d > information intended for the addressee(s) only. Dissemination, copying, > printing or use by anybody else is unauthorised. If you are not the inten= ded > recipient, please delete this message and any attachments and advise the > sender by return e-mail, Thanks. > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@freebsd.org" > > > > -- > the sun shines for all > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle > persone indicate. La diffusione, copia o qualsiasi altra azione derivante > dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualo= ra > abbiate ricevuto questo documento per errore siete cortesemente pregati d= i > darne immediata comunicazione al mittente e di provvedere alla sua > distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain privilege= d > information intended for the addressee(s) only. Dissemination, copying, > printing or use by anybody else is unauthorised. If you are not the inten= ded > recipient, please delete this message and any attachments and advise the > sender by return e-mail, Thanks. > > --=20 the sun shines for all From owner-freebsd-performance@FreeBSD.ORG Mon Aug 3 10:22:36 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5B5C91065673 for ; Mon, 3 Aug 2009 10:22:36 +0000 (UTC) (envelope-from leccine@gmail.com) Received: from mail-fx0-f210.google.com (mail-fx0-f210.google.com [209.85.220.210]) by mx1.freebsd.org (Postfix) with ESMTP id D1F368FC22 for ; Mon, 3 Aug 2009 10:22:35 +0000 (UTC) (envelope-from leccine@gmail.com) Received: by fxm6 with SMTP id 6so1236849fxm.43 for ; Mon, 03 Aug 2009 03:22:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=Jkt6fXAaB3iW/pwjpnLh0K27LHS+ltY6+WTDeMHUDXw=; b=TmSgM5GOlqYE4E4RIsTKIG7V9V/prUj3c2ol7Yu4Av4o1GwfDf2267kafPOg3JNkug mvoVleY3p5OtHCbJaaWQdGFOl1Nw1cT0C9C2CaIgsWeQUBYcH5Z9gmfKcyTzwOohD6Zt B+gsFuwx8p+egFoFbl4vUCKJbCo2tkBQM9CaA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=B/IE6xHRB0kRm87RIwuP7QkAW0yF3R06j9yVaRcv+I2YWF5b6OG6EDG8q2EXUWooR0 Zu+IIbCw66+YmoYGOQmit1CrsqAADjpC1eIewtKClpD/Wx4JiDEKqNwNPMMV8IUeiuKV nrrBcDvIbg/XaGhWWiQCGsXeNUox+A3hO+uso= MIME-Version: 1.0 Received: by 10.204.98.13 with SMTP id o13mr2682969bkn.80.1249293701961; Mon, 03 Aug 2009 03:01:41 -0700 (PDT) In-Reply-To: <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> Date: Mon, 3 Aug 2009 11:01:41 +0100 Message-ID: From: =?ISO-8859-1?B?SXN0duFu?= To: Invernizzi Fabrizio Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: "freebsd-performance@freebsd.org" , Stefan Lambrev Subject: Re: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2009 10:22:36 -0000 what about your sysctls? I would like to see what have you done yet. Here is a previous conversation about that: http://www.mail-archive.com/freebsd-performance@freebsd.org/msg02293.html = Actually I am not aware of the changes in 7.2 or what could be the best value set fo= r you, worth to try to fine tune these. Regards, Istvan On Mon, Aug 3, 2009 at 10:53 AM, Invernizzi Fabrizio < fabrizio.invernizzi@telecomitalia.it> wrote: > Hi > > > > -----Original Message----- > > From: Stefan Lambrev [mailto:stefan.lambrev@moneybookers.com] > > Sent: luned=EC 3 agosto 2009 11.22 > > To: Invernizzi Fabrizio > > Cc: freebsd-performance@freebsd.org > > Subject: Re: Test on 10GBE Intel based network card > > > > Hi, > > > > The limitation that you see is about the max number of packets that > > FreeBSD can handle - it looks like your best performance is reached at > > 64 byte packets? > > If you are meaning in term of Packet per second, you are right. These are > the packet per second measured during tests: > > 64 byte: 610119 Pps > 512 byte: 516917 Pps > 1492 byte: 464962 Pps > > > > Am I correct that the maximum you can reach is around 639,000 packets > > per second? > > Yes, as you can see the maximum is 610119 Pps. > Where does this limit come from? > > > Also you are not routing the traffic, but instead the server handles > > the requests itself and eat CPU to reply? > > Correct. In these first tests I want to "tune" the system, so I am using > the (let me say) worst scenario. > > > > > _______________________________________________ > > > freebsd-performance@freebsd.org mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > > > To unsubscribe, send any mail to "freebsd-performance- > > unsubscribe@freebsd.org > > > " > > > > -- > > Best Wishes, > > Stefan Lambrev > > ICQ# 24134177 > > > > > > > > > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle > persone indicate. La diffusione, copia o qualsiasi altra azione derivante > dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualo= ra > abbiate ricevuto questo documento per errore siete cortesemente pregati d= i > darne immediata comunicazione al mittente e di provvedere alla sua > distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain privilege= d > information intended for the addressee(s) only. Dissemination, copying, > printing or use by anybody else is unauthorised. If you are not the inten= ded > recipient, please delete this message and any attachments and advise the > sender by return e-mail, Thanks. > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@freebsd.org" > --=20 the sun shines for all From owner-freebsd-performance@FreeBSD.ORG Mon Aug 3 10:27:00 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9CD3F1065674 for ; Mon, 3 Aug 2009 10:27:00 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Received: from GRFEDG702BA020.telecomitalia.it (grfedg702ba020.telecomitalia.it [156.54.233.201]) by mx1.freebsd.org (Postfix) with ESMTP id E80DF8FC12 for ; Mon, 3 Aug 2009 10:26:59 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Received: from GRFHUB701BA020.griffon.local (10.188.101.111) by GRFEDG702BA020.telecomitalia.it (10.188.45.101) with Microsoft SMTP Server (TLS) id 8.1.340.0; Mon, 3 Aug 2009 12:26:58 +0200 Received: from GRFMBX702BA020.griffon.local ([10.188.101.11]) by grfhub701ba020.griffon.local ([10.188.101.111]) with mapi; Mon, 3 Aug 2009 12:26:57 +0200 From: Invernizzi Fabrizio To: Stefan Lambrev Date: Mon, 3 Aug 2009 12:26:55 +0200 Thread-Topic: Test on 10GBE Intel based network card Thread-Index: AcoUI0uqyXnc4/suT/+J7T7yVslx/gAAKPbA Message-ID: <36A93B31228D3B49B691AD31652BCAE9A4560DF96A@GRFMBX702BA020.griffon.local> References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> <18AAC16B-3CC0-4C70-A009-00A325AB5932@moneybookers.com> In-Reply-To: <18AAC16B-3CC0-4C70-A009-00A325AB5932@moneybookers.com> Accept-Language: it-IT, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: it-IT, en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "freebsd-performance@freebsd.org" Subject: RE: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2009 10:27:00 -0000 > > If you are meaning in term of Packet per second, you are right. > > These are the packet per second measured during tests: > > > > 64 byte: 610119 Pps > > 512 byte: 516917 Pps > > 1492 byte: 464962 Pps > > > > > >> Am I correct that the maximum you can reach is around 639,000 packets > >> per second? > > > > Yes, as you can see the maximum is 610119 Pps. > > Where does this limit come from? > > I duno - the tests I did before were with SYN packets (random source) > which was my worst scenario, > and the server CPU were busy generating MD5 check sums for > "syncache" (around 35% of the time). > > If I have to compare my results with your, you beat me with factor > 2.5, may be because you use ICMP for the test > and your processor is better then my test stations :) > Also my experience is only with gigabit cards (em driver) and FreeBSD > 7.something_before_1 where the em thread was eating 100% cpu. > If you are lucky LOCK_PROFILING(9) will help you to see where the CPUs > spend their time, if not you will see kernel panic :) I will check, thanks for the hint. > Once problematic locks identified they can be reworked, but I think > the first part is already done > and work on the second already started. > > In my experience increasing hw.em.rxd and hw.em.txd yelled better > results, but I think ixgb already comes tuned by default > as it still doesn't have to support such a large number of different > cards. I did some tuning in the code of the driver e recompiled the kernel in orde= r to reduce context switching (interrupt mitigation) since the driver does = not support POLLING. > Also at the time of my tests there were not support for multi queues > in the OS even if they were supported by the HW, which is changed in > 7.2 (?) It looks like multi queue is working since I can see the load distributed o= ver the 4 cores. Fabrizio Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle per= sone indicate. La diffusione, copia o qualsiasi altra azione derivante dall= a conoscenza di queste informazioni sono rigorosamente vietate. Qualora abb= iate ricevuto questo documento per errore siete cortesemente pregati di dar= ne immediata comunicazione al mittente e di provvedere alla sua distruzione= , Grazie. This e-mail and any attachments is confidential and may contain privileged = information intended for the addressee(s) only. Dissemination, copying, pri= nting or use by anybody else is unauthorised. If you are not the intended r= ecipient, please delete this message and any attachments and advise the sen= der by return e-mail, Thanks. From owner-freebsd-performance@FreeBSD.ORG Mon Aug 3 15:49:24 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4DE6E1065672 for ; Mon, 3 Aug 2009 15:49:24 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.242]) by mx1.freebsd.org (Postfix) with ESMTP id F129C8FC23 for ; Mon, 3 Aug 2009 15:49:23 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: by an-out-0708.google.com with SMTP id d14so1652971and.13 for ; Mon, 03 Aug 2009 08:49:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=KmUDz6G+je493x6hItqS9Yp2sczbvTb4RMJOR8rIv3Y=; b=tYz3IYQNwrutzn0btpBXyxalyBS4tNbCJM0+hNeufO7ekv2pqf8lWAcwNnBombM9MI OS8WPzwimClzoOBIBTCbq0vzzNH0AquhtG1Kq9uZUumTUhxW+WvYXcZMNS37919PlK8Y dxCo5/mQno9e+6iENo4Wsk5SO5QY4srxpmmY0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=eHstekETyG0gdKKnMQFzgyW5TsCPuERyWncUn1jaBcFS96jTjU8NtluFeFElG/OarI ScC+lQ7eYJQfECSET4FEDUzuvHT0JAehfj7zaFl9z7uLPEC6z5Jm6HKeIK6Tx8CqEPPQ UQbhM/bUVlzYYKFW42mLGZTxpHMdm2Br749bQ= MIME-Version: 1.0 Received: by 10.100.163.11 with SMTP id l11mr3731573ane.161.1249312846255; Mon, 03 Aug 2009 08:20:46 -0700 (PDT) In-Reply-To: <36A93B31228D3B49B691AD31652BCAE9A4560DF96A@GRFMBX702BA020.griffon.local> References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> <18AAC16B-3CC0-4C70-A009-00A325AB5932@moneybookers.com> <36A93B31228D3B49B691AD31652BCAE9A4560DF96A@GRFMBX702BA020.griffon.local> Date: Mon, 3 Aug 2009 08:20:46 -0700 Message-ID: <2a41acea0908030820o40438f6erda78927733529a9@mail.gmail.com> From: Jack Vogel To: Invernizzi Fabrizio Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: "freebsd-performance@freebsd.org" , Stefan Lambrev Subject: Re: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2009 15:49:25 -0000 If you go to FreeBSD 8 you will get the improved stack code, and the RX/TX queue pairs will be pinned to cpus. It should improve performance. Make sure you have enough mbuf memory allocated, try increasing the descriptors. Jack On Mon, Aug 3, 2009 at 3:26 AM, Invernizzi Fabrizio < fabrizio.invernizzi@telecomitalia.it> wrote: > > > If you are meaning in term of Packet per second, you are right. > > > These are the packet per second measured during tests: > > > > > > 64 byte: 610119 Pps > > > 512 byte: 516917 Pps > > > 1492 byte: 464962 Pps > > > > > > > > >> Am I correct that the maximum you can reach is around 639,000 packets > > >> per second? > > > > > > Yes, as you can see the maximum is 610119 Pps. > > > Where does this limit come from? > > > > I duno - the tests I did before were with SYN packets (random source) > > which was my worst scenario, > > and the server CPU were busy generating MD5 check sums for > > "syncache" (around 35% of the time). > > > > If I have to compare my results with your, you beat me with factor > > 2.5, may be because you use ICMP for the test > > and your processor is better then my test stations :) > > Also my experience is only with gigabit cards (em driver) and FreeBSD > > 7.something_before_1 where the em thread was eating 100% cpu. > > If you are lucky LOCK_PROFILING(9) will help you to see where the CPUs > > spend their time, if not you will see kernel panic :) > > > I will check, thanks for the hint. > > > Once problematic locks identified they can be reworked, but I think > > the first part is already done > > and work on the second already started. > > > > In my experience increasing hw.em.rxd and hw.em.txd yelled better > > results, but I think ixgb already comes tuned by default > > as it still doesn't have to support such a large number of different > > cards. > > I did some tuning in the code of the driver e recompiled the kernel in > order to reduce context switching (interrupt mitigation) since the driver > does not support POLLING. > > > Also at the time of my tests there were not support for multi queues > > in the OS even if they were supported by the HW, which is changed in > > 7.2 (?) > > It looks like multi queue is working since I can see the load distributed > over the 4 cores. > > > Fabrizio > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle > persone indicate. La diffusione, copia o qualsiasi altra azione derivante > dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora > abbiate ricevuto questo documento per errore siete cortesemente pregati di > darne immediata comunicazione al mittente e di provvedere alla sua > distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain privileged > information intended for the addressee(s) only. Dissemination, copying, > printing or use by anybody else is unauthorised. If you are not the intended > recipient, please delete this message and any attachments and advise the > sender by return e-mail, Thanks. > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@freebsd.org" > From owner-freebsd-performance@FreeBSD.ORG Mon Aug 3 16:11:16 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D1157106566B for ; Mon, 3 Aug 2009 16:11:16 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outW.internet-mail-service.net (outw.internet-mail-service.net [216.240.47.246]) by mx1.freebsd.org (Postfix) with ESMTP id B22A98FC0C for ; Mon, 3 Aug 2009 16:11:16 +0000 (UTC) (envelope-from julian@elischer.org) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id C8687B7543; Mon, 3 Aug 2009 08:59:37 -0700 (PDT) X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id 316B62D6016; Mon, 3 Aug 2009 08:59:02 -0700 (PDT) Message-ID: <4A77094C.8030308@elischer.org> Date: Mon, 03 Aug 2009 08:59:08 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.22 (Macintosh/20090605) MIME-Version: 1.0 To: Invernizzi Fabrizio References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> In-Reply-To: <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Cc: "freebsd-performance@freebsd.org" , Stefan Lambrev Subject: Re: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2009 16:11:17 -0000 Invernizzi Fabrizio wrote: > Hi > > >> -----Original Message----- >> From: Stefan Lambrev [mailto:stefan.lambrev@moneybookers.com] >> Sent: luned́ 3 agosto 2009 11.22 >> To: Invernizzi Fabrizio >> Cc: freebsd-performance@freebsd.org >> Subject: Re: Test on 10GBE Intel based network card >> >> Hi, >> >> The limitation that you see is about the max number of packets that >> FreeBSD can handle - it looks like your best performance is reached at >> 64 byte packets? > > If you are meaning in term of Packet per second, you are right. These are the packet per second measured during tests: > > 64 byte: 610119 Pps > 512 byte: 516917 Pps > 1492 byte: 464962 Pps > > >> Am I correct that the maximum you can reach is around 639,000 packets >> per second? > > Yes, as you can see the maximum is 610119 Pps. > Where does this limit come from? ah that's the whole point of tuning :-) there are severalpossibities: 1/ the card's interrupts are probably attache dto aonly 1 cpu, so that cpu can do no more work 2/ if more than 1 cpu is working, it may be that there is a lock in heavy contention somewhere. is the machine still responsive to other networks while running at maximum capacity on this network? (make sure that the other networks are on a differnet CPU (hmm I can't remember how to do that :-). > >> Also you are not routing the traffic, but instead the server handles >> the requests itself and eat CPU to reply? > > Correct. In these first tests I want to "tune" the system, so I am using the (let me say) worst scenario. > > From owner-freebsd-performance@FreeBSD.ORG Mon Aug 3 16:22:43 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 65625106566B for ; Mon, 3 Aug 2009 16:22:43 +0000 (UTC) (envelope-from raykinsella78@gmail.com) Received: from mail-fx0-f210.google.com (mail-fx0-f210.google.com [209.85.220.210]) by mx1.freebsd.org (Postfix) with ESMTP id 99CC48FC18 for ; Mon, 3 Aug 2009 16:22:42 +0000 (UTC) (envelope-from raykinsella78@gmail.com) Received: by fxm6 with SMTP id 6so1423668fxm.43 for ; Mon, 03 Aug 2009 09:22:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=GyxIFxeRkVzWGE84//oN7CADNkRQTwPcl4cmGORI5sE=; b=bP5vZ3T6JwzbBJ+zE6HYt7n4mJjRZynXMdU15/H88YmQFoiBimp5hr7XAmY0QybBgL Gm2tgmb8K9XnkAnE0NslnvtZ5jS/6Sha8+Fl7FVzO4+PexWePzgibyzAYYi7B2faBC/X kt9wybplB5AooOwUQN5J4Gw5lMJsXLTq17OKs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=osz6XmDEEPoXsd8goEliMmV9Kr3vmNETDpWSG74VChHmeCcEv39n+clNXrJHxXHwQa VjfqsVQx16mNlT7iAtTL5IXBPtUZyaTtmc9SToXwUSgrns+SH3OMFzrgIFImg7VHlvpL AWe/XqNrX1iw4wsweOy5FiE2n9lcTokqs+LSk= MIME-Version: 1.0 Received: by 10.239.180.206 with SMTP id j14mr597228hbg.27.1249316089454; Mon, 03 Aug 2009 09:14:49 -0700 (PDT) In-Reply-To: <584ec6bb0908030819vee58480p43989b742e1b7fd2@mail.gmail.com> References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <584ec6bb0908030819vee58480p43989b742e1b7fd2@mail.gmail.com> Date: Mon, 3 Aug 2009 17:14:49 +0100 Message-ID: <584ec6bb0908030914m74b79dceq9af2581e1b02449a@mail.gmail.com> From: Ray Kinsella To: Invernizzi Fabrizio Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org Subject: Re: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2009 16:22:43 -0000 Hi Fabizio, Ignore my last mail direct to you, 638976 PPS is awful. (today is a national holiday here, my brain is not switched on). To me it looks like interrupt coalescing is not switched on for some reason= . Are you passing any parameters to the driver in boot.conf. Could you retest with vmstat switched on "vmstat 3" and send us the output. I expect we are going to see alot of interrupts. Regards Ray Kinsella On Mon, Aug 3, 2009 at 4:19 PM, Ray Kinsella wrote= : > Hi Fabrizio, > > I am an Intel Network Software Engineer, I test/improve the performance o= f > network device drivers among other things. I will do my best to help you. > > The first thing I would say is that I haven't used the 10GB NICs yet, but= a > rate of ~5 million PPS ((312*1024*1024)/64) is good or bad depending on w= hat > you are doing. i.e. How many NICs are sending on and how many are recievi= ng > on? In a situation where you operate cards in pairs, for instance all the > traffic from card A goes to card B and all the traffic from card B goes t= o > card A , I would consider this quiet low. In a situation where any card w= ill > talk to any card, for instance traffic from card A can go to card B, C or= D, > 5 million pps might be ok. > > The first thing you need to do is play with irq affinities, check out thi= s > blog post http://bramp.net/blog/post. You want to set the irq affinities > such that the rx threads are bound to one smp thread on one core. The nex= t > thing is if possible configure so that network cards that have a 1-1 > relationship to execute on seperate smp threads on the same core. This > should improve the line rate you are seeing. > > Regards > > Ray Kinsella > > > On Mon, Aug 3, 2009 at 9:46 AM, Invernizzi Fabrizio < > fabrizio.invernizzi@telecomitalia.it> wrote: > >> Hi all >> >> I am doing some tests on a BSD system with a 10gbe Intel based network >> card and I have some doubts about the configuration since the performanc= e I >> am experiencing looks (very) poor. >> >> This is the system I am doing test on: >> >> >> >> - HP 380 G5 (XEON X5420, CPU speed: 2.50GHz, BUS speed: 1333 MHz, L2 cac= he >> size: 12 MB, L2 cache speed: 2,5 GHz) with 1 quad-core installed. >> >> - Network card: Silicom PE10G2i-LR - Dual Port Fiber (LR) 10 Gigabit >> Ethernet PCI Express Server Adapter Intel=AE based (chip 82598EB). >> Driver ixgbe-1.8.6 >> >> - FreeBSD 7.2-RELEASE (64 bit) with this options compiled in the kernel >> options ZERO_COPY_SOCKETS # Turn on zero copy >> send code >> options HZ=3D1000 >> options BPF_JITTER >> >> >> >> I worked on the driver settings in order to have big TX/RX rings and low >> interrupt rate (traffic latency is not an issue). >> >> >> >> In order to tune up the system i started with some echo request tests. >> >> These are the maximum Bandwidths I can send without loss: >> >> - 64 byte packets: 312 Mbps (1,64% CPU idle) >> >> - 512 byte packets: 2117 Mbps (1,63% CPU idle) >> >> - 1492 byte packets: 5525 Mbps (1,93% CPU idle) >> >> >> >> Am I right considering these figures lower than expected? >> The system is just managing network traffic! >> >> >> >> Now I have started with netgraph tests, in particular with ng_bpf and th= e >> overall system is going even worst. >> >> I sent some HTTP traffic (597 bytes-long packets) and I configured an >> ng_bpf to filter TCP traffic out from the incoming interface (ix0). >> >> If I use the ngctl msg to see counters on the ng_bpf node, I see extreme= ly >> poor performance: >> >> >> >> - Sending 96Mbps of this traffic I figured out 0.1% packet loss. This >> looks very strange. May be some counter bug? >> >> - Sending 5500Mbps, the netgraph (not the network card driver) is loosin= g >> 21% of the number of sent packets. See below a snapshot of the CPU load >> under traffic load >> >> >> >> CPU: 0.0% user, 0.0% nice, 87.0% system, 9.1% interrupt, 3.9% idle >> >> Mem: 16M Active, 317M Inact, 366M Wired, 108K Cache, 399M Buf, 7222M Fre= e >> >> Swap: 2048M Total, 2048M Free >> >> >> >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMA= ND >> >> 12 root 1 171 ki31 0K 16K RUN 2 20.2H 68.80% idle: >> cpu2 >> >> 11 root 1 171 ki31 0K 16K RUN 3 20.1H 64.70% idle: >> cpu3 >> >> 14 root 1 171 ki31 0K 16K RUN 0 20.2H 64.26% idle: >> cpu0 >> >> 13 root 1 171 ki31 0K 16K RUN 1 20.2H 63.67% idle: >> cpu1 >> >> 38 root 1 -68 - 0K 16K CPU1 1 1:28 34.67% ix0 r= xq >> >> 40 root 1 -68 - 0K 16K CPU2 0 1:26 34.18% ix0 r= xq >> >> 34 root 1 -68 - 0K 16K CPU3 3 1:27 34.08% ix0 r= xq >> >> 36 root 1 -68 - 0K 16K RUN 2 1:26 34.08% ix0 r= xq >> >> 33 root 1 -68 - 0K 16K WAIT 3 0:40 4.05% irq26= 0: >> ix0 >> >> 39 root 1 -68 - 0K 16K WAIT 2 0:41 3.96% irq26= 3: >> ix0 >> >> 35 root 1 -68 - 0K 16K WAIT 0 0:39 3.66% irq26= 1: >> ix0 >> >> 37 root 1 -68 - 0K 16K WAIT 1 0:42 3.47% irq26= 2: >> ix0 >> >> 16 root 1 -32 - 0K 16K WAIT 0 14:53 2.49% swi4: >> clock sio >> >> >> >> >> >> >> >> Am I missing something? >> >> Does someone know some (more) system tuning to have higher traffic rate >> supported? >> >> >> >> Any help is greatly appreciated. >> >> >> >> Fabrizio >> >> >> >> >> >> ------------------------------------------------------------------ >> Telecom Italia >> Fabrizio INVERNIZZI >> Technology - TILAB >> Accesso Fisso e Trasporto >> Via Reiss Romoli, 274 10148 Torino >> Tel. +39 011 2285497 >> Mob. +39 3316001344 >> Fax +39 06 41867287 >> >> >> Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle >> persone indicate. La diffusione, copia o qualsiasi altra azione derivant= e >> dalla conoscenza di queste informazioni sono rigorosamente vietate. Qual= ora >> abbiate ricevuto questo documento per errore siete cortesemente pregati = di >> darne immediata comunicazione al mittente e di provvedere alla sua >> distruzione, Grazie. >> >> This e-mail and any attachments is confidential and may contain privileg= ed >> information intended for the addressee(s) only. Dissemination, copying, >> printing or use by anybody else is unauthorised. If you are not the inte= nded >> recipient, please delete this message and any attachments and advise the >> sender by return e-mail, Thanks. >> >> _______________________________________________ >> freebsd-performance@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >> To unsubscribe, send any mail to " >> freebsd-performance-unsubscribe@freebsd.org" >> > > From owner-freebsd-performance@FreeBSD.ORG Mon Aug 3 16:36:51 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C3B36106572F for ; Mon, 3 Aug 2009 16:36:51 +0000 (UTC) (envelope-from raykinsella78@gmail.com) Received: from mail-bw0-f206.google.com (mail-bw0-f206.google.com [209.85.218.206]) by mx1.freebsd.org (Postfix) with ESMTP id 118CD8FC1B for ; Mon, 3 Aug 2009 16:36:50 +0000 (UTC) (envelope-from raykinsella78@gmail.com) Received: by bwz2 with SMTP id 2so2412299bwz.43 for ; Mon, 03 Aug 2009 09:36:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=s3+cfOPR3IJVvXf9lJLws/ILduooUKdAL3far+Tt8nA=; b=paOxsXvpc7MCPS4tVG27EBTE2cUzOtFS+a8pjJ673uVPDUxrA6Gm3i22F7WRvFSncn dMrdLoNO1MAPhKEcUNGdjbxhVaGN+Ugkx82u5/abq1VZgVNXL5g9ous06wtnwwAN8WUO 0SLYtZKFD93JTZ9o/2FN+q6MP0Y32V0XxhHbU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=R+jkAdgow0L2qe3LxIUDEsFeJbWR9wevV8Iw/+h1xCVli2SSf6ooyfjJ05H3kX2TDU wYBaZV3Fr9P8AOSYs5eLi5RSUQ/kHRa+B/hkapWQi0WG5PJTsFdXsIaQgrOG+mwbI1xO 2+hg/Z+4CsVBy7eOp3ilUhBphECbi32IcOWw0= MIME-Version: 1.0 Received: by 10.239.162.205 with SMTP id m13mr655111hbd.87.1249315633619; Mon, 03 Aug 2009 09:07:13 -0700 (PDT) In-Reply-To: <2a41acea0908030820o40438f6erda78927733529a9@mail.gmail.com> References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> <18AAC16B-3CC0-4C70-A009-00A325AB5932@moneybookers.com> <36A93B31228D3B49B691AD31652BCAE9A4560DF96A@GRFMBX702BA020.griffon.local> <2a41acea0908030820o40438f6erda78927733529a9@mail.gmail.com> Date: Mon, 3 Aug 2009 17:07:13 +0100 Message-ID: <584ec6bb0908030907i4371d2d1y63fc23bb889ae06d@mail.gmail.com> From: Ray Kinsella To: Jack Vogel Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: "freebsd-performance@freebsd.org" , Invernizzi Fabrizio , Stefan Lambrev Subject: Re: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2009 16:36:52 -0000 Hi all, cpuset is the command to set a cpu affinity, there are details @ http://bramp.net/blog/post vmstat -z is the command you need to determine whether there is contention for mbufs. although the cpu usage does not suggest the system is memory constrained. Regards Ray Kinsella On Mon, Aug 3, 2009 at 4:20 PM, Jack Vogel wrote: > If you go to FreeBSD 8 you will get the improved stack code, and the RX/TX > queue pairs > will be pinned to cpus. It should improve performance. > > Make sure you have enough mbuf memory allocated, try increasing the > descriptors. > > Jack > > > On Mon, Aug 3, 2009 at 3:26 AM, Invernizzi Fabrizio < > fabrizio.invernizzi@telecomitalia.it> wrote: > > > > > If you are meaning in term of Packet per second, you are right. > > > > These are the packet per second measured during tests: > > > > > > > > 64 byte: 610119 Pps > > > > 512 byte: 516917 Pps > > > > 1492 byte: 464962 Pps > > > > > > > > > > > >> Am I correct that the maximum you can reach is around 639,000 > packets > > > >> per second? > > > > > > > > Yes, as you can see the maximum is 610119 Pps. > > > > Where does this limit come from? > > > > > > I duno - the tests I did before were with SYN packets (random source) > > > which was my worst scenario, > > > and the server CPU were busy generating MD5 check sums for > > > "syncache" (around 35% of the time). > > > > > > If I have to compare my results with your, you beat me with factor > > > 2.5, may be because you use ICMP for the test > > > and your processor is better then my test stations :) > > > Also my experience is only with gigabit cards (em driver) and FreeBSD > > > 7.something_before_1 where the em thread was eating 100% cpu. > > > If you are lucky LOCK_PROFILING(9) will help you to see where the CPUs > > > spend their time, if not you will see kernel panic :) > > > > > > I will check, thanks for the hint. > > > > > Once problematic locks identified they can be reworked, but I think > > > the first part is already done > > > and work on the second already started. > > > > > > In my experience increasing hw.em.rxd and hw.em.txd yelled better > > > results, but I think ixgb already comes tuned by default > > > as it still doesn't have to support such a large number of different > > > cards. > > > > I did some tuning in the code of the driver e recompiled the kernel in > > order to reduce context switching (interrupt mitigation) since the driver > > does not support POLLING. > > > > > Also at the time of my tests there were not support for multi queues > > > in the OS even if they were supported by the HW, which is changed in > > > 7.2 (?) > > > > It looks like multi queue is working since I can see the load distributed > > over the 4 cores. > > > > > > Fabrizio > > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle > > persone indicate. La diffusione, copia o qualsiasi altra azione derivante > > dalla conoscenza di queste informazioni sono rigorosamente vietate. > Qualora > > abbiate ricevuto questo documento per errore siete cortesemente pregati > di > > darne immediata comunicazione al mittente e di provvedere alla sua > > distruzione, Grazie. > > > > This e-mail and any attachments is confidential and may contain > privileged > > information intended for the addressee(s) only. Dissemination, copying, > > printing or use by anybody else is unauthorised. If you are not the > intended > > recipient, please delete this message and any attachments and advise the > > sender by return e-mail, Thanks. > > > > _______________________________________________ > > freebsd-performance@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > > To unsubscribe, send any mail to " > > freebsd-performance-unsubscribe@freebsd.org" > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@freebsd.org" > From owner-freebsd-performance@FreeBSD.ORG Tue Aug 4 07:55:18 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 97D3E10656A5 for ; Tue, 4 Aug 2009 07:55:18 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Received: from GRFEDG701BA020.telecomitalia.it (grfedg701ba020.telecomitalia.it [156.54.233.200]) by mx1.freebsd.org (Postfix) with ESMTP id 18DEA8FC16 for ; Tue, 4 Aug 2009 07:55:18 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Received: from GRFHUB701BA020.griffon.local (10.188.101.111) by GRFEDG701BA020.telecomitalia.it (10.188.45.100) with Microsoft SMTP Server (TLS) id 8.1.340.0; Tue, 4 Aug 2009 09:55:16 +0200 Received: from GRFMBX702BA020.griffon.local ([10.188.101.12]) by grfhub701ba020.griffon.local ([10.188.101.111]) with mapi; Tue, 4 Aug 2009 09:55:14 +0200 From: Invernizzi Fabrizio To: Julian Elischer Date: Tue, 4 Aug 2009 09:55:13 +0200 Thread-Topic: Test on 10GBE Intel based network card Thread-Index: AcoUU1WjCypibGa2RsmNsqF8ZFQatgAhLlrw Message-ID: <36A93B31228D3B49B691AD31652BCAE9A45696721F@GRFMBX702BA020.griffon.local> References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> <4A77094C.8030308@elischer.org> In-Reply-To: <4A77094C.8030308@elischer.org> Accept-Language: it-IT, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: it-IT, en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "freebsd-performance@freebsd.org" , Stefan Lambrev Subject: RE: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Aug 2009 07:55:19 -0000 > >> The limitation that you see is about the max number of packets that > >> FreeBSD can handle - it looks like your best performance is reached at > >> 64 byte packets? > > > > If you are meaning in term of Packet per second, you are right. These > are the packet per second measured during tests: > > > > 64 byte: 610119 Pps > > 512 byte: 516917 Pps > > 1492 byte: 464962 Pps > > > > > >> Am I correct that the maximum you can reach is around 639,000 packets > >> per second? > > > > Yes, as you can see the maximum is 610119 Pps. > > Where does this limit come from? > > ah that's the whole point of tuning :-) > there are severalpossibities: > 1/ the card's interrupts are probably attache dto aonly 1 cpu, > so that cpu can do no more work This seems not to be the problem. See below a top snapshot during a 64byte-= long packet storm last pid: 8552; load averages: 0.40, 0.09, 0.03 = up 0+20:3= 6:58 09:40:29 124 processes: 13 running, 73 sleeping, 38 waiting CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free Swap: 2048M Total, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: c= pu3 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: c= pu0 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: c= pu2 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: c= pu1 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 rxq 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 rxq 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 rxq 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 rxq .... It looks like the 4 rxq processes are bound to the 4 available cores with e= qual distribution. > 2/ if more than 1 cpu is working, it may be that there is a lock in > heavy contention somewhere. This I think is the problem. I am trying to understand how to 1- see where the heavy contention is (context switching? Some limiting sett= ing?) 2- mitigate it > > is the machine still responsive to other networks while > running at maximum capacity on this network? (make sure that > the other networks are on a differnet CPU (hmm I can't remember how to > do that :-). Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle per= sone indicate. La diffusione, copia o qualsiasi altra azione derivante dall= a conoscenza di queste informazioni sono rigorosamente vietate. Qualora abb= iate ricevuto questo documento per errore siete cortesemente pregati di dar= ne immediata comunicazione al mittente e di provvedere alla sua distruzione= , Grazie. This e-mail and any attachments is confidential and may contain privileged = information intended for the addressee(s) only. Dissemination, copying, pri= nting or use by anybody else is unauthorised. If you are not the intended r= ecipient, please delete this message and any attachments and advise the sen= der by return e-mail, Thanks. From owner-freebsd-performance@FreeBSD.ORG Tue Aug 4 08:14:48 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D5DC6106564A for ; Tue, 4 Aug 2009 08:14:48 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Received: from GRFEDG701BA020.telecomitalia.it (grfedg701ba020.telecomitalia.it [156.54.233.200]) by mx1.freebsd.org (Postfix) with ESMTP id 161FD8FC12 for ; Tue, 4 Aug 2009 08:14:48 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Received: from GRFHUB702BA020.griffon.local (10.188.101.112) by GRFEDG701BA020.telecomitalia.it (10.188.45.100) with Microsoft SMTP Server (TLS) id 8.1.340.0; Tue, 4 Aug 2009 10:14:47 +0200 Received: from GRFMBX702BA020.griffon.local ([10.188.101.12]) by GRFHUB702BA020.griffon.local ([10.188.101.112]) with mapi; Tue, 4 Aug 2009 10:14:46 +0200 From: Invernizzi Fabrizio To: Ray Kinsella Date: Tue, 4 Aug 2009 10:14:45 +0200 Thread-Topic: Test on 10GBE Intel based network card Thread-Index: AcoUVYyMkjEvLBrwTZibM9dtleR3IQAg4lfw Message-ID: <36A93B31228D3B49B691AD31652BCAE9A456967237@GRFMBX702BA020.griffon.local> References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <584ec6bb0908030819vee58480p43989b742e1b7fd2@mail.gmail.com> <584ec6bb0908030914m74b79dceq9af2581e1b02449a@mail.gmail.com> In-Reply-To: <584ec6bb0908030914m74b79dceq9af2581e1b02449a@mail.gmail.com> Accept-Language: it-IT Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: acceptlanguage: it-IT Content-Type: multipart/mixed; boundary="_002_36A93B31228D3B49B691AD31652BCAE9A456967237GRFMBX702BA02_" MIME-Version: 1.0 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: "freebsd-performance@freebsd.org" Subject: RE: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Aug 2009 08:14:49 -0000 --_002_36A93B31228D3B49B691AD31652BCAE9A456967237GRFMBX702BA02_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Ray, >To me it looks like interrupt coalescing is not switched on for some reaso= n. >Are you passing any parameters to the driver in boot.conf. This is my loader.conf kern.ipc.nmbclusters=3D65635 kern.hz=3D1000 net.bpf_jitter.enable=3D1 # net.graph.threads=3D32 # if_em_load=3D"YES" # NETGRAPH TUNING net.graph.maxdata=3D1024 kern.ipc.somaxconn=3D4096 net.inet.tcp.recvspace=3D78840 net.inet.tcp.sendspace=3D78840 kern.ipc.shmmax=3D67108864 kern.ipc.shmmni=3D200 kern.ipc.shmseg=3D128 kern.ipc.semmni=3D70 net.local.stream.sendspace=3D82320 net.local.stream.recvspace=3D82320 net.inet.tcp.local_slowstart_flightsize=3D10 net.inet.tcp.nolocaltimewait=3D1 net.inet.tcp.hostcache.expire=3D3900 kern.maxusers=3D512 kern.ipc.nmbclusters=3D32768 kern.ipc.maxsockets=3D81920 kern.ipc.maxsockbuf=3D1048576 net.inet.tcp.tcbhashsize=3D4096 net.inet.tcp.hostcache.hashsize=3D1024 >Could you retest with vmstat switched on "vmstat 3" and send us the output= . >I expect we are going to see alot of interrupts. Sending 535714 pps (64bytes-long) INTRUDER-64# vmstat 3 procs memory page disks faults cp= u r b w avm fre flt re pi po fr sr da0 da1 in sy cs us= sy id 0 0 0 95420K 7203M 19 0 0 0 17 0 0 0 642 66 1078 0= 2 98 0 0 0 95420K 7203M 0 0 0 0 0 0 2 0 18 65 527 0= 0 100 0 0 0 95420K 7203M 0 0 0 0 0 0 0 0 18 67 527 0= 0 100 0 0 0 95420K 7203M 0 0 0 0 0 0 0 0 17 64 525 0= 0 100 0 0 0 95420K 7203M 0 0 0 0 0 0 0 0 31526 64 31402 = 0 87 13 0 0 0 95420K 7203M 0 0 0 0 0 0 0 0 36767 64 33320 = 0 99 1 0 0 0 95420K 7203M 423 0 0 0 406 0 0 0 36174 384 28107 = 0 99 1 0 0 0 95420K 7203M 0 0 0 0 0 0 0 0 36706 64 27043 = 0 99 1 0 0 0 95420K 7203M 0 0 0 0 0 0 0 0 34006 64 13117 = 0 91 9 2 0 0 95420K 7203M 0 0 0 0 0 0 0 0 17 64 550 0= 1 99 0 0 0 95420K 7203M 0 0 0 0 3 0 3 0 19 68 507 0= 0 100 dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 1.= 7.4 dev.ix.0.%driver: ix dev.ix.0.%location: slot=3D0 function=3D0 dev.ix.0.%pnpinfo: vendor=3D0x8086 device=3D0x10c6 subvendor=3D0x8086 subde= vice=3D0xa15f class=3D0x020000 dev.ix.0.%parent: pci3 dev.ix.0.stats: -1 dev.ix.0.debug: -1 dev.ix.0.flow_control: 0 dev.ix.0.enable_lro: 1 Adaptive Interrupt Mitigation is enabled: dev.ix.0.enable_aim: 1 I did not changed AIM settings since is quite complex to tune them up. I t= ried to reverse engineering the algorithm of AIM (see attached picture) but= I can't obtain tangible improvements playing with these paramenters. My un= derstanding is that I should reduce the low_latency, but it seems not to wo= rk. dev.ix.0.low_latency: 128 dev.ix.0.ave_latency: 400 dev.ix.0.bulk_latency: 1200 Not sure about this dev.ix.0.hdr_split: 0 Not sure about the meaning of dev.ix.0.rx_processing_limit: 100 These are the settings I am using in the ixgbe driver: #define DEFAULT_TXD 1024 #define PERFORM_TXD 2048 #define MAX_TXD 4096 #define MIN_TXD 64 #define DEFAULT_RXD 1024 #define PERFORM_RXD 2048 #define MAX_RXD 4096 #define MIN_RXD 64 #define IXGBE_TX_CLEANUP_THRESHOLD (adapter->num_tx_desc / 1) #define IXGBE_TX_OP_THRESHOLD (adapter->num_tx_desc / 4) I see that I had a good performance improvement setting IXGBE_TX_CLEANUP_TH= RESHOLD to the tx queue size. This is clear to understand since this (greatly) minimizes context switchin= g in send process reducing the number of time the txq function is called. (= Of course with this settings send latency is increased, but this is not an = issue). I can't understand the meaning of these parameters and if they could help: /* * This parameter controls the maximum no of times the driver will loop in * the isr. Minimum Value =3D 1 */ #define MAX_LOOP 10 /* * This parameter controls the duration of transmit watchdog timer. */ #define IXGBE_TX_TIMEOUT 5 /* set to 5 seconds */ Fabrizio Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle per= sone indicate. La diffusione, copia o qualsiasi altra azione derivante dall= a conoscenza di queste informazioni sono rigorosamente vietate. Qualora abb= iate ricevuto questo documento per errore siete cortesemente pregati di dar= ne immediata comunicazione al mittente e di provvedere alla sua distruzione= , Grazie. This e-mail and any attachments is confidential and may contain privileged = information intended for the addressee(s) only. Dissemination, copying, pri= nting or use by anybody else is unauthorised. If you are not the intended r= ecipient, please delete this message and any attachments and advise the sen= der by return e-mail, Thanks. --_002_36A93B31228D3B49B691AD31652BCAE9A456967237GRFMBX702BA02_-- From owner-freebsd-performance@FreeBSD.ORG Tue Aug 4 10:26:29 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0F94E106566C for ; Tue, 4 Aug 2009 10:26:29 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Received: from GRFEDG701BA020.telecomitalia.it (grfedg701ba020.telecomitalia.it [156.54.233.200]) by mx1.freebsd.org (Postfix) with ESMTP id 8796A8FC13 for ; Tue, 4 Aug 2009 10:26:28 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Received: from GRFHUB703BA020.griffon.local (10.188.101.113) by GRFEDG701BA020.telecomitalia.it (10.188.45.100) with Microsoft SMTP Server (TLS) id 8.1.340.0; Tue, 4 Aug 2009 12:26:27 +0200 Received: from GRFMBX702BA020.griffon.local ([10.188.101.12]) by GRFHUB703BA020.griffon.local ([10.188.101.113]) with mapi; Tue, 4 Aug 2009 12:26:25 +0200 From: Invernizzi Fabrizio To: "freebsd-performance@freebsd.org" Date: Tue, 4 Aug 2009 12:26:23 +0200 Thread-Topic: Test on 10GBE Intel based network card Thread-Index: AcoUU1WjCypibGa2RsmNsqF8ZFQatgAhLlrwAAV1u/A= Message-ID: <36A93B31228D3B49B691AD31652BCAE9A4569672AD@GRFMBX702BA020.griffon.local> References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> <4A77094C.8030308@elischer.org> <36A93B31228D3B49B691AD31652BCAE9A45696721F@GRFMBX702BA020.griffon.local> In-Reply-To: <36A93B31228D3B49B691AD31652BCAE9A45696721F@GRFMBX702BA020.griffon.local> Accept-Language: it-IT Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: it-IT Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: RE: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Aug 2009 10:26:29 -0000 Hi all. Going on with my tests. I noticed that I have always a (received) pps/interrupt ratio(per second) t= hat is always about 14 (a very low ratio in my opinion). If I disable aim via sysctl sysctl -w dev.ix.0.enable_aim=3D0 dev.ix.0.enable_aim: 1 -> 0 this ration remain the same. Something going wrong with interrupt coalescin= g, but I can't find out what. Any idea? Fabrizio > -----Original Message----- > From: owner-freebsd-performance@freebsd.org [mailto:owner-freebsd- > performance@freebsd.org] On Behalf Of Invernizzi Fabrizio > Sent: marted=EC 4 agosto 2009 9.55 > To: Julian Elischer > Cc: freebsd-performance@freebsd.org; Stefan Lambrev > Subject: RE: Test on 10GBE Intel based network card > > > >> The limitation that you see is about the max number of packets that > > >> FreeBSD can handle - it looks like your best performance is reached > at > > >> 64 byte packets? > > > > > > If you are meaning in term of Packet per second, you are right. These > > are the packet per second measured during tests: > > > > > > 64 byte: 610119 Pps > > > 512 byte: 516917 Pps > > > 1492 byte: 464962 Pps > > > > > > > > >> Am I correct that the maximum you can reach is around 639,000 packet= s > > >> per second? > > > > > > Yes, as you can see the maximum is 610119 Pps. > > > Where does this limit come from? > > > > ah that's the whole point of tuning :-) > > there are severalpossibities: > > 1/ the card's interrupts are probably attache dto aonly 1 cpu, > > so that cpu can do no more work > > This seems not to be the problem. See below a top snapshot during a > 64byte-long packet storm > > last pid: 8552; load averages: 0.40, 0.09, 0.03 > up 0+20:36:58 09:40:29 > 124 processes: 13 running, 73 sleeping, 38 waiting > CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle > Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free > Swap: 2048M Total, 2048M Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: > cpu3 > 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: > cpu0 > 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: > cpu2 > 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: > cpu1 > 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 > rxq > 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 > rxq > 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 > rxq > 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 > rxq > .... > > It looks like the 4 rxq processes are bound to the 4 available cores with > equal distribution. > > > > > 2/ if more than 1 cpu is working, it may be that there is a lock in > > heavy contention somewhere. > > This I think is the problem. I am trying to understand how to > 1- see where the heavy contention is (context switching? Some limiting > setting?) > 2- mitigate it > > > > > is the machine still responsive to other networks while > > running at maximum capacity on this network? (make sure that > > the other networks are on a differnet CPU (hmm I can't remember how to > > do that :-). > > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle > persone indicate. La diffusione, copia o qualsiasi altra azione derivante > dalla conoscenza di queste informazioni sono rigorosamente vietate. > Qualora abbiate ricevuto questo documento per errore siete cortesemente > pregati di darne immediata comunicazione al mittente e di provvedere alla > sua distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain privilege= d > information intended for the addressee(s) only. Dissemination, copying, > printing or use by anybody else is unauthorised. If you are not the > intended recipient, please delete this message and any attachments and > advise the sender by return e-mail, Thanks. > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance- > unsubscribe@freebsd.org" Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle per= sone indicate. La diffusione, copia o qualsiasi altra azione derivante dall= a conoscenza di queste informazioni sono rigorosamente vietate. Qualora abb= iate ricevuto questo documento per errore siete cortesemente pregati di dar= ne immediata comunicazione al mittente e di provvedere alla sua distruzione= , Grazie. This e-mail and any attachments is confidential and may contain privileged = information intended for the addressee(s) only. Dissemination, copying, pri= nting or use by anybody else is unauthorised. If you are not the intended r= ecipient, please delete this message and any attachments and advise the sen= der by return e-mail, Thanks. From owner-freebsd-performance@FreeBSD.ORG Tue Aug 4 16:17:18 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 01BD0106566B for ; Tue, 4 Aug 2009 16:17:18 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outW.internet-mail-service.net (outw.internet-mail-service.net [216.240.47.246]) by mx1.freebsd.org (Postfix) with ESMTP id D509E8FC0A for ; Tue, 4 Aug 2009 16:17:17 +0000 (UTC) (envelope-from julian@elischer.org) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id 95C69961CA; Tue, 4 Aug 2009 09:17:17 -0700 (PDT) X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id E06492D6006; Tue, 4 Aug 2009 09:17:16 -0700 (PDT) Message-ID: <4A785F20.8050807@elischer.org> Date: Tue, 04 Aug 2009 09:17:36 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.22 (Macintosh/20090605) MIME-Version: 1.0 To: Invernizzi Fabrizio References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> <4A77094C.8030308@elischer.org> <36A93B31228D3B49B691AD31652BCAE9A45696721F@GRFMBX702BA020.griffon.local> In-Reply-To: <36A93B31228D3B49B691AD31652BCAE9A45696721F@GRFMBX702BA020.griffon.local> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-performance@freebsd.org" , Stefan Lambrev Subject: Re: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Aug 2009 16:17:18 -0000 Invernizzi Fabrizio wrote: >>>> The limitation that you see is about the max number of packets that >>>> FreeBSD can handle - it looks like your best performance is reached at >>>> 64 byte packets? >>> If you are meaning in term of Packet per second, you are right. These >> are the packet per second measured during tests: >>> 64 byte: 610119 Pps >>> 512 byte: 516917 Pps >>> 1492 byte: 464962 Pps >>> >>> >>>> Am I correct that the maximum you can reach is around 639,000 packets >>>> per second? >>> Yes, as you can see the maximum is 610119 Pps. >>> Where does this limit come from? >> ah that's the whole point of tuning :-) >> there are severalpossibities: >> 1/ the card's interrupts are probably attache dto aonly 1 cpu, >> so that cpu can do no more work > > This seems not to be the problem. See below a top snapshot during a 64byte-long packet storm > > last pid: 8552; load averages: 0.40, 0.09, 0.03 up 0+20:36:58 09:40:29 > 124 processes: 13 running, 73 sleeping, 38 waiting > CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle > Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free > Swap: 2048M Total, 2048M Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: cpu3 > 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: cpu0 > 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: cpu2 > 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: cpu1 > 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 rxq > 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 rxq > 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 rxq > 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 rxq > .... > > It looks like the 4 rxq processes are bound to the 4 available cores with equal distribution. > > > >> 2/ if more than 1 cpu is working, it may be that there is a lock in >> heavy contention somewhere. > > This I think is the problem. I am trying to understand how to > 1- see where the heavy contention is (context switching? Some limiting setting?) > 2- mitigate it > there ia a lock profiling tool that right now I can't remember the name of.. look it up with google :-) FreeBSD lock profiling tool ah, first hit... http://blogs.epfl.ch/article/23832 >> is the machine still responsive to other networks while >> running at maximum capacity on this network? (make sure that >> the other networks are on a differnet CPU (hmm I can't remember how to >> do that :-). > > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks. From owner-freebsd-performance@FreeBSD.ORG Tue Aug 4 16:41:33 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2CC9E10658DA for ; Tue, 4 Aug 2009 16:41:33 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.244]) by mx1.freebsd.org (Postfix) with ESMTP id D0E348FC13 for ; Tue, 4 Aug 2009 16:41:32 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: by an-out-0708.google.com with SMTP id d14so1958612and.13 for ; Tue, 04 Aug 2009 09:41:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=u8Ld3h1tbGh0X8M/4hjhDSaYSTHlIZ/ty/Igj1MI3qo=; b=CR5T9SNDLLHOP3B1QBwlvg1fnHjxNWIsjK2TrdFVm6HaTX4z0oNu4yhuay3wgd4L8x 0lgqVScasCbxxxZGdawRll+xtbhoiYdjxFGx1NxxYfS6mqA3vcH3ryvcqFDzTEMAp78b k5YpON7wgoAHUoR42vrNMpUnXHO1/KrPOQIrc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=j4f+CgWV7g/a+QE90mqBEuW2JGHVVqBj8A+J7zD+8tpi/Z/c149OUzsdgHlmObZn1v Rn2XaIhwlKO0ZjBIMpKrTcJWqWaVdTdmTlhai77TBHk45w+G5uVnvhpmfZXna1WmirhY 0A5bfunwKzONdzfQ2TIIjZKzmbyaWmuHse7r4= MIME-Version: 1.0 Received: by 10.100.163.11 with SMTP id l11mr5609056ane.161.1249404091966; Tue, 04 Aug 2009 09:41:31 -0700 (PDT) In-Reply-To: <4A785F20.8050807@elischer.org> References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> <4A77094C.8030308@elischer.org> <36A93B31228D3B49B691AD31652BCAE9A45696721F@GRFMBX702BA020.griffon.local> <4A785F20.8050807@elischer.org> Date: Tue, 4 Aug 2009 09:41:31 -0700 Message-ID: <2a41acea0908040941y39f16c8cocb84b001e1e9f0de@mail.gmail.com> From: Jack Vogel To: Julian Elischer Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: "freebsd-performance@freebsd.org" , Invernizzi Fabrizio , Stefan Lambrev Subject: Re: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Aug 2009 16:41:34 -0000 Your nmbclusters is very low, you list it twice so I'm assuming the second value is what it ends up being, 32K :( I would set it to: kern.ipc.nmbclusters=262144 Also, I thought you were using the current driver, but now it looks like you are using something fairly old, use my latest code which is 1.8.8 Jack On Tue, Aug 4, 2009 at 9:17 AM, Julian Elischer wrote: > Invernizzi Fabrizio wrote: > >> The limitation that you see is about the max number of packets that >>>>> FreeBSD can handle - it looks like your best performance is reached at >>>>> 64 byte packets? >>>>> >>>> If you are meaning in term of Packet per second, you are right. These >>>> >>> are the packet per second measured during tests: >>> >>>> 64 byte: 610119 Pps >>>> 512 byte: 516917 Pps >>>> 1492 byte: 464962 Pps >>>> >>>> >>>> Am I correct that the maximum you can reach is around 639,000 packets >>>>> per second? >>>>> >>>> Yes, as you can see the maximum is 610119 Pps. >>>> Where does this limit come from? >>>> >>> ah that's the whole point of tuning :-) >>> there are severalpossibities: >>> 1/ the card's interrupts are probably attache dto aonly 1 cpu, >>> so that cpu can do no more work >>> >> >> This seems not to be the problem. See below a top snapshot during a >> 64byte-long packet storm >> >> last pid: 8552; load averages: 0.40, 0.09, 0.03 >> up >> 0+20:36:58 09:40:29 >> 124 processes: 13 running, 73 sleeping, 38 waiting >> CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle >> Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free >> Swap: 2048M Total, 2048M Free >> >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND >> 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: >> cpu3 >> 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: >> cpu0 >> 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: >> cpu2 >> 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: >> cpu1 >> 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 rxq >> 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 rxq >> 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 rxq >> 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 rxq >> .... >> >> It looks like the 4 rxq processes are bound to the 4 available cores with >> equal distribution. >> >> >> >> 2/ if more than 1 cpu is working, it may be that there is a lock in >>> heavy contention somewhere. >>> >> >> This I think is the problem. I am trying to understand how to >> 1- see where the heavy contention is (context switching? Some limiting >> setting?) >> 2- mitigate it >> >> > > there ia a lock profiling tool that right now I can't remember the name > of.. > > look it up with google :-) FreeBSD lock profiling tool > > ah, first hit... > > http://blogs.epfl.ch/article/23832 > > > > is the machine still responsive to other networks while >>> running at maximum capacity on this network? (make sure that >>> the other networks are on a differnet CPU (hmm I can't remember how to >>> do that :-). >>> >> >> >> >> Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle >> persone indicate. La diffusione, copia o qualsiasi altra azione derivante >> dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora >> abbiate ricevuto questo documento per errore siete cortesemente pregati di >> darne immediata comunicazione al mittente e di provvedere alla sua >> distruzione, Grazie. >> >> This e-mail and any attachments is confidential and may contain privileged >> information intended for the addressee(s) only. Dissemination, copying, >> printing or use by anybody else is unauthorised. If you are not the intended >> recipient, please delete this message and any attachments and advise the >> sender by return e-mail, Thanks. >> > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@freebsd.org" > From owner-freebsd-performance@FreeBSD.ORG Wed Aug 5 09:04:23 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 52B741065673 for ; Wed, 5 Aug 2009 09:04:23 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Received: from GRFEDG701BA020.telecomitalia.it (grfedg701ba020.telecomitalia.it [156.54.233.200]) by mx1.freebsd.org (Postfix) with ESMTP id 642B08FC22 for ; Wed, 5 Aug 2009 09:04:22 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Content-Type: multipart/mixed; boundary="_7ca0b56d-3376-42e6-85e4-182344c683f2_" Received: from GRFHUB702BA020.griffon.local (10.188.101.112) by GRFEDG701BA020.telecomitalia.it (10.188.45.100) with Microsoft SMTP Server (TLS) id 8.1.340.0; Wed, 5 Aug 2009 11:04:19 +0200 Received: from GRFMBX702BA020.griffon.local ([10.188.101.12]) by GRFHUB702BA020.griffon.local ([10.188.101.112]) with mapi; Wed, 5 Aug 2009 11:04:12 +0200 From: Invernizzi Fabrizio To: Jack Vogel , Julian Elischer Date: Wed, 5 Aug 2009 11:04:10 +0200 Thread-Topic: Test on 10GBE Intel based network card Thread-Index: AcoVIm0LnEUSix6PQmaZQulzWV0J9QAiFJuA Message-ID: <36A93B31228D3B49B691AD31652BCAE9A456967403@GRFMBX702BA020.griffon.local> References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> <4A77094C.8030308@elischer.org> <36A93B31228D3B49B691AD31652BCAE9A45696721F@GRFMBX702BA020.griffon.local> <4A785F20.8050807@elischer.org> <2a41acea0908040941y39f16c8cocb84b001e1e9f0de@mail.gmail.com> In-Reply-To: <2a41acea0908040941y39f16c8cocb84b001e1e9f0de@mail.gmail.com> Accept-Language: it-IT Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: it-IT MIME-Version: 1.0 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: "freebsd-performance@freebsd.org" , Stefan Lambrev Subject: RE: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Aug 2009 09:04:24 -0000 --_7ca0b56d-3376-42e6-85e4-182344c683f2_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi Jack, Now I have upgraded the driver to 1.8.6 (official intel last release), but = no better performance. I am about to test with the settings your are sugges= ting. Using the same configuation i used before, i see a reduction on interrupts = (from 1 for 14 packets in 1.7.6 to 1 for 40 packets in 1.8.6). This leades = to a 5% to 10% of free CPU, but no higer packet rate. I tried working with the AIM settings, but no luck. Where can i download your 1.8.8 driver version? Fabrizio ________________________________ From: Jack Vogel [mailto:jfvogel@gmail.com] Sent: marted=EC 4 agosto 2009 18.42 To: Julian Elischer Cc: Invernizzi Fabrizio; freebsd-performance@freebsd.org; Stefan Lambrev Subject: Re: Test on 10GBE Intel based network card Your nmbclusters is very low, you list it twice so I'm assuming the second = value is what it ends up being, 32K :( I would set it to: kern.ipc.nmbclusters=3D262144 Also, I thought you were using the current driver, but now it looks like yo= u are using something fairly old, use my latest code which is 1.8.8 Jack On Tue, Aug 4, 2009 at 9:17 AM, Julian Elischer > wrote: Invernizzi Fabrizio wrote: The limitation that you see is about the max number of packets that FreeBSD can handle - it looks like your best performance is reached at 64 byte packets? If you are meaning in term of Packet per second, you are right. These are the packet per second measured during tests: 64 byte: 610119 Pps 512 byte: 516917 Pps 1492 byte: 464962 Pps Am I correct that the maximum you can reach is around 639,000 packets per second? Yes, as you can see the maximum is 610119 Pps. Where does this limit come from? ah that's the whole point of tuning :-) there are severalpossibities: 1/ the card's interrupts are probably attache dto aonly 1 cpu, so that cpu can do no more work This seems not to be the problem. See below a top snapshot during a 64byte-= long packet storm last pid: 8552; load averages: 0.40, 0.09, 0.03 = up 0+20:3= 6:58 09:40:29 124 processes: 13 running, 73 sleeping, 38 waiting CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free Swap: 2048M Total, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: cp= u3 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: cp= u0 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: cp= u2 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: cp= u1 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 rxq 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 rxq 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 rxq 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 rxq .... It looks like the 4 rxq processes are bound to the 4 available cores with e= qual distribution. 2/ if more than 1 cpu is working, it may be that there is a lock in heavy contention somewhere. This I think is the problem. I am trying to understand how to 1- see where the heavy contention is (context switching? Some limiting sett= ing?) 2- mitigate it there ia a lock profiling tool that right now I can't remember the name of.= . look it up with google :-) FreeBSD lock profiling tool ah, first hit... http://blogs.epfl.ch/article/23832 is the machine still responsive to other networks while running at maximum capacity on this network? (make sure that the other networks are on a differnet CPU (hmm I can't remember how to do that :-). Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle per= sone indicate. La diffusione, copia o qualsiasi altra azione derivante dall= a conoscenza di queste informazioni sono rigorosamente vietate. Qualora abb= iate ricevuto questo documento per errore siete cortesemente pregati di dar= ne immediata comunicazione al mittente e di provvedere alla sua distruzione= , Grazie. This e-mail and any attachments is confidential and may contain privileged = information intended for the addressee(s) only. Dissemination, copying, pri= nting or use by anybody else is unauthorised. If you are not the intended r= ecipient, please delete this message and any attachments and advise the sen= der by return e-mail, Thanks. _______________________________________________ freebsd-performance@freebsd.org mai= ling list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.o= rg" Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle per= sone indicate. La diffusione, copia o qualsiasi altra azione derivante dall= a conoscenza di queste informazioni sono rigorosamente vietate. Qualora abb= iate ricevuto questo documento per errore siete cortesemente pregati di dar= ne immediata comunicazione al mittente e di provvedere alla sua distruzione= , Grazie. This e-mail and any attachments is confidential and may contain privileged = information intended for the addressee(s) only. Dissemination, copying, pri= nting or use by anybody else is unauthorised. If you are not the intended r= ecipient, please delete this message and any attachments and advise the sen= der by return e-mail, Thanks. [cid:00000000000000000000000000000001@TI.Disclaimer]Rispetta l'ambiente. No= n stampare questa mail se non =E8 necessario. --_7ca0b56d-3376-42e6-85e4-182344c683f2_-- From owner-freebsd-performance@FreeBSD.ORG Wed Aug 5 10:13:26 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6D23F1065679 for ; Wed, 5 Aug 2009 10:13:26 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Received: from GRFEDG701BA020.telecomitalia.it (grfedg701ba020.telecomitalia.it [156.54.233.200]) by mx1.freebsd.org (Postfix) with ESMTP id 9B8BC8FC0A for ; Wed, 5 Aug 2009 10:13:25 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Content-Type: multipart/mixed; boundary="_f70d3edc-3e50-4297-b616-09fd857ffdd3_" Received: from GRFHUB701BA020.griffon.local (10.188.101.111) by GRFEDG701BA020.telecomitalia.it (10.188.45.100) with Microsoft SMTP Server (TLS) id 8.1.340.0; Wed, 5 Aug 2009 12:13:24 +0200 Received: from GRFMBX702BA020.griffon.local ([10.188.101.12]) by grfhub701ba020.griffon.local ([10.188.101.111]) with mapi; Wed, 5 Aug 2009 12:13:24 +0200 From: Invernizzi Fabrizio To: Jack Vogel , Julian Elischer Date: Wed, 5 Aug 2009 12:13:23 +0200 Thread-Topic: Test on 10GBE Intel based network card Thread-Index: AcoVIm0LnEUSix6PQmaZQulzWV0J9QAksHgA Message-ID: <36A93B31228D3B49B691AD31652BCAE9A45696743F@GRFMBX702BA020.griffon.local> References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> <4A77094C.8030308@elischer.org> <36A93B31228D3B49B691AD31652BCAE9A45696721F@GRFMBX702BA020.griffon.local> <4A785F20.8050807@elischer.org> <2a41acea0908040941y39f16c8cocb84b001e1e9f0de@mail.gmail.com> In-Reply-To: <2a41acea0908040941y39f16c8cocb84b001e1e9f0de@mail.gmail.com> Accept-Language: it-IT, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: it-IT, en-US MIME-Version: 1.0 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: "freebsd-performance@freebsd.org" , Stefan Lambrev Subject: RE: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Aug 2009 10:13:26 -0000 --_f70d3edc-3e50-4297-b616-09fd857ffdd3_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable No improvement with kern.ipc.nmbclusters=3D262144 and 1.8.6 driver :<((((( ++fabrizio ------------------------------------------------------------------ Telecom Italia Fabrizio INVERNIZZI Technology - TILAB Accesso Fisso e Trasporto Via Reiss Romoli, 274 10148 Torino Tel. +39 011 2285497 Mob. +39 3316001344 Fax +39 06 41867287 ________________________________ From: Jack Vogel [mailto:jfvogel@gmail.com] Sent: marted=EC 4 agosto 2009 18.42 To: Julian Elischer Cc: Invernizzi Fabrizio; freebsd-performance@freebsd.org; Stefan Lambrev Subject: Re: Test on 10GBE Intel based network card Your nmbclusters is very low, you list it twice so I'm assuming the second = value is what it ends up being, 32K :( I would set it to: kern.ipc.nmbclusters=3D262144 Also, I thought you were using the current driver, but now it looks like yo= u are using something fairly old, use my latest code which is 1.8.8 Jack On Tue, Aug 4, 2009 at 9:17 AM, Julian Elischer > wrote: Invernizzi Fabrizio wrote: The limitation that you see is about the max number of packets that FreeBSD can handle - it looks like your best performance is reached at 64 byte packets? If you are meaning in term of Packet per second, you are right. These are the packet per second measured during tests: 64 byte: 610119 Pps 512 byte: 516917 Pps 1492 byte: 464962 Pps Am I correct that the maximum you can reach is around 639,000 packets per second? Yes, as you can see the maximum is 610119 Pps. Where does this limit come from? ah that's the whole point of tuning :-) there are severalpossibities: 1/ the card's interrupts are probably attache dto aonly 1 cpu, so that cpu can do no more work This seems not to be the problem. See below a top snapshot during a 64byte-= long packet storm last pid: 8552; load averages: 0.40, 0.09, 0.03 = up 0+20:3= 6:58 09:40:29 124 processes: 13 running, 73 sleeping, 38 waiting CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free Swap: 2048M Total, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: cp= u3 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: cp= u0 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: cp= u2 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: cp= u1 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 rxq 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 rxq 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 rxq 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 rxq .... It looks like the 4 rxq processes are bound to the 4 available cores with e= qual distribution. 2/ if more than 1 cpu is working, it may be that there is a lock in heavy contention somewhere. This I think is the problem. I am trying to understand how to 1- see where the heavy contention is (context switching? Some limiting sett= ing?) 2- mitigate it there ia a lock profiling tool that right now I can't remember the name of.= . look it up with google :-) FreeBSD lock profiling tool ah, first hit... http://blogs.epfl.ch/article/23832 is the machine still responsive to other networks while running at maximum capacity on this network? (make sure that the other networks are on a differnet CPU (hmm I can't remember how to do that :-). Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle per= sone indicate. La diffusione, copia o qualsiasi altra azione derivante dall= a conoscenza di queste informazioni sono rigorosamente vietate. Qualora abb= iate ricevuto questo documento per errore siete cortesemente pregati di dar= ne immediata comunicazione al mittente e di provvedere alla sua distruzione= , Grazie. This e-mail and any attachments is confidential and may contain privileged = information intended for the addressee(s) only. Dissemination, copying, pri= nting or use by anybody else is unauthorised. If you are not the intended r= ecipient, please delete this message and any attachments and advise the sen= der by return e-mail, Thanks. _______________________________________________ freebsd-performance@freebsd.org mai= ling list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.o= rg" Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle per= sone indicate. La diffusione, copia o qualsiasi altra azione derivante dall= a conoscenza di queste informazioni sono rigorosamente vietate. Qualora abb= iate ricevuto questo documento per errore siete cortesemente pregati di dar= ne immediata comunicazione al mittente e di provvedere alla sua distruzione= , Grazie. This e-mail and any attachments is confidential and may contain privileged = information intended for the addressee(s) only. Dissemination, copying, pri= nting or use by anybody else is unauthorised. If you are not the intended r= ecipient, please delete this message and any attachments and advise the sen= der by return e-mail, Thanks. [cid:00000000000000000000000000000001@TI.Disclaimer]Rispetta l'ambiente. No= n stampare questa mail se non =E8 necessario. --_f70d3edc-3e50-4297-b616-09fd857ffdd3_-- From owner-freebsd-performance@FreeBSD.ORG Fri Aug 7 09:40:16 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B693610656D2 for ; Fri, 7 Aug 2009 09:40:16 +0000 (UTC) (envelope-from grarpamp@gmail.com) Received: from mail-ew0-f206.google.com (mail-ew0-f206.google.com [209.85.219.206]) by mx1.freebsd.org (Postfix) with ESMTP id 246CD8FC1E for ; Fri, 7 Aug 2009 09:40:15 +0000 (UTC) Received: by ewy2 with SMTP id 2so1449824ewy.43 for ; Fri, 07 Aug 2009 02:40:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type:content-transfer-encoding; bh=JQ5e9ibwZL36BdA48VN6vnEymTI28oBJFibxnW+bxmA=; b=WM7yhnF51v40J3gG14EdqnJOZGv87UnDmyKBpWVYdb09uwQomEUrxLnC/9WCjyv9h+ ntXDSN1Pp1bCgCYwwn4d4Zr7Kj57rJkEBgNjhbQyeSXPTjVXfOP8E+lCK3FBkTD7k1c2 VNXRbLhwrN++qncRr7x1snbdV21h4lcgXoFlI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type :content-transfer-encoding; b=W1AaC4rccaKMh9BXOxnjijCGyE6os6WzZEBdkkOaQx2O/UaATENKSFiVALhRWN69jo lB2OtuMWCWbbLCxuTU36D2LXLK941ET24Q8AnNs9hzQCDOCo0FZzUEPFJuxeRrnz08Rj S8+ok1fLttjuPOKEevXfMjeI4+YcLND7xfSGg= MIME-Version: 1.0 Received: by 10.210.114.15 with SMTP id m15mr1137747ebc.0.1249636651909; Fri, 07 Aug 2009 02:17:31 -0700 (PDT) Date: Fri, 7 Aug 2009 05:17:31 -0400 Message-ID: From: grarpamp To: freebsd-performance@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Fri, 07 Aug 2009 16:27:16 +0000 Subject: RELENG_7 heavy disk = system crawls X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Aug 2009 09:40:17 -0000 Hi. I'm running RELENG_7 on an older single P4. It has a lot of disk on it that does mainly sequential read/write of gigs of data. The data disks are hanging off a dumb ata133 pdc20269 card, they use geli aes 128 and zfs sha256, single spindles. Free ram, no swap, free disk, no net, etc. In short, whenever I'm doing sequential disk stuff, human interface system performance tanks, big time. Keystrokes take a second or two to appear, X11 focus raises take forever, firefox is a dog... after waiting 10min for the system to load it off disk, run it and paint it to X11, mouse movements are largely ignored unless moved very slowly. Even typing killall is painful to do :) I am very near the limit on wired mem, despite telling zfs to use only 96MiB. And sometimes X11 can't get the ram it wants and dies mid session. But neither of those should affect performance, the kernel will deal with low mem one way or another, drastically, not by bogging down [no swap enabled anyways] right? I used to do similar stuff on RELENG_4 on an old dual PII with plain UFS and I could run gigs of disk data, all the spindles, 8 of them, through sha1, all at once, and still have good interactive performance. Yet fire off one sequential read/write, perhaps piped into sha1 for more cpu load and it's game over. I can renice the user process way down into idprio with no effect. So I doubt giving rtprio to all the human processes would do anything. Is RELENG_7 or GELI+ZFS the dog here? Top of course shows GELI+ZFS eating all the cpu. Ok, fine, so how do I say, hey, I don't care about disk, it'll finish eventually, just give me my gui and keystrokes back? It just doesn't seem right that this one subsystem should be monopolizing the cpu this way? Help/cluebat? Thanks. From owner-freebsd-performance@FreeBSD.ORG Sat Aug 8 05:53:35 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8D052106566B for ; Sat, 8 Aug 2009 05:53:35 +0000 (UTC) (envelope-from freebsd@sopwith.solgatos.com) Received: from sopwith.solgatos.com (pool-173-50-131-130.ptldor.fios.verizon.net [173.50.131.130]) by mx1.freebsd.org (Postfix) with ESMTP id 7FFD78FC08 for ; Sat, 8 Aug 2009 05:53:34 +0000 (UTC) Received: by sopwith.solgatos.com (Postfix, from userid 66) id 7FC6CB64F; Fri, 7 Aug 2009 22:07:11 -0700 (PDT) Received: from localhost by sopwith.solgatos.com (8.8.8/6.24) id FAA18701; Sat, 8 Aug 2009 05:33:59 GMT Message-Id: <200908080533.FAA18701@sopwith.solgatos.com> To: freebsd-performance@freebsd.org In-reply-to: Your message of "Fri, 07 Aug 2009 05:17:31 EDT." Date: Fri, 07 Aug 2009 22:33:59 PDT From: Dieter Subject: Re: RELENG_7 heavy disk = system crawls X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Aug 2009 05:53:35 -0000 > Hi. I'm running RELENG_7 on an older single P4. It has a lot of > disk on it that does mainly sequential read/write of gigs of data. > In short, whenever I'm doing sequential disk stuff, human interface > system performance tanks, big time. > I used to do similar stuff on RELENG_4 on an old dual PII with plain > UFS and I could run gigs of disk data, all the spindles, 8 of them, > through sha1, all at once, and still have good interactive performance. > > Yet fire off one sequential read/write, perhaps piped into sha1 for > more cpu load and it's game over. I can renice the user process way > down into idprio with no effect. So I doubt giving rtprio to all the > human processes would do anything. > > Is RELENG_7 or GELI+ZFS the dog here? Top of course shows > GELI+ZFS eating all the cpu. Ok, fine, so how do I say, hey, I don't > care about disk, it'll finish eventually, just give me my gui and > keystrokes back? > > It just doesn't seem right that this one subsystem should be > monopolizing the cpu this way? Help/cluebat? Thanks. Are you *sure* that the cpu is your bottleneck? I see problems where sequential read/write of gigs of data on one disk interferes with i/o to other disks, and it isn't a cpu problem in my case. Just the opposite, cpu bound jobs don't create the problem, i/o bound jobs do. As expected, idprio doesn't help my problem either. I suspect that the problem is the i/o bound job is hogging the disk cache or some similar resource. I run FFS, no ZFS, and I've seen the same problem on 6.0, 6.2, 7.0 and 7.1. Unix grew up on machines where cpu was always the scarce resource, so nice only cares about cpu. Today i/o is often the scarce resource, so we need a way to nice i/o up/down similar to nicing cpu up/down. FreeBSD is supposed to be the "performance" BSD, but watching it leave several disks idle while one disk does i/o makes me wonder. From owner-freebsd-performance@FreeBSD.ORG Sat Aug 8 09:02:49 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B48BB106564A for ; Sat, 8 Aug 2009 09:02:49 +0000 (UTC) (envelope-from grarpamp@gmail.com) Received: from ey-out-2122.google.com (ey-out-2122.google.com [74.125.78.26]) by mx1.freebsd.org (Postfix) with ESMTP id 3A8E48FC1A for ; Sat, 8 Aug 2009 09:02:48 +0000 (UTC) Received: by ey-out-2122.google.com with SMTP id 9so523531eyd.7 for ; Sat, 08 Aug 2009 02:02:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type:content-transfer-encoding; bh=XtPunWYkcOzmEVObxOH0CxMZx0Wg1s6dYpc1+lIsiC8=; b=fIIGdXZrg9oq0ZLcLVwX2sPcQiiuj6EU0WY05IocPz4LqQRQxWxDVLM5hWFu3gnvsf 1jZRn2QgNEfxdfPEO+N+WZG4VhO3ez9dIUoTbrkpEnu9XD2xDRKwLfqC2EvPrxZ3vh7W 8qNS4DmsnkxiXMYrEE27GFbonJ2PvxUm3Mvm8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type :content-transfer-encoding; b=UziNRb/xZ+SFztZfyyBgF3ve0P4es8ZZu2ehwQdpTAZI8UWA2KyYcTInm/u/drC46M X7ejqqqMnSlW6eScesemEMd2FLcRGjmjU+7gc5xk3ayly3qusAGTXoJHdTp0DZzrFS7E mAwTWcjiK86BM8n0n9fD4E6r1hs9znpySVEzA= MIME-Version: 1.0 Received: by 10.210.114.15 with SMTP id m15mr2581371ebc.0.1249722168175; Sat, 08 Aug 2009 02:02:48 -0700 (PDT) Date: Sat, 8 Aug 2009 05:02:48 -0400 Message-ID: From: grarpamp To: freebsd-performance@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Sat, 08 Aug 2009 12:43:56 +0000 Subject: RELENG_7 heavy disk = system crawls X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Aug 2009 09:02:49 -0000 > Are you *sure* that the cpu is your bottleneck? Well, I've got a gig free in /usr/local which is ufs2+softdeps. So I just dd if=dev/zero bs=1m of=zero there. Disk was 100% busy, cpu was 10-15% system, 10% user, about 16MiB/sec. Of course since that is my system spindle, and it was busied out by dd, I had to wait 10 sec or so for vi to load to write this note during it :) But the rest of the interface was responsive. Though I've never run RELENG_4 on this box, I'd venture it would feel similar in this case. I can dd if=/dev/ad[n].eli of=/dev/null bs=1m and use 75% system all in geli, 27% disk busy, 20MiB/sec. Interface was slower but reasonable. Now do a dd if=file of=/dev/null bs=1m where file is on zfs on the geli spindle above and the system melts down. Half cpu in geli, half spread over about 8 spa_zio's, 4MiB/sec, all system time, disk 100% busy. I'm not sure yet how to isolate cpu from i/o under my geli+zfs setup. I think they're mated together. Curiously, I've got about 104 spa_zio procs all in tq->tq DL state. Only about 20 or so have more than a little system time on them. But about 9 zfs mounts so that may be ok, don't know. Don't get me wrong, FreeBSD has been my primary os since RELENG_2_2. And it's been great, still is. I recommend and use it all the time. It's just that this workload has really put the screws to things and I don't see a way out. I'd like to deploy geli+zfs everywhere but if I can't login remotely because some user has it busied out on something I've no knobs to control, umm, yeah :) I'm curious to see what others running geli_aes128+zfs_sha256 are seeing in this regard. And I'd love to see what a fast dual or more core amd64 system would be like under this workload. As to your i/o thing, I think back in RELENG_4 that if all the spindles were on the same pata controller/interrupt, monopolistic loads could occur. Seemed to be a hardware thing, not BSD. IE: At the moment I've got a half dozen spindles and filesystems spread out under RELENG_4 all happily doing find | xargs sha1 at once, no problems. That hardware is set for update to RELENG_7 or RELENG_8 in a few weeks. > we need a way to nice i/o up/down That would be handy for sure. User spindles, system spindles, storage, net, keyboard, etc. # systime spread 11 root 1 171 ki31 0K 8K RUN 46.5H 88.18% idle: cpu0 3215 root 1 -8 - 0K 8K geli:w 737:30 1.76% g_eli[0] ad6 607 root 1 -8 - 0K 8K geli:w 158:12 0.00% g_eli[0] ad4 3235 root 1 -16 - 0K 24K tq->tq 69:41 0.00% spa_zio 3229 root 1 -16 - 0K 24K tq->tq 69:40 0.00% spa_zio 3228 root 1 -16 - 0K 24K tq->tq 69:39 0.00% spa_zio 3234 root 1 -16 - 0K 24K tq->tq 69:39 0.00% spa_zio 3233 root 1 -16 - 0K 24K tq->tq 69:39 0.00% spa_zio 3232 root 1 -16 - 0K 24K tq->tq 69:39 0.00% spa_zio 3231 root 1 -16 - 0K 24K tq->tq 69:38 0.00% spa_zio 3230 root 1 -16 - 0K 24K tq->tq 69:37 0.00% spa_zio 1135 user 1 44 0 169M 152M select 56:02 0.00% XFree86 954 root 1 -16 - 0K 24K tq->tq 17:10 0.00% spa_zio 958 root 1 -16 - 0K 24K tq->tq 17:10 0.00% spa_zio 956 root 1 -16 - 0K 24K tq->tq 17:09 0.00% spa_zio 953 root 1 -16 - 0K 24K tq->tq 17:09 0.00% spa_zio 957 root 1 -16 - 0K 24K tq->tq 17:09 0.00% spa_zio 952 root 1 -16 - 0K 24K tq->tq 17:09 0.00% spa_zio 951 root 1 -16 - 0K 24K tq->tq 17:09 0.00% spa_zio 955 root 1 -16 - 0K 24K tq->tq 17:09 0.00% spa_zio 613 root 1 -8 - 0K 8K geli:w 16:12 0.00% g_eli[0] ad7 3 root 1 -8 - 0K 8K - 15:05 0.00% g_up