From owner-freebsd-performance@FreeBSD.ORG Tue Aug 4 16:17:18 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 01BD0106566B for ; Tue, 4 Aug 2009 16:17:18 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outW.internet-mail-service.net (outw.internet-mail-service.net [216.240.47.246]) by mx1.freebsd.org (Postfix) with ESMTP id D509E8FC0A for ; Tue, 4 Aug 2009 16:17:17 +0000 (UTC) (envelope-from julian@elischer.org) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id 95C69961CA; Tue, 4 Aug 2009 09:17:17 -0700 (PDT) X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id E06492D6006; Tue, 4 Aug 2009 09:17:16 -0700 (PDT) Message-ID: <4A785F20.8050807@elischer.org> Date: Tue, 04 Aug 2009 09:17:36 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.22 (Macintosh/20090605) MIME-Version: 1.0 To: Invernizzi Fabrizio References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> <4A77094C.8030308@elischer.org> <36A93B31228D3B49B691AD31652BCAE9A45696721F@GRFMBX702BA020.griffon.local> In-Reply-To: <36A93B31228D3B49B691AD31652BCAE9A45696721F@GRFMBX702BA020.griffon.local> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-performance@freebsd.org" , Stefan Lambrev Subject: Re: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Aug 2009 16:17:18 -0000 Invernizzi Fabrizio wrote: >>>> The limitation that you see is about the max number of packets that >>>> FreeBSD can handle - it looks like your best performance is reached at >>>> 64 byte packets? >>> If you are meaning in term of Packet per second, you are right. These >> are the packet per second measured during tests: >>> 64 byte: 610119 Pps >>> 512 byte: 516917 Pps >>> 1492 byte: 464962 Pps >>> >>> >>>> Am I correct that the maximum you can reach is around 639,000 packets >>>> per second? >>> Yes, as you can see the maximum is 610119 Pps. >>> Where does this limit come from? >> ah that's the whole point of tuning :-) >> there are severalpossibities: >> 1/ the card's interrupts are probably attache dto aonly 1 cpu, >> so that cpu can do no more work > > This seems not to be the problem. See below a top snapshot during a 64byte-long packet storm > > last pid: 8552; load averages: 0.40, 0.09, 0.03 up 0+20:36:58 09:40:29 > 124 processes: 13 running, 73 sleeping, 38 waiting > CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle > Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free > Swap: 2048M Total, 2048M Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: cpu3 > 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: cpu0 > 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: cpu2 > 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: cpu1 > 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 rxq > 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 rxq > 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 rxq > 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 rxq > .... > > It looks like the 4 rxq processes are bound to the 4 available cores with equal distribution. > > > >> 2/ if more than 1 cpu is working, it may be that there is a lock in >> heavy contention somewhere. > > This I think is the problem. I am trying to understand how to > 1- see where the heavy contention is (context switching? Some limiting setting?) > 2- mitigate it > there ia a lock profiling tool that right now I can't remember the name of.. look it up with google :-) FreeBSD lock profiling tool ah, first hit... http://blogs.epfl.ch/article/23832 >> is the machine still responsive to other networks while >> running at maximum capacity on this network? (make sure that >> the other networks are on a differnet CPU (hmm I can't remember how to >> do that :-). > > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks.