From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 04:16:43 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 24D40106564A for ; Tue, 1 Jul 2008 04:16:43 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id D60598FC17 for ; Tue, 1 Jul 2008 04:16:42 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KDXEw-0000tx-MQ for freebsd-net@freebsd.org; Tue, 01 Jul 2008 00:13:14 -0400 Message-ID: <4869B025.9080006@gtcomm.net> Date: Tue, 01 Jul 2008 00:18:45 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: FreeBSD Net References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> In-Reply-To: <4869ACFC.5020205@gtcomm.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 04:16:43 -0000 Dual opteron 2212, amd64 kernel, GENERIC (same setup as my 270 I just posted except this is 64 bit and different NIC) Intel dual port 82571 , nothing changed in sysctl except fw and fastfw input (em0) output packets errs bytes packets errs bytes colls 455729 69094 27343822 9 0 1412 0 455582 67566 27334938 3 0 566 0 455525 66033 27331518 3 0 566 0 456632 68384 27397938 3 0 566 0 457144 70006 27428604 4 0 776 0 453551 71307 27213226 7 0 1038 0 455282 74669 27317022 6 0 1068 0 452268 75878 27136104 4 0 744 0 455447 76942 27326838 4 0 744 0 455408 68166 27324522 5 0 922 0 456113 74600 27366798 4 0 744 0 456316 68522 27378996 5 0 922 0 455888 71665 27353322 5 0 1034 0 453448 76407 27206904 4 0 744 0 453907 79158 27234444 4 0 744 0 452921 66532 27175278 4 0 744 0 456049 60947 27363048 8 0 1216 0 em0: Excessive collisions = 0 em0: Sequence errors = 0 em0: Defer count = 0 em0: Missed Packets = 131239807 em0: Receive No Buffers = 154088217 em0: Receive Length Errors = 0 em0: Receive errors = 0 em0: Crc errors = 0 em0: Alignment errors = 0 em0: Collision/Carrier extension errors = 0 em0: RX overruns = 4986006 em0: watchdog timeouts = 0 em0: XON Rcvd = 0 em0: XON Xmtd = 0 em0: XOFF Rcvd = 0 em0: XOFF Xmtd = 0 em0: Good Packets Rcvd = 969917996 em0: Good Packets Xmtd = 98120420 em0: TSO Contexts Xmtd = 2062 em0: TSO Contexts Failed = 0 errrrrrrrrrrorss..... 79 processes: 6 running, 52 sleeping, 3 stopped, 18 waiting CPU states: 0.0% user, 0.0% nice, 28.2% system, 0.0% interrupt, 71.8% idle Mem: 18M Active, 787M Inact, 241M Wired, 196K Cache, 213M Buf, 928M Free Swap: 8192M Total, 8192M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12 root 171 ki31 0K 16K CPU2 2 348:19 99.02% idle: cpu2 13 root 171 ki31 0K 16K CPU1 1 347:02 99.02% idle: cpu1 37 root -68 - 0K 16K CPU3 3 37:48 99.02% em0 taskq 14 root 171 ki31 0K 16K RUN 0 340:08 85.94% idle: cpu0 38 root -68 - 0K 16K - 1 8:46 12.06% em1 taskq How do you like those IDLE CPUS :) except 3 of course.. :> Paul wrote: > Dual opteron 270 > 32 bit GENERIC KERNEL Nothing changed in sysctl except forwarding and > ip forwarding > Broadcom interfaces on board NIC > > last pid: 11557; load averages: 1.13, 0.83, > 0.48 > up 0+03:24:26 21:58:38 > 70 processes: 6 running, 46 sleeping, 18 waiting > CPU states: 0.0% user, 0.0% nice, 0.2% system, 28.8% interrupt, > 71.1% idle > Mem: 9124K Active, 6844K Inact, 26M Wired, 9776K Buf, 1957M Free > Swap: 4096M Total, 4096M Free > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 13 root 171 ki31 0K 8K CPU1 1 203:39 98.93% idle: cpu1 > 14 root 171 ki31 0K 8K RUN 0 203:26 98.05% idle: cpu0 > 33 root -68 - 0K 8K CPU2 2 13:06 93.99% irq24: bge0 > 11 root 171 ki31 0K 8K CPU3 3 201:59 79.10% idle: cpu3 > 34 root -68 - 0K 8K WAIT 3 1:49 18.90% irq25: bge1 > 12 root 171 ki31 0K 8K RUN 2 190:42 4.00% idle: cpu2 > > input (bge0) output > packets errs bytes packets errs bytes colls > 410143 65619 24608586 1 0 226 0 > 410350 66749 24621006 1 0 178 0 > 409452 65382 24567126 1 0 178 0 > 410338 63157 24620286 1 0 178 0 > 410125 65777 24607446 1 0 178 0 > 409794 63018 24587706 1 0 178 0 > 408208 67566 24492486 1 0 178 0 > 408416 70305 24504906 1 0 178 0 > 407919 68339 24475206 1 0 178 0 > > > sysctl -a | grep bge.0.stats > dev.bge.0.stats.FramesDroppedDueToFilters: 0 > dev.bge.0.stats.DmaWriteQueueFull: 310574781 > dev.bge.0.stats.DmaWriteHighPriQueueFull: 0 > dev.bge.0.stats.NoMoreRxBDs: 0 > dev.bge.0.stats.InputDiscards: 20942213 > dev.bge.0.stats.InputErrors: 15 > dev.bge.0.stats.RecvThresholdHit: 51440366 > dev.bge.0.stats.DmaReadQueueFull: 0 > dev.bge.0.stats.DmaReadHighPriQueueFull: 0 > dev.bge.0.stats.SendDataCompQueueFull: 0 > dev.bge.0.stats.RingSetSendProdIndex: 25738223 > dev.bge.0.stats.RingStatusUpdate: 51719806 > dev.bge.0.stats.Interrupts: 51719806 > dev.bge.0.stats.AvoidedInterrupts: 0 > dev.bge.0.stats.SendThresholdHit: 0 > dev.bge.0.stats.rx.Octets: 3477501149 > dev.bge.0.stats.rx.Fragments: 116 > dev.bge.0.stats.rx.UcastPkts: 519429709 > dev.bge.0.stats.rx.MulticastPkts: 0 > dev.bge.0.stats.rx.FCSErrors: 3 > dev.bge.0.stats.rx.AlignmentErrors: 0 > dev.bge.0.stats.rx.xonPauseFramesReceived: 0 > dev.bge.0.stats.rx.xoffPauseFramesReceived: 0 > dev.bge.0.stats.rx.ControlFramesReceived: 0 > dev.bge.0.stats.rx.xoffStateEntered: 0 > dev.bge.0.stats.rx.FramesTooLong: 0 > dev.bge.0.stats.rx.Jabbers: 0 > dev.bge.0.stats.rx.UndersizePkts: 0 > dev.bge.0.stats.rx.inRangeLengthError: 0 > dev.bge.0.stats.rx.outRangeLengthError: 0 > dev.bge.0.stats.tx.Octets: 2215096864 > dev.bge.0.stats.tx.Collisions: 0 > dev.bge.0.stats.tx.XonSent: 0 > dev.bge.0.stats.tx.XoffSent: 0 > dev.bge.0.stats.tx.flowControlDone: 0 > dev.bge.0.stats.tx.InternalMacTransmitErrors: 0 > dev.bge.0.stats.tx.SingleCollisionFrames: 0 > dev.bge.0.stats.tx.MultipleCollisionFrames: 0 > dev.bge.0.stats.tx.DeferredTransmissions: 0 > dev.bge.0.stats.tx.ExcessiveCollisions: 0 > dev.bge.0.stats.tx.LateCollisions: 0 > dev.bge.0.stats.tx.UcastPkts: 25738364 > dev.bge.0.stats.tx.MulticastPkts: 0 > dev.bge.0.stats.tx.BroadcastPkts: 15 > dev.bge.0.stats.tx.CarrierSenseErrors: 0 > dev.bge.0.stats.tx.Discards: 0 > dev.bge.0.stats.tx.Errors: 0 > > errors..ERRORS!)(@!*() > > Hitting that 400k pps limit again, and this is a slower machine than > my dual 2212. > Going to compile 7-STABLE with options for the cpu and will report back > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >