From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 08:50:35 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1F2901065682 for ; Wed, 2 Jul 2008 08:50:35 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id CFC1D8FC26 for ; Wed, 2 Jul 2008 08:50:34 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KDxzQ-0007sm-LX for freebsd-net@freebsd.org; Wed, 02 Jul 2008 04:47:00 -0400 Message-ID: <486B41D5.3060609@gtcomm.net> Date: Wed, 02 Jul 2008 04:52:37 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: FreeBSD Net References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> In-Reply-To: <486A9A0E.6060308@elischer.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 08:50:35 -0000 SMP DISABLED on my Opteron 2212 (ULE, Preemption on) Yields ~750kpps in em0 and out em1 (one direction) I am miffed why this yields more pps than a) with all 4 cpus running and b) 4 cpus with lagg load balanced over 3 incoming connections so 3 taskq threads I would be willing to set up test equipment (several servers plugged into a switch) with ipkvm and power port access if someone or a group of people want to figure out ways to improve the routing process, ipfw, and lagg. Maximum PPS with one ipfw rule on UP: tops out about 570Kpps.. almost 200kpps lower ? (frown) I'm going to drop in a 3ghz opteron instead of the 2ghz 2212 that's in here and see how that scales, using UP same kernel etc I have now. Julian Elischer wrote: > Paul wrote: >> ULE without PREEMPTION is now yeilding better results. >> input (em0) output >> packets errs bytes packets errs bytes colls >> 571595 40639 34564108 1 0 226 0 >> 577892 48865 34941908 1 0 178 0 >> 545240 84744 32966404 1 0 178 0 >> 587661 44691 35534512 1 0 178 0 >> 587839 38073 35544904 1 0 178 0 >> 587787 43556 35540360 1 0 178 0 >> 540786 39492 32712746 1 0 178 0 >> 572071 55797 34595650 1 0 178 0 >> >> *OUCH, IPFW HURTS.. >> loading ipfw, and adding one ipfw rule allow ip from any to any drops >> 100Kpps off :/ what's up with THAT? >> unloaded ipfw module and back 100kpps more again, that's not right >> with ONE rule.. :/ > > ipfw need sto gain a lock on hte firewall before running, > and is quite complex.. I can believe it.. > > in FreeBSD 4.8 I was able to use ipfw and filter 1Gb between two > interfaces (bridged) but I think it has slowed down since then due to > the SMP locking. > > >> >> em0 taskq is still jumping cpus.. is there any way to lock it to one >> cpu or is this just a function of ULE >> >> running a tar czpvf all.tgz * and seeing if pps changes.. >> negligible.. guess scheduler is doing it's job at least.. >> >> Hmm. even when it's getting 50-60k errors per second on the interface >> I can still SCP a file through that interface although it's not >> fast.. 3-4MB/s.. >> >> You know, I wouldn't care if it added 5ms latency to the packets when >> it was doing 1mpps as long as it didn't drop any.. Why can't it do >> that? Queue them up and do them in bigggg chunks so none are >> dropped........hmm? >> >> 32 bit system is compiling now.. won't do > 400kpps with GENERIC >> kernel, as with 64 bit did 450k with GENERIC, although that could be >> the difference between opteron 270 and opteron 2212.. >> >> Paul >> >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > >