From owner-freebsd-performance@FreeBSD.ORG Wed Feb 6 22:42:24 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0EF4716A419; Wed, 6 Feb 2008 22:42:24 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 7B09E13C45E; Wed, 6 Feb 2008 22:42:23 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 3CAD91B10EF8; Wed, 6 Feb 2008 23:42:22 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00, NORMAL_HTTP_TO_IP autolearn=ham version=3.2.3 Received: from [10.1.1.2] (unknown [192.168.25.10]) by blah.sun-fish.com (Postfix) with ESMTP id 6A8091B10EDC; Wed, 6 Feb 2008 23:42:19 +0100 (CET) Message-ID: <47AA37CA.8020208@moneybookers.com> Date: Thu, 07 Feb 2008 00:42:18 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Kris Kennaway References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> <47A799A6.3070502@moneybookers.com> <47A84751.8020109@moneybookers.com> <47A8D233.8020506@FreeBSD.org> <47A8DCD6.3060209@moneybookers.com> <47A8E1F1.4040309@FreeBSD.org> <47A98CDC.2090407@moneybookers.com> <47A993D0.1060901@FreeBSD.org> <47A99736.8060809@moneybookers.com> <47A99B16.6030305@FreeBSD.org> <47A9B636.3040509@moneybookers.com> <47A9C43A.3030203@moneybookers.com> <47AA1858.3050307@FreeBSD.org> In-Reply-To: <47AA1858.3050307@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.91.2/5713/Wed Feb 6 21:05:15 2008 on blah.cmotd.com X-Virus-Status: Clean Cc: freebsd-performance@freebsd.org Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 22:42:24 -0000 Greetings, Kris Kennaway wrote: > Stefan Lambrev wrote: > >>> I'll use again hwpmc and LOCK_PROFILING to see what's going on. >>> And will try the same benchmark on quad core processor as now >>> numbers of cores/cpus matter :) >>> >> Here are promised results - http://89.186.204.158/lock_profiling-8.txt Finally I got pmcstat working - http://89.186.204.158/hwpmc-p4.txt The stats are gathered during 600kpp incoming. I think that syncache or what calls MD5Transform is not SMP able, and that's why outgoing 250kpps is the limit that I can't beat. > > Thanks. There is further work needed on the route locking, and also > you are hitting limitations of the em driver (or possibly hardware; if > you only have a single transmit queue then outbound packets from > multiple CPUs have to be serialized in the driver no matter what). > Hopefully there will be further improvements in the coming months, and > these changes will also migrate into CVS. > > If you want to start hacking things to see how much further progress > is feasible, you can apply the attached hack that nulls out all route > locking :) This should be OK as long as your routes are not changing, > although you might get some spam on the console (if this is excessive, > comment out the printfs also ;-). It may not help much though, all > the contention will probably just fall through onto the ethernet driver. I'll do tomorrow. I still did not tested how if_lagg will perform in current situation. I thing that I didn't reached the maximum of the network card today. In this configuration best values was 1,122,674 incoming packets + 210kpps outgoing with both cpu cores 100% busy. systat -ifstat showed: em0 in 63.985 MB/s 63.985 MB/s 47.465 GB out 11.674 MB/s 11.723 MB/s 12.975 GB It is good, that now I hit a bottleneck that can be upgraded, if needed :) > >> Btw I got kernel panic first time when I run sysctl >> debug.lock.prof.stats > > Yeah, it is a bit broken in 8.0 even in CVS. Also make sure not to > reset it while the CPUs are loaded :) > >> I'm still trying to get hwpmc working with my cpu's and new kernel. >> Do you have any patches Kris? >> Is it supposed to work with your sources on my CPU? >> I can fetch your latest src/lib/libpmc from from p4 if this will help :) > > It works on my systems...try with libpmc from my branch, make sure to > install the new includes first and then rebuild and reinstall libpmc > and pmcstat. I have attached a patch against the CVS libpmc which > might be easier than checking it out from p4...it relies on kernel > changes also though, which are in the kernel you already have but not > in CVS. > > Kris Yes there was a problem with the installed include file, I edit it by hand, compiled new libpmc and it works again :)