From owner-freebsd-performance@FreeBSD.ORG Tue Feb 5 22:02:10 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0A34216A419; Tue, 5 Feb 2008 22:02:10 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 7F51413C43E; Tue, 5 Feb 2008 22:02:09 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id A84B21B10EF8; Tue, 5 Feb 2008 23:02:07 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, J_CHICKENPOX_55,NORMAL_HTTP_TO_IP autolearn=no version=3.2.3 Received: from [10.1.1.2] (unknown [192.168.25.10]) by blah.sun-fish.com (Postfix) with ESMTP id A1C9F1B10EA4; Tue, 5 Feb 2008 23:02:04 +0100 (CET) Message-ID: <47A8DCD6.3060209@moneybookers.com> Date: Wed, 06 Feb 2008 00:01:58 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Kris Kennaway References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> <47A799A6.3070502@moneybookers.com> <47A84751.8020109@moneybookers.com> <47A8D233.8020506@FreeBSD.org> In-Reply-To: <47A8D233.8020506@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.91.2/5700/Tue Feb 5 20:08:59 2008 on blah.cmotd.com X-Virus-Status: Clean Cc: freebsd-performance@freebsd.org, Andrew Thompson Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 22:02:10 -0000 Hello, Kris Kennaway wrote: > Stefan Lambrev wrote: > >>>>> Thanks for investigating this. One thing to note is that ip flows >>>>> from >>>>> the same connection always go down the same interface, this is >>>>> because >>>>> Ethernet is not allowed to reorder frames. The hash uses >>>>> src-mac, dst-mac, src-ip and dst-ip (see lagg_hashmbuf), make sure >>>>> when >>>>> performance testing that your traffic varies in these values. Adding >>>>> tcp/udp ports to the hashing may help. >>>>> >>>> The traffic, that I generate is with random/spoofed src part, so it >>>> is split between interfaces for sure :) >>>> >>>> Here you can find results when under load from hwpmc and >>>> lock_profiling: >>>> http://89.186.204.158/lock_profiling-lagg.txt > > OK, this shows the following major problems: > > 39 22375065 1500649 5690741 3 0 119007 > 712359 /usr/src/sys/net/route.c:147 (sleep mutex:radix node head) > 21 3012732 1905704 1896914 1 1 14102 > 496427 /usr/src/sys/netinet/ip_output.c:594 (sleep mutex:rtentry) > 22 120 2073128 47 2 44109 0 > 3 > /usr/src/sys/modules/if_lagg/../../net/ieee8023ad_lacp.c:503 > (rw:if_lagg rwlock) > 39 17857439 4262576 5690740 3 0 95072 > 1484738 /usr/src/sys/net/route.c:197 (sleep mutex:rtentry) > > It looks like the if_lagg one has been fixed already in 8.0, it could > probably be backported but requires some other infrastructure that > might not be in 7.0. > > The others are to do with concurrent transmission of packets (it is > doing silly things with route lookups). kmacy has a WIP that fixes > this. If you are interested in testing an 8.0 kernel with the fixes > let me know. Well those servers are only for tests so I can test everything, but at some point I'll have to make final decision what to use in production :) > >>>> http://89.186.204.158/lagg-gprof.txt >>>> >>> http://89.186.204.158/lagg2-gprof.txt I forget this file :) >>> >> I found that MD5Transform aways uses ~14% (with rx/txcsum enabled or >> disabled). > > Yeah, these don't have anything to do with MD5. Well I didn't find from where MD5Transform() is called, so I guess it's a some 'magic', that I still do not understand ;) > >> And when using without lagg MD5Transform pick up to 20% of the time. >> Is this normal? > > It is probably from the syncache. You could disable it > (net.inet.tcp.syncookies_only) if you don't need strong protection > against SYN flooding. > > Kris How the server perform during SYN flooding is exactly what I test at the moment :) So I can't disable this. Just for information, if someone is interested - I looked how linux (2.6.22-14-generic ubuntu) perform in the same situation .. by default it doesn't perform at all - it hardly replays to 100-200 packets/s, with syncookies enabled it can handle up to 70-90,000 pps (250-270,000 compared to freebsd), but the server is very loaded and not very responsible. Of course this doesn't mean that FreeBSD can't perform better ;) I plan to test iptables, newer kernel, various options, and may be few others distros.