Date: Tue, 05 Feb 2008 23:23:45 +0100 From: Kris Kennaway <kris@FreeBSD.org> To: Stefan Lambrev <stefan.lambrev@moneybookers.com> Cc: freebsd-performance@freebsd.org, Andrew Thompson <thompsa@FreeBSD.org> Subject: Re: network performance Message-ID: <47A8E1F1.4040309@FreeBSD.org> In-Reply-To: <47A8DCD6.3060209@moneybookers.com> References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <m21w7x5ilg.wl%gnn@neville-neil.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> <47A799A6.3070502@moneybookers.com> <47A84751.8020109@moneybookers.com> <47A8D233.8020506@FreeBSD.org> <47A8DCD6.3060209@moneybookers.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Stefan Lambrev wrote: > Hello, > > Kris Kennaway wrote: >> Stefan Lambrev wrote: >> >>>>>> Thanks for investigating this. One thing to note is that ip flows >>>>>> from >>>>>> the same connection always go down the same interface, this is >>>>>> because >>>>>> Ethernet is not allowed to reorder frames. The hash uses >>>>>> src-mac, dst-mac, src-ip and dst-ip (see lagg_hashmbuf), make sure >>>>>> when >>>>>> performance testing that your traffic varies in these values. Adding >>>>>> tcp/udp ports to the hashing may help. >>>>>> >>>>> The traffic, that I generate is with random/spoofed src part, so it >>>>> is split between interfaces for sure :) >>>>> >>>>> Here you can find results when under load from hwpmc and >>>>> lock_profiling: >>>>> http://89.186.204.158/lock_profiling-lagg.txt >> >> OK, this shows the following major problems: >> >> 39 22375065 1500649 5690741 3 0 119007 >> 712359 /usr/src/sys/net/route.c:147 (sleep mutex:radix node head) >> 21 3012732 1905704 1896914 1 1 14102 >> 496427 /usr/src/sys/netinet/ip_output.c:594 (sleep mutex:rtentry) >> 22 120 2073128 47 2 44109 0 >> 3 >> /usr/src/sys/modules/if_lagg/../../net/ieee8023ad_lacp.c:503 >> (rw:if_lagg rwlock) >> 39 17857439 4262576 5690740 3 0 95072 >> 1484738 /usr/src/sys/net/route.c:197 (sleep mutex:rtentry) >> >> It looks like the if_lagg one has been fixed already in 8.0, it could >> probably be backported but requires some other infrastructure that >> might not be in 7.0. >> >> The others are to do with concurrent transmission of packets (it is >> doing silly things with route lookups). kmacy has a WIP that fixes >> this. If you are interested in testing an 8.0 kernel with the fixes >> let me know. > Well those servers are only for tests so I can test everything, but at > some point I'll have to make final decision what to use in production :) http://www.freebsd.org/~kris/p4-net.tbz is a sys/ tarball from my p4 branch, which includes these and other optimizations. >>>>> http://89.186.204.158/lagg-gprof.txt >>>>> >>>> http://89.186.204.158/lagg2-gprof.txt I forget this file :) >>>> >>> I found that MD5Transform aways uses ~14% (with rx/txcsum enabled or >>> disabled). >> >> Yeah, these don't have anything to do with MD5. > Well I didn't find from where MD5Transform() is called, so I guess it's > a some 'magic', that I still do not understand ;) MD5Transform is an internal function called by other MD5* functions. Check netinet/tcp_syncache.c >> It is probably from the syncache. You could disable it >> (net.inet.tcp.syncookies_only) if you don't need strong protection >> against SYN flooding. >> >> Kris > How the server perform during SYN flooding is exactly what I test at the > moment :) > So I can't disable this. I thought this trace was on the machine you are transmitting the SYNs from, perhaps I misunderstood. > Just for information, if someone is interested - I looked how linux > (2.6.22-14-generic ubuntu) perform in the same situation .. by default > it doesn't perform at all - it hardly replays to 100-200 packets/s, > with syncookies enabled it can handle up to 70-90,000 pps (250-270,000 > compared to freebsd), but the server is very loaded and not very > responsible. > Of course this doesn't mean that FreeBSD can't perform better ;) What do you mean "compared to freebsd"? Kris
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?47A8E1F1.4040309>