From owner-freebsd-current@FreeBSD.ORG Sun Jan 27 09:30:35 2008 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 552A016A419 for ; Sun, 27 Jan 2008 09:30:35 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 7A6B213C43E for ; Sun, 27 Jan 2008 09:30:34 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id E39A51B10EF2; Sun, 27 Jan 2008 10:30:32 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-8.8 required=5.0 tests=ALL_TRUSTED,BAYES_00, J_CHICKENPOX_52,J_CHICKENPOX_55,J_CHICKENPOX_56,NORMAL_HTTP_TO_IP autolearn=no version=3.2.3 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id 1B9151B10EE0; Sun, 27 Jan 2008 10:30:26 +0100 (CET) Message-ID: <479C4F31.7090804@moneybookers.com> Date: Sun, 27 Jan 2008 11:30:25 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (X11/20071120) MIME-Version: 1.0 To: Max Laier References: <479A2389.2000802@moneybookers.com> <200801262017.52091.max@love2party.net> <479B9F4F.5010705@moneybookers.com> <200801262227.36970.max@love2party.net> In-Reply-To: <200801262227.36970.max@love2party.net> Content-Type: multipart/mixed; boundary="------------070403050709000400040305" X-Virus-Scanned: ClamAV 0.91.2/5572/Sun Jan 27 06:16:23 2008 on blah.cmotd.com X-Virus-Status: Clean X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-current@freebsd.org Subject: Re: FreeBSD 7, bridge, PF and syn flood = very bad performance X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Jan 2008 09:30:35 -0000 This is a multi-part message in MIME format. --------------070403050709000400040305 Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit Goo Day, Max Laier wrote: > On Saturday 26 January 2008, Stefan Lambrev wrote: > >> Max Laier wrote: >> >>> On Friday 25 January 2008, Stefan Lambrev wrote: >>> >>>> Greetings, >>>> >>>> Does anyone try to see PF with "keep state" in action when under syn >>>> flood attack? >>>> I tried to get some help in freebsd-pf@, because the test firewall, >>>> that I build hardly can handle 2-5MB/s syn flood. >>>> Unfortunately I do not saw useful advice. >>>> The problem is that a quad core bridge firewall running freebsd 7 >>>> amd64 with PF is near useless and can't handle "small" SYN ddos. >>>> >>>> Here is the schema that I'm testing: >>>> web server (freebsd) - freebsd (bridged interfaces) - gigabit switch >>>> - clients + flooders >>>> In this configuration ~25MB/s syn flood (and I think this limit is >>>> because of my switch) is not a problem and the web server responds >>>> without a problem. >>>> With this configuration netperf -l 610 -p 10303 -H 10.3.3.1 shows >>>> 116MB/s stable speed , so I guess there are no problems with cables, >>>> hardware and etc :) >>>> >>>> But when I start pf (see below the config file) the traffic drops to >>>> 2-3MB/s and the web server is hardly accessible. >>>> It seems that device polling helps a lot in this situation, and at >>>> least the bridge firewall is accessible. Without "polling" the >>>> firewall is so heavily loaded >>>> that even commands like "date" take few seconds to finish, with 2 >>>> cores at ~100% idle at same time. >>>> >>>> I have "flat profiles" from hwpmc, and I think it indicates a >>>> problem: >>>> >>>> (bridge, pf enabled, polling enabled, sched_ule - I have profiles >>>> and for other combinations too if needed) >>>> % cumulative self self total >>>> time seconds seconds calls ms/call ms/call name >>>> 24.0 268416.00 268416.00 0 100.00% >>>> _mtx_lock_sleep >>>> >>> Can you build a kernel with LOCK_PROFILING and try to figure out >>> which lock is causing this? >>> >> Yes I can build kernel with LOCK_PROFILING. >> But I have no idea how to use it :) >> Can you point me to some documentation? >> > > man LOCK_PROFILING > > basically: > # sysctl debug.lock.prof.enable=1 && sleep 60 && \ > sysctl debug.lock.prof.enable=0 && \ > sysctl debug.lock.prof.stats > log > > while under attack to sample one minute of lock statistics. > > Well I think the interesting lines from this experiment are: max total wait_total count avg wait_avg cnt_hold cnt_lock name 39 25328476 70950955 9015860 2 7 5854948 6309848 /usr/src/sys/contrib/pf/net/pf.c:6729 (sleep mutex:pf task mtx) 936935 10645209 350 50 212904 7 110 47 /usr/src/sys/contrib/pf/net/pf.c:980 (sleep mutex:pf task mtx) 41 10528492 1422891 1492295 7 0 155627 216812 /usr/src/sys/dev/em/if_em.c:980 (sleep mutex:em1) 26 5894103 2275517 2254004 2 1 427066 715901 /usr/src/sys/net/if_bridge.c:2082 (sleep mutex:if_bridge) 34 5466679 118638 761766 7 0 1198 5794 /usr/src/sys/dev/em/if_em.c:980 (sleep mutex:em0) 24 4274965 1952823 2253930 1 0 201352 691434 /usr/src/sys/net/if_bridge.c:1991 (sleep mutex:if_bridge) 28 3067953 800284 1492265 2 0 113423 294092 /usr/src/sys/net/if_bridge.c:1674 (sleep mutex:em1) 776401 1972047 0 69 28580 0 0 0 /usr/src/sys/kern/uipc_sockbuf.c:145 (sx:so_snd_sx) 775844 1970701 0 69 28560 0 1 0 /usr/src/sys/netinet/tcp_usrreq.c:779 (sleep mutex:inp) 22 1552808 922 761744 2 0 6 405 /usr/src/sys/dev/em/if_em.c:949 (sleep mutex:em0) 19 1508717 94 761736 1 0 51 24 /usr/src/sys/net/if_bridge.c:1674 (sleep mutex:em0) 15 713930 7045 590468 1 0 1778 3364 /usr/src/sys/kern/kern_timeout.c:419 (spin mutex:callout) 9 693209 4395 579397 1 0 1305 2129 /usr/src/sys/kern/kern_timeout.c:500 (spin mutex:callout) 23 569860 423 88509 6 0 51 100 /usr/src/sys/kern/subr_taskqueue.c:71 (spin mutex:fast_taskqueue) 46 489089 188 90306 5 0 6 7 /usr/src/sys/kern/subr_sleepqueue.c:232 (spin mutex:sleepq chain) 102 488839 28464 19935 24 1 15840 5849 /usr/src/sys/dev/em/if_em.c:1563 (sleep mutex:em1) 70692 443077 0 24 18461 0 0 0 /usr/src/sys/sys/buf.h:280 (lockmgr:bufwait) 61 291437 6501 8148 35 0 5664 1610 /usr/src/sys/dev/em/if_em.c:1563 (sleep mutex:em0) 27 2760115 474506 1346693 2 0 102015 137670 /usr/src/sys/dev/em/if_em.c:949 (sleep mutex:em1) 246691 246691 0 1 246691 0 0 0 /usr/src/sys/netinet/tcp_timer.c:423 (sleep mutex:tcp) 13 121639 10 60134 2 0 0 2 /usr/src/sys/kern/kern_clock.c:224 (spin mutex:sched lock 0) 13 119466 1 60135 1 0 0 1 /usr/src/sys/kern/kern_clock.c:224 (spin mutex:sched lock 3) 9 111044 5 60134 1 0 0 1 /usr/src/sys/kern/kern_clock.c:224 (spin mutex:sched lock 1) 107 107 246687 1 107 246687 0 1 /usr/src/sys/netinet/tcp_timer.c:438 (sleep mutex:inp) you can see the whole file here - http://89.186.204.158/profiling.txt -- Best Wishes, Stefan Lambrev ICQ# 24134177 --------------070403050709000400040305--