From owner-freebsd-pf@FreeBSD.ORG Fri Mar 7 20:56:59 2008 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ADBE11065685 for ; Fri, 7 Mar 2008 20:56:59 +0000 (UTC) (envelope-from max@love2party.net) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.186]) by mx1.freebsd.org (Postfix) with ESMTP id 2D3BB8FC18 for ; Fri, 7 Mar 2008 20:56:58 +0000 (UTC) (envelope-from max@love2party.net) Received: from vampire.homelinux.org (dslb-088-066-002-164.pools.arcor-ip.net [88.66.2.164]) by mrelayeu.kundenserver.de (node=mrelayeu4) with ESMTP (Nemesis) id 0ML21M-1JXjce0MnP-00020X; Fri, 07 Mar 2008 21:56:57 +0100 Received: (qmail 36149 invoked by uid 80); 7 Mar 2008 20:56:22 -0000 Received: from 192.168.4.151 (SquirrelMail authenticated user mlaier) by router.laiers.local with HTTP; Fri, 7 Mar 2008 21:56:22 +0100 (CET) Message-ID: <54535.192.168.4.151.1204923382.squirrel@router.laiers.local> In-Reply-To: <523685.2819.qm@web53701.mail.re2.yahoo.com> References: <523685.2819.qm@web53701.mail.re2.yahoo.com> Date: Fri, 7 Mar 2008 21:56:22 +0100 (CET) From: "Max Laier" To: "Lorenz Helleis" User-Agent: SquirrelMail/1.4.13 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-Provags-ID: V01U2FsdGVkX1+LMgy82eLq19RrZM5NkNFp+UIwEI4h9Zwc0ai EGY/0CzEuEsyfFJ3EGm1JkHyYJbcKVXTWyyFUxu6bdKu8atF5J aCEGRY7rgHwSOEEETydGw== Cc: freebsd-pf@freebsd.org Subject: Re: Res: Res: Dropped Packets X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Mar 2008 20:56:59 -0000 AGAIN: PLEASE DON'T TOP-POST! Am Fr, 7.03.2008, 19:16, schrieb Lorenz Helleis: > Max.. > > the Current entry is not 5005. I got this value after "pfctl -d"... then these numbers are completely useless! > the number of concurrent connections is 70.000 Okay, so let's say every connection just passes ~10pps (that's not even 7kB/s with standard TCP) then you have to forward 700kpps. This is a *huge* load, even without firewalling. If you count in scrubbing and "just" statefull lookups, this is about the maximum that you can hope to push with commodity hardware. Sure, PCIe has removed one of the worst bottlenecks, but as I pointed out in my other reply - pf is still "giant"-locked and thus poses a bottleneck of it's own, but there are few (if any) alternatives. If you are serious about wanting a *firewall* for security. Otherwise you can use IPFW w/o states! Which will give some concurrency and less per-packet overhead due to fewer sanity checks. > > In this moment my firewall is disable until i find a solution to solve > this problem. I think i will try to increase the number of states and > change the NIC. > > I use a Gigabit card and the traffic is 300Mbs and the concurrent sessions > 70.000. > > And now i'm studing about tables entries, src-nodes .. > > > Provérbios 1:27 > > Mas Deus escolheu as coisas loucas deste mundo para confundir as > sábias; e Deus escolheu as coisas fracas deste mundo para confundir as > fortes; > > ----- Mensagem original ---- > De: Max Laier > Para: freebsd-pf@freebsd.org > Cc: Lorenz Helleis ; Chris Marlatt > > Enviadas: Sexta-feira, 7 de Março de 2008 14:55:52 > Assunto: Re: Res: Dropped Packets > > [ please don't top-post ] > > On Friday 07 March 2008, Lorenz Helleis wrote: >> I don't think that is a hardware problem, sometimes the "congestion >> rate" increase to 1500,0/s and the "state-mismatch" to 300.0/s.. I >> don't know if it is normal... >> >> I think that the conections is being droped when increase a lot the >> number of packets on the network. >> >> >> >> can you tell me about your firewall ? I will need to install a biggest >> one here, and I'm a little afraid to do. Can you show me some >> configuration? the traffic of you network?, hardware? conections ? >> >> look some configurations.... do i need to increase something ? >> >> >> # pfctl -sm >> states hard limit 100000 >> src-nodes hard limit 10000 >> frags hard limit 5000 >> tables hard limit 1000 >> table-entries hard limit 200000 >> >> >> # top >> >> load averages: 0.20, 0.12, 0.09 >> 13:29:40 35 processes: 34 idle, 1 on processor >> CPU0 states: 0.6% user, 0.0% nice, 0.7% system, 0.0% interrupt, >> 98.7% idle CPU1 states: 0.1% user, 0.0% nice, 0.2% system, 0.0% >> interrupt, 99.7% idle >> >> # vmstat -i >> >> interrupt total rate >> irq0/clock 257506609 199 >> irq0/ipi 183393879 142 >> irq81/em0 8638587188 6706 >> irq83/skc0 6011660768 4667 >> irq80/fxp0 2292732543 1779 > > These interrupt numbers don't seem to match up with the above load > numbers. I'd expect a higher interrupt load. You could also try to > replace the sk(4) adapter with another em(4) or the like? I have had > trouble with sk(4) in the past. > >> irq64/ahc0 7012560 5 >> irq112/pckbc0 8 0 >> Total 17390893555 13501 >> >> # pfctl -si >> >> State Table Total Rate >> current entries 5005 >> searches 30026832082 441000.4/s > > 441kpps are quite a load! And this is with only 5000 connections. While > FreeBSD can forward 1Mpps and more on commodity hardware 500-700kpps is > probably the limit with (sensible) firewalling. It'd be surprised if you > could do significantly better with anything else. N.B. that this could > be improved by using fine grained locking for pf - this is on my TODO > list for quite some time, but I didn't yet get to it. > >> inserts 406964726 5977.0/s >> removals 406959721 5977.0/s >> Counters >> match 417436387 6130.8/s >> bad-offset 0 0.0/s >> fragment 1939 0.0/s >> short 154 0.0/s >> normalize 34858 0.5/s >> memory 0 0.0/s >> bad-timestamp 0 0.0/s >> congestion 834349 12.3/s >> ip-option 24 0.0/s >> proto-cksum 5572 0.1/s >> state-mismatch 491286 7.2/s >> >> >> >> >> >> Provérbios 1:27 >> >> Mas Deus escolheu as coisas loucas deste mundo para confundir as >> sábias; e Deus escolheu as coisas fracas deste mundo para confundir as >> fortes; >> >> ----- Mensagem original ---- >> De: Chris Marlatt >> Para: Lorenz Helleis >> Cc: freebsd-pf@freebsd.org >> Enviadas: Sexta-feira, 7 de Março de 2008 12:26:03 >> Assunto: Re: Dropped Packets >> >> Lorenz Helleis wrote: >> > hello. >> > >> > I have a firewall with 75.000 simultaneous conections, and i set the >> > limit to 100.000. >> > >> > I think the hardware is OK, but when increase the traffic on the >> > network, some connections is dropped. I did not increase other >> > value, like table, src-nodes.... How do I know if is everthing ok >> > with the other values ? >> > >> > what happen if the number of connections touch the limit of 100.000 ? >> > it will drop the idle conections ? or what ? >> >> From my experience new connections will appear to timeout as PF has no >> more sessions available for new connections. As sessions die off >> organically new connections will be permitted but there is nothing >> actively killing old / idle connections to make way for new sessions if >> the limit is reached. >> >> >> Depending on how much memory you have you should be fine increasing the >> max session limit. I've had some of my firewalls over 1,000,000 >> sessions without a problem. >> >> You may want to check your switch for errors and watch your interface >> (netstat -I IFACE -nd 1) to see when/where your drops are. What kind of >> cpu usage are you seeing when you start dropping the packets? >> >> Regards, >> >> Chris >> >> >> >> >> >> >> Abra sua conta no Yahoo! Mail, o único sem limite de espaço para >> armazenamento! http://br.mail.yahoo.com/ >> _______________________________________________ >> freebsd-pf@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-pf >> To unsubscribe, send any mail to "freebsd-pf-unsubscribe@freebsd.org" > > > > -- > /"\ Best regards, | mlaier@freebsd.org > \ / Max Laier | ICQ #67774661 > X http://pf4freebsd.love2party.net/ | mlaier@EFnet > / \ ASCII Ribbon Campaign | Against HTML Mail and News > > > > > > > Abra sua conta no Yahoo! Mail, o único sem limite de espaço para > armazenamento! > http://br.mail.yahoo.com/ -- /"\ Best regards, | mlaier@freebsd.org \ / Max Laier | ICQ #67774661 X http://pf4freebsd.love2party.net/ | mlaier@EFnet / \ ASCII Ribbon Campaign | Against HTML Mail and News