Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 31 Mar 2006 18:38:06 -0500
From:      Christopher McGee <chris@xecu.net>
To:        Greg Hennessy <Greg.Hennessy@nviz.net>
Cc:        freebsd-pf@freebsd.org
Subject:   Re: Traffic mysteriously dropping
Message-ID:  <442DBD5E.3090905@xecu.net>
In-Reply-To: <000001c654db$a5203440$0a00a8c0@thebeast>
References:  <000001c654db$a5203440$0a00a8c0@thebeast>

next in thread | previous in thread | raw e-mail | index | archive | help


Greg Hennessy wrote:

>>The first rule is block log all.  I put the http rules right 
>>after that  rule, or I lose connections.  
>>    
>>
>
>I would suggest minimising the amount of non stateful rules in the policy.
>They have a habit of biting one in the rear. 
>
>  
>
There are only 3 or 4 rules in the ruleset that are non-stateful.  I can 
try to eliminate those also.

>>The packets are not being logged as 
>>blocked. 
>>    
>>
>
>A block further down the policy without a 'log' may cause this. 
>
>One of the joys of debugging a last match policy. 
>
>  
>
All rules that are block are also using log.  A lot of the pass rules do 
not because it generates such enormous logs.  I can try enable logging 
on every rule temporarily in order to troubleshoot this if necessary.

>>The just never show up on the internal nic. 
>>    
>>
>
>I take it you've simultaneously tcpdumped both ingress and egress interfaces
>to confirm this ? Packets disappearing during traversal implies a routing
>issue. 
>
>  
>
Yes, if I tcpdump on em0, pflog0, and em1 simultaneously during a 
traffic test, the traffic hits em0, and never shows up as blocked in 
pflog0 and never shows up at all on em1.  As I stated, it's only 1 out 
of a bunch of connections, so there is no rule blocking all the traffic.

>  
>
>>I can make this a  bi-direction rule instead of keep state 
>>    
>>
>
>That's inefficient as each packet forces a rule base traversal.
>
>Try coding a tcp flow as  
>
>pass log quick on em inet proto tcp from source to dest port something keep
>state flags S/SA
>
>
>I said 'em' above, not em[0-9]. 
>
>Using interface groups without directionality, means that a single rule will
>match the flow on both the ingress and egress interfaces. 
>  
>Combined with antispoof, it makes for simpler policy 
>
>  
>
I have coded the rule as explained above and even as the first rule 
after the default block rule, it still drops traffic.  If I change it to 
non-stateful, it doesn't drop the connections.  I can't seem to get away 
from the thought of a state mis-match, however, I don't know why it 
would consistently do it on these http connections.

>>and it still drops traffic if I 
>>move it down in the list.  If I leave it where it is, and 
>>make it a keep state rule, it drops the connections also.
>>    
>>
>
>What other blocks are in the policy ? 
>
>  
>
I don't believe I'm doing any specifc blocks.  Just the default block 
and then allow what we need after that.

>>At peak times, the state table grows as large as 17000 states 
>>or so.  
>>    
>>
>
>If that's the case, the default table size is inadequate, I would set it to
>at least 25000. 
>
>  
>
I bumped up the state limit a long time ago.  I haven't run into issues 
with that.  There is adequate room in the state table even during peak 
times.  Unfortunately, it's dropping the traffic even during slow times.

>>Generic egress rules are a little difficult because I'm trying to do 
>>traffic shaping of certain traffic. 
>>    
>>
>
>Get the policy working without drops 1st, then shape. 
>
>  
>
Agreed, altq has been disabled until I can get the traffic flowing smoothly.

>>A side note, the machine is not doing any NAT.  Tagging seems like it
>>    
>>
>would require more 
>  
>
>>overhead than  what the firewall is doing already. 
>>    
>>
>
>It doesn't. Tagging works per stateful flow, not per packet.  Using tagging
>will permit you to significantly reduce the size of the rulebase. 
>
>  
>
The ruleset will be getting a significant rewrite, however, time has not 
permitted it yet.

Chris



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?442DBD5E.3090905>