Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 03 Apr 2006 17:12:48 -0400
From:      Christopher McGee <chris@xecu.net>
To:        Greg Hennessy <Greg.Hennessy@nviz.net>
Cc:        freebsd-pf@freebsd.org
Subject:   Re: Traffic mysteriously dropping
Message-ID:  <44318FD0.8050508@xecu.net>
In-Reply-To: <000001c65566$6e3bb630$0a00a8c0@thebeast>
References:  <000001c65566$6e3bb630$0a00a8c0@thebeast>

next in thread | previous in thread | raw e-mail | index | archive | help
Greg Hennessy wrote:

>>All rules that are block are also using log.  A lot of the 
>>pass rules do not because it generates such enormous logs.  I 
>>can try enable logging on every rule temporarily in order to 
>>troubleshoot this if necessary.
>>    
>>
>
>I would, you need to see what exactly is matching a flow. 
>
>  
>
>>> 
>>>
>>>      
>>>
>>Yes, if I tcpdump on em0, pflog0, and em1 simultaneously 
>>during a traffic test, the traffic hits em0, and never shows 
>>up as blocked in pflog0 and never shows up at all on em1.  As 
>>I stated, it's only 1 out of a bunch of connections, so there 
>>is no rule blocking all the traffic.
>>    
>>
>
>Hmmm, are you using route-to or such like in the policy ? If its not going
>out the interface you expect, it may be going out through another. 
>Time to tcpdump on everything including localhost to be sure. 
>
>Silly question,  is Jumbo frames enabled on one of the end points or are you
>running stock sized ethernet framing everywhere ?
>
>Has the firewall ever does transparent web caching ?
>
>Does the traffic route successfully if you disable pf with pfctl -d ? That
>should quickly determine if it's a routing or a firewall issue. 
>
>  
>
I am not using anything advanced like route-to.  The frames are the 
standard size, and this is not doing any kind of web caching or anything 
since the network behind it is mostly just mail servers, and a few web 
servers.  Unfortunately, since this is a production firewall, I can't 
really disable pf in this scenario.  I have done a simultaneous tcpdump 
on all the network interfaces, and pflog0 and lo0.  It did pretty much 
what I thought.  It would hit the outside interface, not even hit pf, 
and never pass through.  I have also found that it does log state 
mis-matches when this happens.  I found this with pf debugging enabled.

>>>Using interface groups without directionality, means that a 
>>>      
>>>
>>single rule 
>>    
>>
>>>will match the flow on both the ingress and egress interfaces.
>>> 
>>>Combined with antispoof, it makes for simpler policy
>>>
>>> 
>>>
>>>      
>>>
>>I have coded the rule as explained above and even as the 
>>first rule after the default block rule, it still drops 
>>traffic.  If I change it to non-stateful, it doesn't drop the 
>>connections.  I can't seem to get away from the thought of a 
>>state mis-match, however, I don't know why it would 
>>consistently do it on these http connections.
>>    
>>
>
>Hmmm, possibly something strange with the stack on the endpoints. 
>
>Are you using scrub in the policy ? 
>  
>
I am using scrub in the policy, however, as I've detailed above, this 
doesn't play a roll.

>>>What other blocks are in the policy ? 
>>>
>>> 
>>>
>>>      
>>>
>>I don't believe I'm doing any specifc blocks.  Just the 
>>default block and then allow what we need after that.
>>    
>>
>
>Time to do a quick grep to be completely sure, it's easy to miss one by just
>reading through a policy that large. 
>  
>
I have logging enabled in all the rules, so it will be logged no matter 
what it gets blocked by.  I think I have actually found the problem.  It 
is the state-mismatch, and it's because in our test scenario, all the 
requests are coming from 1 client.  There is a thread about this at 
http://www.benzedrine.cx/pf/msg07505.html.  If I choke down the 
tcp.closed time on the rule that allows this, it seems to minimize the 
problem.  Thanks for all the help everyone.

Chris



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?44318FD0.8050508>