From owner-freebsd-pf@FreeBSD.ORG Mon Aug 8 11:52:58 2005 Return-Path: X-Original-To: freebsd-pf@freebsd.org Delivered-To: freebsd-pf@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 83A1A16A41F for ; Mon, 8 Aug 2005 11:52:58 +0000 (GMT) (envelope-from slapinid@gmail.com) Received: from zproxy.gmail.com (zproxy.gmail.com [64.233.162.202]) by mx1.FreeBSD.org (Postfix) with ESMTP id DA04143D45 for ; Mon, 8 Aug 2005 11:52:57 +0000 (GMT) (envelope-from slapinid@gmail.com) Received: by zproxy.gmail.com with SMTP id z6so565723nzd for ; Mon, 08 Aug 2005 04:52:57 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=K0HdW+UK235IeoGc+0rcOm2xAv/1GK7Tcy+FF/AEIy+MCRllaSZoz/Ojh68xHpX2pnS7gaupT2R1I5s/2u7Bv0zUsPJ+3LhYmpjpRqFW80ReNUFtuz7w30E2wN3uNAJP2WrTRL2qnuoPDs9sNTReetnqwQEp0ctfL50lvmpFVFg= Received: by 10.36.36.14 with SMTP id j14mr2519750nzj; Mon, 08 Aug 2005 04:52:57 -0700 (PDT) Received: by 10.36.33.4 with HTTP; Mon, 8 Aug 2005 04:52:55 -0700 (PDT) Message-ID: <48239d390508080452270c8d10@mail.gmail.com> Date: Mon, 8 Aug 2005 15:52:55 +0400 From: Sergey Lapin To: freebsd-pf@freebsd.org In-Reply-To: <200508060411.05482.max@love2party.net> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline References: <48239d390508040958265ce62@mail.gmail.com> <48239d3905080504297b3ebc89@mail.gmail.com> <200508060411.05482.max@love2party.net> Subject: Re: Fwd: pf problems X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Aug 2005 11:52:58 -0000 > What version of FreeBSD are you running? 5.4-RELEASE > Do you have a SMP/PREEMPTION kernel? No > Does setting debug.mpsafenet=3D0 in loader.conf change the situation? Do= you > have a chance to attach a remote debugger or can you try to break into th= e > debugger from the console? Will try it, thanks for idea. Here's more proper description of the problem: Hi. While trying to configure pf on our freebsd 5.4 router we have encountered three bugs, one of which is minor, second is more serious. And the worst of all, because of the second bug we have to use workaround and that workaround triggers third bug which is actually critical (because it hangs device and floods the network). I'll try to explain these bugs briefly. Please advice where to report each fo them. We haev no idea if they are PF global or FreeBSD specific. We are going to repeat all the scenarios on OpenBSD. =20 Background: we have two ISP connections and it is the source of all our problems - looks like work in such a mode is not debugged very well in pf. =20 Setup: Let's say ISP1 gives us 1.0.0.0/24 block with their gateway at 1.0.0.1. Our router is at 1.0.0.254. Similarly, ISP2 gives us 2.0.0.0/24, their gateway is at 2.0.0.1 and our router is at 2.0.0.254. ISP2 gateway (2.0.0.1) is our default gateway. We have DMZ where hosts of both ISP network live (dmz_net1=3D1.0.0.128/25 and dmz_net2=3D2.0.0.128/25). +---------------+ 1.0.0.254 | | -------------------+ ext_if1 | | | dmz_net1 | dmz_if +-------- 2.0.0.254 | | dmz_net2 -------------------+ ext_if2 | | | +---------------+ =20 Note that firewall rules below are oversimplified to make rule listing compact. In our "real" firewall we differentiate between TCP and UDP/ICMP traffic to make sure all outgoing TCP connections use "modulate state" insted of "keep state" etc. Also ruleset below loosened in terms it uses "any" where our real ruleset explicitly defines networks. Again, this is only to demostrate problem with minimum amount of rules. =20 **************************************************** ****************** BUG#1 (minor) ******************* **************************************************** =20 When pf blocks incoming packet with "block return" rule, it does not return RST or ICMP packet to the interface from which original packet came from but always use default gateway instead. This way if we have default gateway set to ISP2's 2.0.0.1 and packet destined to 1.0.0.254 comes from ISP1 interface (ext_if1) and this packet gets blocked with "block return", the TCP RST packet with source address 1.0.0.254 will be sent through 2.0.0.1 gateway. Obviously, ISP2 drops packets which source does not belong to their network so basically "block return" does not work at all. =20 **************************************************** ****************** Bug#2 (severe) ****************** **************************************************** =20 ruleset: #########################################################33 =20 # nat outgoing connections on each internet interface nat on $ext_if1 from { $dmz_net2 } to any -> ($ext_if1) nat on $ext_if2 from { $dmz_net1 } to any -> ($ext_if2) =20 # default deny silently block drop all =20 # pass in quick any packets destined for the gateway itself pass in quick on $dmz_if from any to $dmz_if keep state =20 pass quick on lo0 =20 # Classify traffic from DMZ # Allow all outgoing connections from DMZ =20 pass in on $dmz_if inet from $dmz_net1 to any keep state tag DMZ_TO_EXT1 pass in on $dmz_if inet from $dmz_net2 to any keep state tag DMZ_TO_EXT2 =20 # Allow gateway to route between different networks on the DMZ pass in on $dmz_if inet from { $dmz_net1, $dmz_net2 } to { $dmz_net1, $dmz_net2 } keep state tag DMZ_TO_DMZ =20 # Reroute OUT traffic appropriately pass out quick on $ext_if1 route-to ($ext_if2 $ext_gw2) tagged DMZ_TO_EXT2 keep state pass out quick on $ext_if2 route-to ($ext_if1 $ext_gw1) tagged DMZ_TO_EXT1 keep state =20 # general "pass out" rules for external interfaces pass out on { $ext_if1, $ext_if2, $dmz_if } from any to any keep state =20 #########################################################33 =20 Note that route-to rules should re-route all the traffic to proper interface regardless of default gateway specified. =20 What happens with such a rules:=20 * When a host from dmz_net1 sends packets to the internet and default gateway is ISP1 one, everything works fine and no translation is performed. * When we change default gateway to the ISP2 very interesting thing happens: packet from dmz_net1 host leaves ext_if1 BUT it gets translated like it should leave ext_if2! =20 I do not understand this one completely. =20 **************************************************** ***************** Bug#3 (critical) ***************** **************************************************** =20 Because route-to for "out" rules did not work well, we decided to add the redirection for "in" rules so we added =20 pass in quick on $dmz_if route-to ($ext_if1 $ext_gw1) tagged DMZ_TO_EXT1 keep state pass in quick on $dmz_if route-to ($ext_if2 $ext_gw2) tagged DMZ_TO_EXT2 keep state =20 right before before "pass out quick" rules. Everything started working right and it worked fine for some time but then firewall died (machine do not respond to keyboard and the only bring it back to live was to bounce power). After rebooting situation repeated - router was working well for 2 to 10 minutes and then unexpectedly freeze. We discovered that problem was caused by broadcast UDP packet coming on dmz_if and destined to 255.255.255.255 Later we discovered that this situation is triggered by any packet for which both conditions are true: 1. destination MAC is broadcast 2. destination IP is none of router's directly connected networks =20 Any such a packet kills the router. Actually, router is not completely dead - it sends that damn packet over and over at huge speed to the outer interface. =20 Even if I'm doing something completely wrong, router should NOT hang that w= ay. =20 =20 No idea if these bugs are PF's or they are FreeBSD specific.