From owner-freebsd-ipfw@freebsd.org Tue Oct 20 23:11:36 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 09C64A1514C for ; Tue, 20 Oct 2015 23:11:36 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from mail.7sq.com.au (mail.7sq.com.au [119.148.74.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7FED9141 for ; Tue, 20 Oct 2015 23:11:34 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id 0FD862C322E; Wed, 21 Oct 2015 09:00:20 +1000 (AEST) Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id YZlWCz70Bl8Q; Wed, 21 Oct 2015 09:00:19 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id CFE842C322F; Wed, 21 Oct 2015 09:00:19 +1000 (AEST) X-Virus-Scanned: amavisd-new at mail.7sq.com.au Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 0m4TtaeoATmv; Wed, 21 Oct 2015 09:00:19 +1000 (AEST) Received: from [192.168.156.153] (reddog2.lnk.telstra.net [110.142.196.96]) by mail.7sq.com.au (Postfix) with ESMTPSA id 6B25D2C322E; Wed, 21 Oct 2015 09:00:19 +1000 (AEST) Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Subject: Re: Kernel NAT issues From: Nathan Aherne In-Reply-To: <20151014232026.S15983@sola.nimnet.asn.au> Date: Wed, 21 Oct 2015 09:02:33 +1000 Cc: freebsd-ipfw@freebsd.org Message-Id: <9908EC22-344F-4D0B-8930-7D2C70B084A1@reddog.com.au> References: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> <20151013142301.B67283@sola.nimnet.asn.au> <20151014232026.S15983@sola.nimnet.asn.au> To: Ian Smith X-Mailer: Apple Mail (2.2102) Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Oct 2015 23:11:36 -0000 Hi Ian, Thank you very much for your response! Sorry about the late response, I = have been offline for a few days. I think I may have worked this issue out. I am bringing up a bunch of = Jails today to test my firewall rules in the hopes that I have corrected = my problem. I will reply back either way. Regards, Nathan > On 15 Oct 2015, at 12:51 am, Ian Smith wrote: >=20 > On Tue, 13 Oct 2015 13:50:04 +1000, Nathan Aherne wrote: >> Hi Ian, >>=20 >> Thank you for your response. >>=20 >> I didn=FF=FFt post my ruleset because I should be able to fix the = issue=20 >> myself but I see now that my request to explain =FF=FFhow NAT works=FF=FF= was=20 >> incorrect. >>=20 >> I have now included my ruleset below (as well as my initial email). >=20 > Hi Nathan, >=20 > I was really hoping someone who knows more about stateful rule = handling=20 > (and jail networking) might have a go at this. Oh well I'll try, but=20= > I'm a lousy mindreader, and really don't know which of the below=20 > constitutes 'hairpin NAT'. Perhaps showing your 'netstat -finet -an'=20= > and 'netstat -finet -rn' may shed light on routing? And 'ifconfig'? >=20 >> # Enable NAT >> ipfw nat 1 config ip $jip same_ports log >=20 > I'm assuming that $jip is your WAN IP, AAA.BBB.CCC.DDD .. and that=20 > WWW.XXX.YYY.ZZZ, from your posts in August, is another public IP = routed=20 > to you, and so traffic to it won't be subject to NAT .. correct? But=20= > the WWW... address and all 10.0/16 addresses are jails, not any = separate=20 > boxes you gateway for, right? Just the one external interface, right? >=20 >> 00005 allow ip from any to any via lo0 >> 00006 deny ip from any to not me in via bce0 >> 00100 nat 1 log ip from any to AAA.BBB.CCC.DDD recv bce0 >> 00101 check-state >=20 > Ok, inbound from WAN is nat'd and existing stateful flows followed by=20= > executing the rule that originally kept state. Where this is a = skipto,=20 > skipto will be performed. But where it's a nat rule, I've no idea ..=20= > see below, but you really don't want to add keep-state (again) there. >=20 >> 00110 allow icmp from any to WWW.XXX.YYY .ZZZ = recv bce0 keep-state >=20 > Hmm. I'd limit this to perhaps icmptypes 0,3,8,11 - though a = stateless=20 > rule would make more sense especially for inbound ICMP. But moving on = .. >=20 >> 00111 allow tcp from any to WWW.XXX.YYY .ZZZ = dst-port 65222 recv bce0 setup keep-state >=20 > Ok, but showting why plain text works better than HTML on lists :) >=20 >> 00112 allow icmp from WWW.XXX.YYY .ZZZ to any = xmit bce0 keep-state >> 00113 allow tcp from WWW.XXX.YYY .ZZZ to any = dst-port 53,80,443,22,65222 xmit bce0 setup keep-state >> 00114 allow udp from WWW.XXX.YYY .ZZZ to any = dst-port 53,123 xmit bce0 keep-state >=20 > Smells ok. >=20 >> 00120 skipto 65501 log tcp from any to 10.0.0.0/16 recv bce0 setup = keep-state >> 00121 skipto 65501 log udp from any to 10.0.0.0/16 recv bce0 = keep-state >=20 > Whoa, 65501 is your outbound NAT rule, albeit conditionally, and it's=20= > got a problem .. see below. These two are inbound traffic (recv) and = as=20 > is, skipping to 65501 will fall through two outbound rules to be = denied. >=20 > Either allow them here directly, or likely better, skipto a separate > target that then allows (or denies) them, if that's what you intended? >=20 >> 00122 skipto 65501 log tcp from 10.0.0.0/16 to not 10.0.0.0/16 xmit = bce0 setup keep-state >> 00123 skipto 65501 log udp from 10.0.0.0/16 to not 10.0.0.0/16 xmit = bce0 keep-state >=20 > Ok, this traffic does needs to be NAT'd on the way out. >=20 >> 00200 allow log tcp from any to 10.0.0.1 dst-port 22,80,443 in setup = keep-state >> 00200 allow log tcp from 10.0.0.1 to any dst-port 22,80,443 out setup = keep-state >> 00200 allow log udp from 10.0.0.1 to any dst-port 53 out keep-state >=20 > Not clear why these tcp ports are open inbound and outbound? = Presumably=20 > this is jail-to-jail traffic? Perhaps not relevant to your problem. >=20 >> 00201 allow log tcp from any to 10.0.0.2 dst-port 22,80,443 in setup = keep-state >> 00201 allow log tcp from 10.0.0.2 to any dst-port 22,80,443 out setup = keep-state >> 00201 allow log udp from 10.0.0.2 to any dst-port 53 out keep-state >> 65500 deny log ip from any to any >=20 > Ok. >=20 >> 65501 nat 1 log ip from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 = keep-state >=20 > This the target for outbound traffix, xmit bce0, so nat is = appropriate. =20 > Does jail-to-jail traffic travels via lo1? Or what? >=20 > This won't do anything to inbound traffic, but that really shouldn't = get=20 > here except returns as the result of check-state - not from 120 & 121. >=20 > But keep-state is not ok, state was already set on the skipto. I = don't=20 > know how this extra keep-state might behave - does anyone have an = idea? >=20 > Use 'ip4' rather than 'ip' in case this ever sees any ipv6 traffic. >=20 >> 65502 allow log ip from AAA.BBB.CCC.DDD to any xmit bce0 keep-state >=20 > So, only remaining traffic is outbound from the host itself, and = traffic=20 > that is to 10.0/16, but not from AAA... is to be dropped, correct? >=20 > I'm not sure whether 'allow ip .. keep-state' covers tcp, udp, icmp=20 > states .. myself, I'd go for separate rules for each eg tcp, udp, .. = and=20 > I'd do it somewhere else than as a fall through from outbound nat = rule,=20 > it's confusing here, to me anyway .. unless I've missed the reason? >=20 >> 65534 deny log ip from any to any >> 65535 deny ip from any to any >=20 >=20 > Ok, now for your demo of the problem from the later mail, which I've=20= > reformated to quote properly, so: >=20 >> To further illustrate my issue, this is a small log output. >>=20 >> I am running host google.com in the jail, which=20= >> has the IP 10.0.0.1. The UNKNOWN line is logging on the check-state=20= >> rule. >=20 > I see you don't have logging on 101 above now. Probably best. >=20 >> I would expect the first piece of traffic out would be UNKNOWN=20 >> (does not have an entry in the state table) but it seems the=20 >> returning traffic is also showing as UNKNOWN (the second 101). >=20 > I've never logged a check-state, but UNKNOWN may not mean that .. >=20 >> You can see that the traffic is returning on the same port it went=20 >> out on, so its obviously the returning traffic. I am not sure why=20 >> state is not being kept? >=20 > Well perhaps it is .. the return packet is from 8.8.8.8 to 10.0.0.1, = so=20 > it's been correctly NAT'd on the way in. Get rid of that keep-state = on=20 > the nat rule at 65501 and see if not creating double entries in the=20 > state table helps. And change the skipto target on 120 & 121 to only=20= > pass outbound traffic to outbound NAT rule/s. >=20 > Once you've done outbound NAT, probably best just to 'allow [log] = all'? >=20 >> Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 10.0.0.1:57446 = 8.8.8.8:53 out via bce0 >> Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP = 10.0.0.1:57446 8.8.8.8:53 out via bce0 >> Oct 13 15:50:42 host4 kernel: ipfw: 65501 Nat UDP 10.0.0.1:57446 = 8.8.8.8:53 out via bce0 >> Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 8.8.8.8:53 = 10.0.0.1:57446 in via bce0 >> Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 8.8.8.8:53 = 10.0.0.1:57446 in via bce0 >> Oct 13 15:50:42 host4 kernel: ipfw: 65534 Deny UDP 8.8.8.8:53 = 10.0.0.1:57446 in via bce0 >=20 > That said, I can see why this return packet would be denied even if it=20= > were in the nat table: it would execute 'skipto 65501', which nat rule=20= > does not apply, as it's not outbound, and rule 65502 does not apply, = as=20 > it's neither from AAA... nor outbound, so it's then denied by 65534. >=20 > Hope this helps. Please cc me on any response to the list. >=20 > It would be great if someone else might care to lend an oar here; I'm=20= > paddling out of my depth. >=20 > cheers, Ian >=20 > [..]