Date: Wed, 21 Oct 2015 09:02:33 +1000 From: Nathan Aherne <nathan@reddog.com.au> To: Ian Smith <smithi@nimnet.asn.au> Cc: freebsd-ipfw@freebsd.org Subject: Re: Kernel NAT issues Message-ID: <9908EC22-344F-4D0B-8930-7D2C70B084A1@reddog.com.au> In-Reply-To: <20151014232026.S15983@sola.nimnet.asn.au> References: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> <20151013142301.B67283@sola.nimnet.asn.au> <C1C25100-FBD4-42F4-94F7-965B270D927F@reddog.com.au> <20151014232026.S15983@sola.nimnet.asn.au>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Ian, Thank you very much for your response! Sorry about the late response, I = have been offline for a few days. I think I may have worked this issue out. I am bringing up a bunch of = Jails today to test my firewall rules in the hopes that I have corrected = my problem. I will reply back either way. Regards, Nathan > On 15 Oct 2015, at 12:51 am, Ian Smith <smithi@nimnet.asn.au> wrote: >=20 > On Tue, 13 Oct 2015 13:50:04 +1000, Nathan Aherne wrote: >> Hi Ian, >>=20 >> Thank you for your response. >>=20 >> I didn=FF=FFt post my ruleset because I should be able to fix the = issue=20 >> myself but I see now that my request to explain =FF=FFhow NAT works=FF=FF= was=20 >> incorrect. >>=20 >> I have now included my ruleset below (as well as my initial email). >=20 > Hi Nathan, >=20 > I was really hoping someone who knows more about stateful rule = handling=20 > (and jail networking) might have a go at this. Oh well I'll try, but=20= > I'm a lousy mindreader, and really don't know which of the below=20 > constitutes 'hairpin NAT'. Perhaps showing your 'netstat -finet -an'=20= > and 'netstat -finet -rn' may shed light on routing? And 'ifconfig'? >=20 >> # Enable NAT >> ipfw nat 1 config ip $jip same_ports log >=20 > I'm assuming that $jip is your WAN IP, AAA.BBB.CCC.DDD .. and that=20 > WWW.XXX.YYY.ZZZ, from your posts in August, is another public IP = routed=20 > to you, and so traffic to it won't be subject to NAT .. correct? But=20= > the WWW... address and all 10.0/16 addresses are jails, not any = separate=20 > boxes you gateway for, right? Just the one external interface, right? >=20 >> 00005 allow ip from any to any via lo0 >> 00006 deny ip from any to not me in via bce0 >> 00100 nat 1 log ip from any to AAA.BBB.CCC.DDD recv bce0 >> 00101 check-state >=20 > Ok, inbound from WAN is nat'd and existing stateful flows followed by=20= > executing the rule that originally kept state. Where this is a = skipto,=20 > skipto will be performed. But where it's a nat rule, I've no idea ..=20= > see below, but you really don't want to add keep-state (again) there. >=20 >> 00110 allow icmp from any to WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ = recv bce0 keep-state >=20 > Hmm. I'd limit this to perhaps icmptypes 0,3,8,11 - though a = stateless=20 > rule would make more sense especially for inbound ICMP. But moving on = .. >=20 >> 00111 allow tcp from any to WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ = dst-port 65222 recv bce0 setup keep-state >=20 > Ok, but showting why plain text works better than HTML on lists :) >=20 >> 00112 allow icmp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any = xmit bce0 keep-state >> 00113 allow tcp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any = dst-port 53,80,443,22,65222 xmit bce0 setup keep-state >> 00114 allow udp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any = dst-port 53,123 xmit bce0 keep-state >=20 > Smells ok. >=20 >> 00120 skipto 65501 log tcp from any to 10.0.0.0/16 recv bce0 setup = keep-state >> 00121 skipto 65501 log udp from any to 10.0.0.0/16 recv bce0 = keep-state >=20 > Whoa, 65501 is your outbound NAT rule, albeit conditionally, and it's=20= > got a problem .. see below. These two are inbound traffic (recv) and = as=20 > is, skipping to 65501 will fall through two outbound rules to be = denied. >=20 > Either allow them here directly, or likely better, skipto a separate > target that then allows (or denies) them, if that's what you intended? >=20 >> 00122 skipto 65501 log tcp from 10.0.0.0/16 to not 10.0.0.0/16 xmit = bce0 setup keep-state >> 00123 skipto 65501 log udp from 10.0.0.0/16 to not 10.0.0.0/16 xmit = bce0 keep-state >=20 > Ok, this traffic does needs to be NAT'd on the way out. >=20 >> 00200 allow log tcp from any to 10.0.0.1 dst-port 22,80,443 in setup = keep-state >> 00200 allow log tcp from 10.0.0.1 to any dst-port 22,80,443 out setup = keep-state >> 00200 allow log udp from 10.0.0.1 to any dst-port 53 out keep-state >=20 > Not clear why these tcp ports are open inbound and outbound? = Presumably=20 > this is jail-to-jail traffic? Perhaps not relevant to your problem. >=20 >> 00201 allow log tcp from any to 10.0.0.2 dst-port 22,80,443 in setup = keep-state >> 00201 allow log tcp from 10.0.0.2 to any dst-port 22,80,443 out setup = keep-state >> 00201 allow log udp from 10.0.0.2 to any dst-port 53 out keep-state >> 65500 deny log ip from any to any >=20 > Ok. >=20 >> 65501 nat 1 log ip from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 = keep-state >=20 > This the target for outbound traffix, xmit bce0, so nat is = appropriate. =20 > Does jail-to-jail traffic travels via lo1? Or what? >=20 > This won't do anything to inbound traffic, but that really shouldn't = get=20 > here except returns as the result of check-state - not from 120 & 121. >=20 > But keep-state is not ok, state was already set on the skipto. I = don't=20 > know how this extra keep-state might behave - does anyone have an = idea? >=20 > Use 'ip4' rather than 'ip' in case this ever sees any ipv6 traffic. >=20 >> 65502 allow log ip from AAA.BBB.CCC.DDD to any xmit bce0 keep-state >=20 > So, only remaining traffic is outbound from the host itself, and = traffic=20 > that is to 10.0/16, but not from AAA... is to be dropped, correct? >=20 > I'm not sure whether 'allow ip .. keep-state' covers tcp, udp, icmp=20 > states .. myself, I'd go for separate rules for each eg tcp, udp, .. = and=20 > I'd do it somewhere else than as a fall through from outbound nat = rule,=20 > it's confusing here, to me anyway .. unless I've missed the reason? >=20 >> 65534 deny log ip from any to any >> 65535 deny ip from any to any >=20 >=20 > Ok, now for your demo of the problem from the later mail, which I've=20= > reformated to quote properly, so: >=20 >> To further illustrate my issue, this is a small log output. >>=20 >> I am running host google.com <http://google.com/>; in the jail, which=20= >> has the IP 10.0.0.1. The UNKNOWN line is logging on the check-state=20= >> rule. >=20 > I see you don't have logging on 101 above now. Probably best. >=20 >> I would expect the first piece of traffic out would be UNKNOWN=20 >> (does not have an entry in the state table) but it seems the=20 >> returning traffic is also showing as UNKNOWN (the second 101). >=20 > I've never logged a check-state, but UNKNOWN may not mean that .. >=20 >> You can see that the traffic is returning on the same port it went=20 >> out on, so its obviously the returning traffic. I am not sure why=20 >> state is not being kept? >=20 > Well perhaps it is .. the return packet is from 8.8.8.8 to 10.0.0.1, = so=20 > it's been correctly NAT'd on the way in. Get rid of that keep-state = on=20 > the nat rule at 65501 and see if not creating double entries in the=20 > state table helps. And change the skipto target on 120 & 121 to only=20= > pass outbound traffic to outbound NAT rule/s. >=20 > Once you've done outbound NAT, probably best just to 'allow [log] = all'? >=20 >> Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 10.0.0.1:57446 = 8.8.8.8:53 out via bce0 >> Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP = 10.0.0.1:57446 8.8.8.8:53 out via bce0 >> Oct 13 15:50:42 host4 kernel: ipfw: 65501 Nat UDP 10.0.0.1:57446 = 8.8.8.8:53 out via bce0 >> Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 8.8.8.8:53 = 10.0.0.1:57446 in via bce0 >> Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 8.8.8.8:53 = 10.0.0.1:57446 in via bce0 >> Oct 13 15:50:42 host4 kernel: ipfw: 65534 Deny UDP 8.8.8.8:53 = 10.0.0.1:57446 in via bce0 >=20 > That said, I can see why this return packet would be denied even if it=20= > were in the nat table: it would execute 'skipto 65501', which nat rule=20= > does not apply, as it's not outbound, and rule 65502 does not apply, = as=20 > it's neither from AAA... nor outbound, so it's then denied by 65534. >=20 > Hope this helps. Please cc me on any response to the list. >=20 > It would be great if someone else might care to lend an oar here; I'm=20= > paddling out of my depth. >=20 > cheers, Ian >=20 > [..]
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9908EC22-344F-4D0B-8930-7D2C70B084A1>