From owner-freebsd-ipfw@freebsd.org Wed Nov 18 00:48:22 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6407CA32A69 for ; Wed, 18 Nov 2015 00:48:22 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from mail.7sq.com.au (mail.7sq.com.au [119.148.74.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A0F1F17A0 for ; Wed, 18 Nov 2015 00:48:21 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id 3947E2C19A2; Wed, 18 Nov 2015 10:36:24 +1000 (AEST) Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id qgsxlj1XWnFf; Wed, 18 Nov 2015 10:36:24 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id F3DBC2C3231; Wed, 18 Nov 2015 10:36:23 +1000 (AEST) X-Virus-Scanned: amavisd-new at mail.7sq.com.au Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id z1Kot2PoRaBf; Wed, 18 Nov 2015 10:36:23 +1000 (AEST) Received: from [192.168.156.153] (reddog2.lnk.telstra.net [110.142.196.96]) by mail.7sq.com.au (Postfix) with ESMTPSA id 7CF582C19A2; Wed, 18 Nov 2015 10:36:23 +1000 (AEST) Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Subject: Re: Kernel NAT issues From: Nathan Aherne In-Reply-To: <9908EC22-344F-4D0B-8930-7D2C70B084A1@reddog.com.au> Date: Wed, 18 Nov 2015 10:40:14 +1000 Cc: freebsd-ipfw@freebsd.org Message-Id: <32DEEFB3-E41F-40CD-8E1A-520FB261C572@reddog.com.au> References: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> <20151013142301.B67283@sola.nimnet.asn.au> <20151014232026.S15983@sola.nimnet.asn.au> <9908EC22-344F-4D0B-8930-7D2C70B084A1@reddog.com.au> To: Ian Smith X-Mailer: Apple Mail (2.2104) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Nov 2015 00:48:22 -0000 Hi Everyone, I think I have worked this out and have a working Stateful IPFW NAT (its = worked fine for a few weeks) rule set. Hopefully this saves someone else = a few weeks of their lives. For some reason hairpin (loopback nat or nat reflection) does not seem = to be working, which is why I chose IPFW in the first place. I am actually getting some really weird log results from IPFW (for inter = jail communication ). IPFW logs show traffic flowing in the opposite = direction to what tshark shows. Tshark shows the traffic in the = direction I would expect. IPFW is blocking the traffic because it is = seeing it as going the wrong way. Anyone have an idea how I can solve = this issue, I would super appreciate any pointers! To clarify inter jail communication - not using the local jail IPs but = DNS - they all resolve to the same public IP. I cannot use split DNS. = ##########################################################################= ##########################################################################= ### #!/bin/sh # Configuration wif=3D"bce0" # WAN interface jif=3D"lo1" # Jail interface jsn=3D"10.0.0.0/16" # Jail subnet plip=3D"10.0.0.1" # Proxy IP hwip=3D=93aaa.bbb.ccc.ddd" # Host WAN IP nwip=3D=93www.xxx.yyy.zzz" # NAT WAN IP # Script variables cmd=3D"ipfw -q add" ks=3D"keep-state" sks=3D"setup keep-state" # Flush all rules ipfw -q -f flush # Enable NAT ipfw nat 1 config ip $nwip log # Allow all loopback traffic $cmd 5 allow ip from any to any via lo0 # Block any traffic not for Host $cmd 6 deny ip from any to not me in via $wif $cmd 100 nat 1 log ip from any to $nwip recv $wif $cmd 101 check-state log # Host $cmd 110 allow icmp from any to $hwip in via $wif $ks $cmd 111 allow tcp from any to $hwip 65222 in via $wif $sks $cmd 112 allow icmp from $hwip to any out via $wif $ks $cmd 113 allow tcp from $hwip to any 53, 80, 443, 22, 65222 out via $wif = $sks $cmd 114 allow udp from $hwip to any 53, 123 out via $wif $ks # Incoming NAT $cmd 120 skipto 65510 log tcp from any to $jsn recv $wif $sks $cmd 121 skipto 65510 log udp from any to $jsn recv $wif $ks # Outgoing NAT $cmd 122 skipto 65510 log tcp from $jsn to not $jsn xmit $wif $sks $cmd 123 skipto 65510 log udp from $jsn to not $jsn xmit $wif $ks # JAILS RULES 200-65000 # Block any other traffic $cmd 65501 deny log ip from any to any # Outgoing NAT $cmd 65510 nat 1 log ip from $jsn to any xmit $wif $cmd 65511 allow log ip from $nwip to any xmit $wif # Block any other traffic #$cmd 65519 deny log ip from any to any # Incoming NAT $cmd 65520 allow log ip from any to $jsn recv $wif # Block any other traffic $cmd 65522 deny log ip from any to any = ##########################################################################= ##########################################################################= ### Regards, Nathan > On 21 Oct 2015, at 9:02 am, Nathan Aherne = wrote: >=20 > Hi Ian, >=20 > Thank you very much for your response! Sorry about the late response, = I have been offline for a few days. >=20 > I think I may have worked this issue out. I am bringing up a bunch of = Jails today to test my firewall rules in the hopes that I have corrected = my problem. I will reply back either way. >=20 > Regards, >=20 > Nathan >=20 >> On 15 Oct 2015, at 12:51 am, Ian Smith wrote: >>=20 >> On Tue, 13 Oct 2015 13:50:04 +1000, Nathan Aherne wrote: >>> Hi Ian, >>>=20 >>> Thank you for your response. >>>=20 >>> I didn=FF=FFt post my ruleset because I should be able to fix the = issue=20 >>> myself but I see now that my request to explain =FF=FFhow NAT = works=FF=FF was=20 >>> incorrect. >>>=20 >>> I have now included my ruleset below (as well as my initial email). >>=20 >> Hi Nathan, >>=20 >> I was really hoping someone who knows more about stateful rule = handling=20 >> (and jail networking) might have a go at this. Oh well I'll try, but=20= >> I'm a lousy mindreader, and really don't know which of the below=20 >> constitutes 'hairpin NAT'. Perhaps showing your 'netstat -finet -an'=20= >> and 'netstat -finet -rn' may shed light on routing? And 'ifconfig'? >>=20 >>> # Enable NAT >>> ipfw nat 1 config ip $jip same_ports log >>=20 >> I'm assuming that $jip is your WAN IP, AAA.BBB.CCC.DDD .. and that=20 >> WWW.XXX.YYY.ZZZ, from your posts in August, is another public IP = routed=20 >> to you, and so traffic to it won't be subject to NAT .. correct? But=20= >> the WWW... address and all 10.0/16 addresses are jails, not any = separate=20 >> boxes you gateway for, right? Just the one external interface, = right? >>=20 >>> 00005 allow ip from any to any via lo0 >>> 00006 deny ip from any to not me in via bce0 >>> 00100 nat 1 log ip from any to AAA.BBB.CCC.DDD recv bce0 >>> 00101 check-state >>=20 >> Ok, inbound from WAN is nat'd and existing stateful flows followed by=20= >> executing the rule that originally kept state. Where this is a = skipto,=20 >> skipto will be performed. But where it's a nat rule, I've no idea ..=20= >> see below, but you really don't want to add keep-state (again) there. >>=20 >>> 00110 allow icmp from any to WWW.XXX.YYY .ZZZ = recv bce0 keep-state >>=20 >> Hmm. I'd limit this to perhaps icmptypes 0,3,8,11 - though a = stateless=20 >> rule would make more sense especially for inbound ICMP. But moving = on .. >>=20 >>> 00111 allow tcp from any to WWW.XXX.YYY .ZZZ = dst-port 65222 recv bce0 setup keep-state >>=20 >> Ok, but showting why plain text works better than HTML on lists :) >>=20 >>> 00112 allow icmp from WWW.XXX.YYY .ZZZ to any = xmit bce0 keep-state >>> 00113 allow tcp from WWW.XXX.YYY .ZZZ to any = dst-port 53,80,443,22,65222 xmit bce0 setup keep-state >>> 00114 allow udp from WWW.XXX.YYY .ZZZ to any = dst-port 53,123 xmit bce0 keep-state >>=20 >> Smells ok. >>=20 >>> 00120 skipto 65501 log tcp from any to 10.0.0.0/16 recv bce0 setup = keep-state >>> 00121 skipto 65501 log udp from any to 10.0.0.0/16 recv bce0 = keep-state >>=20 >> Whoa, 65501 is your outbound NAT rule, albeit conditionally, and it's=20= >> got a problem .. see below. These two are inbound traffic (recv) and = as=20 >> is, skipping to 65501 will fall through two outbound rules to be = denied. >>=20 >> Either allow them here directly, or likely better, skipto a separate >> target that then allows (or denies) them, if that's what you = intended? >>=20 >>> 00122 skipto 65501 log tcp from 10.0.0.0/16 to not 10.0.0.0/16 xmit = bce0 setup keep-state >>> 00123 skipto 65501 log udp from 10.0.0.0/16 to not 10.0.0.0/16 xmit = bce0 keep-state >>=20 >> Ok, this traffic does needs to be NAT'd on the way out. >>=20 >>> 00200 allow log tcp from any to 10.0.0.1 dst-port 22,80,443 in setup = keep-state >>> 00200 allow log tcp from 10.0.0.1 to any dst-port 22,80,443 out = setup keep-state >>> 00200 allow log udp from 10.0.0.1 to any dst-port 53 out keep-state >>=20 >> Not clear why these tcp ports are open inbound and outbound? = Presumably=20 >> this is jail-to-jail traffic? Perhaps not relevant to your problem. >>=20 >>> 00201 allow log tcp from any to 10.0.0.2 dst-port 22,80,443 in setup = keep-state >>> 00201 allow log tcp from 10.0.0.2 to any dst-port 22,80,443 out = setup keep-state >>> 00201 allow log udp from 10.0.0.2 to any dst-port 53 out keep-state >>> 65500 deny log ip from any to any >>=20 >> Ok. >>=20 >>> 65501 nat 1 log ip from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 = keep-state >>=20 >> This the target for outbound traffix, xmit bce0, so nat is = appropriate. =20 >> Does jail-to-jail traffic travels via lo1? Or what? >>=20 >> This won't do anything to inbound traffic, but that really shouldn't = get=20 >> here except returns as the result of check-state - not from 120 & = 121. >>=20 >> But keep-state is not ok, state was already set on the skipto. I = don't=20 >> know how this extra keep-state might behave - does anyone have an = idea? >>=20 >> Use 'ip4' rather than 'ip' in case this ever sees any ipv6 traffic. >>=20 >>> 65502 allow log ip from AAA.BBB.CCC.DDD to any xmit bce0 keep-state >>=20 >> So, only remaining traffic is outbound from the host itself, and = traffic=20 >> that is to 10.0/16, but not from AAA... is to be dropped, correct? >>=20 >> I'm not sure whether 'allow ip .. keep-state' covers tcp, udp, icmp=20= >> states .. myself, I'd go for separate rules for each eg tcp, udp, .. = and=20 >> I'd do it somewhere else than as a fall through from outbound nat = rule,=20 >> it's confusing here, to me anyway .. unless I've missed the reason? >>=20 >>> 65534 deny log ip from any to any >>> 65535 deny ip from any to any >>=20 >>=20 >> Ok, now for your demo of the problem from the later mail, which I've=20= >> reformated to quote properly, so: >>=20 >>> To further illustrate my issue, this is a small log output. >>>=20 >>> I am running host google.com in the jail, which=20= >>> has the IP 10.0.0.1. The UNKNOWN line is logging on the check-state=20= >>> rule. >>=20 >> I see you don't have logging on 101 above now. Probably best. >>=20 >>> I would expect the first piece of traffic out would be UNKNOWN=20 >>> (does not have an entry in the state table) but it seems the=20 >>> returning traffic is also showing as UNKNOWN (the second 101). >>=20 >> I've never logged a check-state, but UNKNOWN may not mean that .. >>=20 >>> You can see that the traffic is returning on the same port it went=20= >>> out on, so its obviously the returning traffic. I am not sure why=20 >>> state is not being kept? >>=20 >> Well perhaps it is .. the return packet is from 8.8.8.8 to 10.0.0.1, = so=20 >> it's been correctly NAT'd on the way in. Get rid of that keep-state = on=20 >> the nat rule at 65501 and see if not creating double entries in the=20= >> state table helps. And change the skipto target on 120 & 121 to only=20= >> pass outbound traffic to outbound NAT rule/s. >>=20 >> Once you've done outbound NAT, probably best just to 'allow [log] = all'? >>=20 >>> Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 10.0.0.1:57446 = 8.8.8.8:53 out via bce0 >>> Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP = 10.0.0.1:57446 8.8.8.8:53 out via bce0 >>> Oct 13 15:50:42 host4 kernel: ipfw: 65501 Nat UDP 10.0.0.1:57446 = 8.8.8.8:53 out via bce0 >>> Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 8.8.8.8:53 = 10.0.0.1:57446 in via bce0 >>> Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 8.8.8.8:53 = 10.0.0.1:57446 in via bce0 >>> Oct 13 15:50:42 host4 kernel: ipfw: 65534 Deny UDP 8.8.8.8:53 = 10.0.0.1:57446 in via bce0 >>=20 >> That said, I can see why this return packet would be denied even if = it=20 >> were in the nat table: it would execute 'skipto 65501', which nat = rule=20 >> does not apply, as it's not outbound, and rule 65502 does not apply, = as=20 >> it's neither from AAA... nor outbound, so it's then denied by 65534. >>=20 >> Hope this helps. Please cc me on any response to the list. >>=20 >> It would be great if someone else might care to lend an oar here; I'm=20= >> paddling out of my depth. >>=20 >> cheers, Ian >>=20 >> [..] >=20 > _______________________________________________ > freebsd-ipfw@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > To unsubscribe, send any mail to = "freebsd-ipfw-unsubscribe@freebsd.org" From owner-freebsd-ipfw@freebsd.org Wed Nov 18 14:17:49 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9EF41A31143 for ; Wed, 18 Nov 2015 14:17:49 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 76CBB1FEB for ; Wed, 18 Nov 2015 14:17:49 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (ppp121-45-231-48.lns20.per1.internode.on.net [121.45.231.48]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id tAIEHYMf033001 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Wed, 18 Nov 2015 06:17:37 -0800 (PST) (envelope-from julian@freebsd.org) Subject: Re: Kernel NAT issues To: Nathan Aherne , Ian Smith References: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> <20151013142301.B67283@sola.nimnet.asn.au> <20151014232026.S15983@sola.nimnet.asn.au> <9908EC22-344F-4D0B-8930-7D2C70B084A1@reddog.com.au> <32DEEFB3-E41F-40CD-8E1A-520FB261C572@reddog.com.au> Cc: freebsd-ipfw@freebsd.org From: Julian Elischer Message-ID: <564C8879.8070307@freebsd.org> Date: Wed, 18 Nov 2015 22:17:29 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <32DEEFB3-E41F-40CD-8E1A-520FB261C572@reddog.com.au> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Nov 2015 14:17:49 -0000 On 11/18/15 8:40 AM, Nathan Aherne wrote: > For some reason hairpin (loopback nat or nat reflection) does not seem to be working, which is why I chose IPFW in the first place. it would be good to see a diagram of what this actually means. From owner-freebsd-ipfw@freebsd.org Wed Nov 18 16:46:34 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5F1A9A32797 for ; Wed, 18 Nov 2015 16:46:34 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CBFFB17AE; Wed, 18 Nov 2015 16:46:33 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id tAIGkOEi037826; Thu, 19 Nov 2015 03:46:25 +1100 (EST) (envelope-from smithi@nimnet.asn.au) Date: Thu, 19 Nov 2015 03:46:24 +1100 (EST) From: Ian Smith To: Julian Elischer cc: Nathan Aherne , freebsd-ipfw@freebsd.org Subject: Re: Kernel NAT issues In-Reply-To: <564C8879.8070307@freebsd.org> Message-ID: <20151119032200.T27669@sola.nimnet.asn.au> References: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> <20151013142301.B67283@sola.nimnet.asn.au> <20151014232026.S15983@sola.nimnet.asn.au> <9908EC22-344F-4D0B-8930-7D2C70B084A1@reddog.com.au> <32DEEFB3-E41F-40CD-8E1A-520FB261C572@reddog.com.au> <564C8879.8070307@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Nov 2015 16:46:34 -0000 On Wed, 18 Nov 2015 22:17:29 +0800, Julian Elischer wrote: > On 11/18/15 8:40 AM, Nathan Aherne wrote: > > For some reason hairpin (loopback nat or nat reflection) does not seem to > > be working, which is why I chose IPFW in the first place. > it would be good to see a diagram of what this actually means. Anything like ? http://kb.juniper.net/InfoCenter/index?page=content&id=KB24639&actp=search Was this so one jail can only access service/s provided by other jail/s, both/all with internal NAT'd addresses, by using only the public address and port of the 'router', which IIRC this is a single system with jails? If so, what sort of routing is setup on both host and jails? (blindfolded, no idea where I've pinned the donkey's tail :) cheers, Ian From owner-freebsd-ipfw@freebsd.org Fri Nov 20 23:12:22 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F0F0AA33416 for ; Fri, 20 Nov 2015 23:12:22 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from mail.7sq.com.au (mail.7sq.com.au [119.148.74.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6FA2A1837; Fri, 20 Nov 2015 23:12:21 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id 7E7BB2C3241; Sat, 21 Nov 2015 09:08:14 +1000 (AEST) Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id kBtjACVFzQUU; Sat, 21 Nov 2015 09:08:14 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id 43CC12C32B8; Sat, 21 Nov 2015 09:08:14 +1000 (AEST) X-Virus-Scanned: amavisd-new at mail.7sq.com.au Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id PMSJdcvxF3_K; Sat, 21 Nov 2015 09:08:14 +1000 (AEST) Received: from [172.20.10.3] (unknown [1.132.70.186]) by mail.7sq.com.au (Postfix) with ESMTPSA id 620ED2C3241; Sat, 21 Nov 2015 09:08:13 +1000 (AEST) Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Subject: Re: Kernel NAT issues From: Nathan Aherne In-Reply-To: <20151119032200.T27669@sola.nimnet.asn.au> Date: Sat, 21 Nov 2015 09:12:12 +1000 Cc: Julian Elischer , freebsd-ipfw@freebsd.org Message-Id: <9D81BDD4-200C-40AB-AB24-B1112881E43A@reddog.com.au> References: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> <20151013142301.B67283@sola.nimnet.asn.au> <20151014232026.S15983@sola.nimnet.asn.au> <9908EC22-344F-4D0B-8930-7D2C70B084A1@reddog.com.au> <32DEEFB3-E41F-40CD-8E1A-520FB261C572@reddog.com.au> <564C8879.8070307@freebsd.org> <20151119032200.T27669@sola.nimnet.asn.au> To: Ian Smith X-Mailer: Apple Mail (2.2104) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Nov 2015 23:12:23 -0000 I am not exactly sure how to draw the setup so it doesn=E2=80=99t = confuse the situation. The setup is extremely simple (I am not running = vimage), jails running on the 10.0.0.0/16 (cloned lo1 interface) network = or with public IPs. The jails with private IPs are the HTTP app jails. = The Host runs a HTTP Proxy (nginx) and forwards traffic to each HTTP App = jail based on the URL it receives. The jails with public IPs are things = like database jails which cannot be proxied by the Host. I can happily communicate with any jail from my laptop (externally) but = when I want one jail to communicate with another jail (for example an = App Jail communicating with the database jail) the traffic shows as = backwards (destination:port -> source:port) in the IPFW logs (tshark = shows the traffic correctly source:port -> destination:port). The jail = to jail traffic tries to go over the lo1 interface (backwards) and is = blocked. Below is some IPFW logs of an App jail (10.0.0.25) = communicating with the database jail (aaa.bbb.ccc.ddd) IPFW logs. The lines labelled UNKNOWN is the check-state rule = (everything is labelled UNKNOWN even if it is KNOWN traffic) Nov 21 08:49:07 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 = 10.0.0.25:42957 out via lo1 Nov 21 08:49:07 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 = 10.0.0.25:42957 out via lo1 Nov 21 08:49:10 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 = 10.0.0.25:42957 out via lo1 Nov 21 08:49:10 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 = 10.0.0.25:42957 out via lo1 Nov 21 08:49:13 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 = 10.0.0.25:42957 out via lo1 Nov 21 08:49:13 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 = 10.0.0.25:42957 out via lo1 Nov 21 08:49:16 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 = 10.0.0.25:42957 out via lo1 Nov 21 08:49:16 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 = 10.0.0.25:42957 out via lo1 tshark output (loopback and wan interface capture for port 5432) Capturing on 'Loopback' and 'bce0' 1 0.000000 10.0.0.25 -> eee.fff.gg.hhh TCP 64 42957=E2=86=925432 = [SYN] Seq=3D0 Win=3D65535 Len=3D0 MSS=3D16344 WS=3D64 SACK_PERM=3D1 = TSval=3D142885525 TSecr=3D0 2 3.013905 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142888539 TSecr=3D0 3 6.241658 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142891767 TSecr=3D0 4 9.451516 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142894976 TSecr=3D0 5 12.654656 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142898180 TSecr=3D0 6 15.863900 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142901389 TSecr=3D0 7 22.076655 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142907602 TSecr=3D0 > If so, what sort of routing is setup on both host and jails? Routing is what would be added by default (whatever the host system adds = when adding an IP), there is no custom routing. I have wondered if I = need to modify the routing table to get this to work.=20 Below is the output of netstat -rn www.xxx.yy.zzz is the gateway address eee.fff.gg.hhh is the database jail public IP aaa.bbb.cc.ddd is the public IP for NAT lll.mmm.nn.ooo is the Hosts public IP Routing tables Internet: Destination Gateway Flags Netif Expire default www.xxx.yy.zzz UGS bce0 10.0.0.1 link#6 UH lo1 10.0.0.2 link#6 UH lo1 10.0.0.3 link#6 UH lo1 10.0.0.4 link#6 UH lo1 10.0.0.5 link#6 UH lo1 10.0.0.6 link#6 UH lo1 10.0.0.7 link#6 UH lo1 10.0.0.8 link#6 UH lo1 10.0.0.9 link#6 UH lo1 10.0.0.10 link#6 UH lo1 10.0.0.11 link#6 UH lo1 10.0.0.12 link#6 UH lo1 10.0.0.13 link#6 UH lo1 10.0.0.14 link#6 UH lo1 10.0.0.15 link#6 UH lo1 10.0.0.16 link#6 UH lo1 10.0.0.17 link#6 UH lo1 10.0.0.18 link#6 UH lo1 10.0.0.19 link#6 UH lo1 10.0.0.20 link#6 UH lo1 10.0.0.21 link#6 UH lo1 10.0.0.22 link#6 UH lo1 10.0.0.23 link#6 UH lo1 10.0.0.24 link#6 UH lo1 10.0.0.25 link#6 UH lo1 10.0.0.26 link#6 UH lo1 www.xxx.yy.zzz/25 link#1 U bce0 eee.fff.gg.hhh link#1 UHS lo0 eee.fff.gg.hhh/32 link#1 U bce0 aaa.bbb.cc.ddd link#1 UHS lo0 aaa.bbb.cc.ddd/32 link#1 U bce0 lll.mmm.nn.ooo link#1 UHS lo0 127.0.0.1 link#5 UH lo0 Internet6: Destination Gateway Flags = Netif Expire ::/96 ::1 UGRS = lo0 ::1 link#5 UH = lo0 ::ffff:0.0.0.0/96 ::1 UGRS = lo0 fe80::/10 ::1 UGRS = lo0 fe80::%lo0/64 link#5 U = lo0 fe80::1%lo0 link#5 UHS = lo0 ff01::%lo0/32 ::1 U = lo0 ff02::/16 ::1 UGRS = lo0 ff02::%lo0/32 ::1 U = lo0 > Anything like ? > = http://kb.juniper.net/InfoCenter/index?page=3Dcontent&id=3DKB24639&actp=3D= search = Yes just like that. Regards, Nathan > On 19 Nov 2015, at 2:46 am, Ian Smith wrote: >=20 > On Wed, 18 Nov 2015 22:17:29 +0800, Julian Elischer wrote: >> On 11/18/15 8:40 AM, Nathan Aherne wrote: >>> For some reason hairpin (loopback nat or nat reflection) does not = seem to >>> be working, which is why I chose IPFW in the first place. >=20 >> it would be good to see a diagram of what this actually means. >=20 > Anything like ? > = http://kb.juniper.net/InfoCenter/index?page=3Dcontent&id=3DKB24639&actp=3D= search >=20 > Was this so one jail can only access service/s provided by other = jail/s,=20 > both/all with internal NAT'd addresses, by using only the public = address=20 > and port of the 'router', which IIRC this is a single system with = jails? >=20 > If so, what sort of routing is setup on both host and jails? >=20 > (blindfolded, no idea where I've pinned the donkey's tail :) >=20 > cheers, Ian From owner-freebsd-ipfw@freebsd.org Sat Nov 21 02:06:05 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7159EA32A17 for ; Sat, 21 Nov 2015 02:06:05 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from mail.7sq.com.au (mail.7sq.com.au [119.148.74.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E260B1606; Sat, 21 Nov 2015 02:06:04 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id 2DCCF2C3306; Sat, 21 Nov 2015 12:02:02 +1000 (AEST) Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id PGnIceYe0nKO; Sat, 21 Nov 2015 12:02:01 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id DEE922C32F7; Sat, 21 Nov 2015 12:02:01 +1000 (AEST) X-Virus-Scanned: amavisd-new at mail.7sq.com.au Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id DQZV5QxVrZ6v; Sat, 21 Nov 2015 12:02:01 +1000 (AEST) Received: from [172.20.10.3] (unknown [1.132.78.127]) by mail.7sq.com.au (Postfix) with ESMTPSA id 12BFC2C3308; Sat, 21 Nov 2015 12:02:01 +1000 (AEST) Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Subject: Re: Kernel NAT issues From: Nathan Aherne In-Reply-To: <9D81BDD4-200C-40AB-AB24-B1112881E43A@reddog.com.au> Date: Sat, 21 Nov 2015 12:06:01 +1000 Cc: Julian Elischer , freebsd-ipfw@freebsd.org Message-Id: <3BF360A8-35E6-4043-8AFF-87D983F29C66@reddog.com.au> References: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> <20151013142301.B67283@sola.nimnet.asn.au> <20151014232026.S15983@sola.nimnet.asn.au> <9908EC22-344F-4D0B-8930-7D2C70B084A1@reddog.com.au> <32DEEFB3-E41F-40CD-8E1A-520FB261C572@reddog.com.au> <564C8879.8070307@freebsd.org> <20151119032200.T27669@sola.nimnet.asn.au> <9D81BDD4-200C-40AB-AB24-B1112881E43A@reddog.com.au> To: Ian Smith X-Mailer: Apple Mail (2.2104) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Nov 2015 02:06:05 -0000 I had a bit of a think about how to describe what I am trying to = achieve. I am treating each jail likes its own little "virtual machine=E2=80=9D. = The jail provides certain services, using things like nginx or nodejs, = php-fpm, mysql or postgresql. The jails can control connections to = themselves by configuring the firewall ports that are opened on the IP = their IP (10.0.0.0/16 or a public IP). I know the jails have no = firewall of their own, the firewall is configured from the host. I want each jail or =E2=80=9Cvirtual machine=E2=80=9D to be able to = communicate with one another and the wider internet. When a jail does a = DNS query for another App jail, it may get a public IP on its own Host = (or it may get another host) and it has no issues being able to = communicate with another jail on the same host. At the moment all of the above is working perfectly except for jail to = jail communication on the same host (when the communication is not = directly between 10.0.0.0/16 IP addresses). Regards, Nathan > On 21 Nov 2015, at 9:12 am, Nathan Aherne = wrote: >=20 > I am not exactly sure how to draw the setup so it doesn=E2=80=99t = confuse the situation. The setup is extremely simple (I am not running = vimage), jails running on the 10.0.0.0/16 (cloned lo1 interface) network = or with public IPs. The jails with private IPs are the HTTP app jails. = The Host runs a HTTP Proxy (nginx) and forwards traffic to each HTTP App = jail based on the URL it receives. The jails with public IPs are things = like database jails which cannot be proxied by the Host. >=20 > I can happily communicate with any jail from my laptop (externally) = but when I want one jail to communicate with another jail (for example = an App Jail communicating with the database jail) the traffic shows as = backwards (destination:port -> source:port) in the IPFW logs (tshark = shows the traffic correctly source:port -> destination:port). The jail = to jail traffic tries to go over the lo1 interface (backwards) and is = blocked. Below is some IPFW logs of an App jail (10.0.0.25) = communicating with the database jail (aaa.bbb.ccc.ddd) >=20 > IPFW logs. The lines labelled UNKNOWN is the check-state rule = (everything is labelled UNKNOWN even if it is KNOWN traffic) >=20 > Nov 21 08:49:07 host5 kernel: ipfw: 101 UNKNOWN TCP = eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 > Nov 21 08:49:07 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 = 10.0.0.25:42957 out via lo1 > Nov 21 08:49:10 host5 kernel: ipfw: 101 UNKNOWN TCP = eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 > Nov 21 08:49:10 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 = 10.0.0.25:42957 out via lo1 > Nov 21 08:49:13 host5 kernel: ipfw: 101 UNKNOWN TCP = eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 > Nov 21 08:49:13 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 = 10.0.0.25:42957 out via lo1 > Nov 21 08:49:16 host5 kernel: ipfw: 101 UNKNOWN TCP = eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 > Nov 21 08:49:16 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 = 10.0.0.25:42957 out via lo1 >=20 > tshark output (loopback and wan interface capture for port 5432) >=20 > Capturing on 'Loopback' and 'bce0' > 1 0.000000 10.0.0.25 -> eee.fff.gg.hhh TCP 64 42957=E2=86=925432 = [SYN] Seq=3D0 Win=3D65535 Len=3D0 MSS=3D16344 WS=3D64 SACK_PERM=3D1 = TSval=3D142885525 TSecr=3D0 > 2 3.013905 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142888539 TSecr=3D0 > 3 6.241658 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142891767 TSecr=3D0 > 4 9.451516 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142894976 TSecr=3D0 > 5 12.654656 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142898180 TSecr=3D0 > 6 15.863900 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142901389 TSecr=3D0 > 7 22.076655 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142907602 TSecr=3D0 >=20 >=20 >> If so, what sort of routing is setup on both host and jails? >=20 > Routing is what would be added by default (whatever the host system = adds when adding an IP), there is no custom routing. I have wondered if = I need to modify the routing table to get this to work.=20 >=20 > Below is the output of netstat -rn >=20 > www.xxx.yy .zzz is the gateway address > eee.fff.gg.hhh is the database jail public IP > aaa.bbb.cc.ddd is the public IP for NAT > lll.mmm.nn.ooo is the Hosts public IP >=20 >=20 > Routing tables >=20 > Internet: > Destination Gateway Flags Netif Expire > default www.xxx.yy .zzz UGS = bce0 > 10.0.0.1 link#6 UH lo1 > 10.0.0.2 link#6 UH lo1 > 10.0.0.3 link#6 UH lo1 > 10.0.0.4 link#6 UH lo1 > 10.0.0.5 link#6 UH lo1 > 10.0.0.6 link#6 UH lo1 > 10.0.0.7 link#6 UH lo1 > 10.0.0.8 link#6 UH lo1 > 10.0.0.9 link#6 UH lo1 > 10.0.0.10 link#6 UH lo1 > 10.0.0.11 link#6 UH lo1 > 10.0.0.12 link#6 UH lo1 > 10.0.0.13 link#6 UH lo1 > 10.0.0.14 link#6 UH lo1 > 10.0.0.15 link#6 UH lo1 > 10.0.0.16 link#6 UH lo1 > 10.0.0.17 link#6 UH lo1 > 10.0.0.18 link#6 UH lo1 > 10.0.0.19 link#6 UH lo1 > 10.0.0.20 link#6 UH lo1 > 10.0.0.21 link#6 UH lo1 > 10.0.0.22 link#6 UH lo1 > 10.0.0.23 link#6 UH lo1 > 10.0.0.24 link#6 UH lo1 > 10.0.0.25 link#6 UH lo1 > 10.0.0.26 link#6 UH lo1 > www.xxx.yy.zzz/25 link#1 U = bce0 > eee.fff.gg.hhh link#1 UHS lo0 > eee.fff.gg.hhh/32 link#1 U bce0 > aaa.bbb.cc .ddd link#1 UHS = lo0 > aaa.bbb.cc.ddd/32 link#1 U bce0 > lll.mmm.nn.ooo link#1 UHS lo0 > 127.0.0.1 link#5 UH lo0 >=20 > Internet6: > Destination Gateway Flags = Netif Expire > ::/96 ::1 UGRS = lo0 > ::1 link#5 UH = lo0 > ::ffff:0.0.0.0/96 ::1 UGRS = lo0 > fe80::/10 ::1 UGRS = lo0 > fe80::%lo0/64 link#5 U = lo0 > fe80::1%lo0 link#5 UHS = lo0 > ff01::%lo0/32 ::1 U = lo0 > ff02::/16 ::1 UGRS = lo0 > ff02::%lo0/32 ::1 U = lo0 >=20 >> Anything like ? >> = http://kb.juniper.net/InfoCenter/index?page=3Dcontent&id=3DKB24639&actp=3D= search = > Yes just like that. >=20 > Regards, >=20 > Nathan >=20 >> On 19 Nov 2015, at 2:46 am, Ian Smith > wrote: >>=20 >> On Wed, 18 Nov 2015 22:17:29 +0800, Julian Elischer wrote: >>> On 11/18/15 8:40 AM, Nathan Aherne wrote: >>>> For some reason hairpin (loopback nat or nat reflection) does not = seem to >>>> be working, which is why I chose IPFW in the first place. >>=20 >>> it would be good to see a diagram of what this actually means. >>=20 >> Anything like ? >> = http://kb.juniper.net/InfoCenter/index?page=3Dcontent&id=3DKB24639&actp=3D= search = >>=20 >> Was this so one jail can only access service/s provided by other = jail/s,=20 >> both/all with internal NAT'd addresses, by using only the public = address=20 >> and port of the 'router', which IIRC this is a single system with = jails? >>=20 >> If so, what sort of routing is setup on both host and jails? >>=20 >> (blindfolded, no idea where I've pinned the donkey's tail :) >>=20 >> cheers, Ian >=20 From owner-freebsd-ipfw@freebsd.org Sat Nov 21 14:08:50 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 61C8CA33D75 for ; Sat, 21 Nov 2015 14:08:50 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2CEC21220 for ; Sat, 21 Nov 2015 14:08:47 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (50-196-156-133-static.hfc.comcastbusiness.net [50.196.156.133]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id tALE8Wvk053874 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Sat, 21 Nov 2015 06:08:36 -0800 (PST) (envelope-from julian@freebsd.org) Subject: Re: Kernel NAT issues To: Nathan Aherne References: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> <20151013142301.B67283@sola.nimnet.asn.au> <20151014232026.S15983@sola.nimnet.asn.au> <9908EC22-344F-4D0B-8930-7D2C70B084A1@reddog.com.au> <32DEEFB3-E41F-40CD-8E1A-520FB261C572@reddog.com.au> <564C8879.8070307@freebsd.org> <20151119032200.T27669@sola.nimnet.asn.au> <9D81BDD4-200C-40AB-AB24-B1112881E43A@reddog.com.au> <3BF360A8-35E6-4043-8AFF-87D983F29C66@reddog.com.au> Cc: "freebsd-ipfw@freebsd.org" From: Julian Elischer Message-ID: <56507ADA.8030202@freebsd.org> Date: Sat, 21 Nov 2015 22:08:26 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <3BF360A8-35E6-4043-8AFF-87D983F29C66@reddog.com.au> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Nov 2015 14:08:50 -0000 I note, no answers yet.. I plan on spending some time tomorrow trying to understand this issue. just letting you know.. On 21/11/2015 10:06 AM, Nathan Aherne wrote: > I had a bit of a think about how to describe what I am trying to > achieve. > > I am treating each jail likes its own little "virtual machine”. The > jail provides certain services, using things like nginx or nodejs, > php-fpm, mysql or postgresql. The jails can control connections to > themselves by configuring the firewall ports that are opened on the > IP their IP (10.0.0.0/16 or a public IP). I know the jails have no > firewall of their own, the firewall is configured from the host. > > I want each jail or “virtual machine” to be able to communicate with > one another and the wider internet. When a jail does a DNS query for > another App jail, it may get a public IP on its own Host (or it may > get another host) and it has no issues being able to communicate > with another jail on the same host. > > At the moment all of the above is working perfectly except for jail > to jail communication on the same host (when the communication is > not directly between 10.0.0.0/16 IP addresses). > > Regards, > > Nathan > >> On 21 Nov 2015, at 9:12 am, Nathan Aherne > > wrote: >> >> I am not exactly sure how to draw the setup so it doesn’t confuse >> the situation. The setup is extremely simple (I am not running >> vimage), jails running on the 10.0.0.0/16 (cloned lo1 interface) >> network or with public IPs. The jails with private IPs are the HTTP >> app jails. The Host runs a HTTP Proxy (nginx) and forwards traffic >> to each HTTP App jail based on the URL it receives. The jails with >> public IPs are things like database jails which cannot be proxied >> by the Host. >> >> I can happily communicate with any jail from my laptop (externally) >> but when I want one jail to communicate with another jail (for >> example an App Jail communicating with the database jail) the >> traffic shows as backwards (destination:port -> source:port) in the >> IPFW logs (tshark shows the traffic correctly source:port -> >> destination:port). The jail to jail traffic tries to go over the >> lo1 interface (backwards) and is blocked. Below is some IPFW logs >> of an App jail (10.0.0.25) communicating with the database jail >> (aaa.bbb.ccc.ddd) >> >> IPFW logs. The lines labelled UNKNOWN is the check-state rule >> (everything is labelled UNKNOWN even if it is KNOWN traffic) >> >> Nov 21 08:49:07 host5 kernel: ipfw: 101 UNKNOWN TCP >> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> Nov 21 08:49:07 host5 kernel: ipfw: 65501 Deny TCP >> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> Nov 21 08:49:10 host5 kernel: ipfw: 101 UNKNOWN TCP >> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> Nov 21 08:49:10 host5 kernel: ipfw: 65501 Deny TCP >> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> Nov 21 08:49:13 host5 kernel: ipfw: 101 UNKNOWN TCP >> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> Nov 21 08:49:13 host5 kernel: ipfw: 65501 Deny TCP >> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> Nov 21 08:49:16 host5 kernel: ipfw: 101 UNKNOWN TCP >> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> Nov 21 08:49:16 host5 kernel: ipfw: 65501 Deny TCP >> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> >> tshark output (loopback and wan interface capture for port 5432) >> >> Capturing on 'Loopback' and 'bce0' >> 1 0.000000 10.0.0.25 -> eee.fff.gg.hhh TCP 64 42957→5432 [SYN] >> Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142885525 >> TSecr=0 >> 2 3.013905 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP >> Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 >> WS=64 SACK_PERM=1 TSval=142888539 TSecr=0 >> 3 6.241658 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP >> Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 >> WS=64 SACK_PERM=1 TSval=142891767 TSecr=0 >> 4 9.451516 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP >> Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 >> WS=64 SACK_PERM=1 TSval=142894976 TSecr=0 >> 5 12.654656 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP >> Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 >> WS=64 SACK_PERM=1 TSval=142898180 TSecr=0 >> 6 15.863900 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP >> Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 >> WS=64 SACK_PERM=1 TSval=142901389 TSecr=0 >> 7 22.076655 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP >> Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 >> WS=64 SACK_PERM=1 TSval=142907602 TSecr=0 >> >> >>> If so, what sort of routing is setup on both host and jails? >> >> Routing is what would be added by default (whatever the host system >> adds when adding an IP), there is no custom routing. I have >> wondered if I need to modify the routing table to get this to work. >> >> Below is the output of netstat -rn >> >> www.xxx.yy .zzz is the gateway address >> eee.fff.gg.hhh is the database jail public IP >> aaa.bbb.cc.ddd is the public IP for NAT >> lll.mmm.nn.ooo is the Hosts public IP >> >> >> Routing tables >> >> Internet: >> Destination Gateway Flags Netif Expire >> default www.xxx.yy .zzz UGS bce0 >> 10.0.0.1 link#6 UH lo1 >> 10.0.0.2 link#6 UH lo1 >> 10.0.0.3 link#6 UH lo1 >> 10.0.0.4 link#6 UH lo1 >> 10.0.0.5 link#6 UH lo1 >> 10.0.0.6 link#6 UH lo1 >> 10.0.0.7 link#6 UH lo1 >> 10.0.0.8 link#6 UH lo1 >> 10.0.0.9 link#6 UH lo1 >> 10.0.0.10 link#6 UH lo1 >> 10.0.0.11 link#6 UH lo1 >> 10.0.0.12 link#6 UH lo1 >> 10.0.0.13 link#6 UH lo1 >> 10.0.0.14 link#6 UH lo1 >> 10.0.0.15 link#6 UH lo1 >> 10.0.0.16 link#6 UH lo1 >> 10.0.0.17 link#6 UH lo1 >> 10.0.0.18 link#6 UH lo1 >> 10.0.0.19 link#6 UH lo1 >> 10.0.0.20 link#6 UH lo1 >> 10.0.0.21 link#6 UH lo1 >> 10.0.0.22 link#6 UH lo1 >> 10.0.0.23 link#6 UH lo1 >> 10.0.0.24 link#6 UH lo1 >> 10.0.0.25 link#6 UH lo1 >> 10.0.0.26 link#6 UH lo1 >> www.xxx.yy.zzz/25 link#1 U >> bce0 >> eee.fff.gg.hhh link#1 UHS lo0 >> eee.fff.gg.hhh/32 link#1 U bce0 >> aaa.bbb.cc .ddd link#1 UHS lo0 >> aaa.bbb.cc.ddd/32 link#1 U bce0 >> lll.mmm.nn.ooo link#1 UHS lo0 >> 127.0.0.1 link#5 UH lo0 >> >> Internet6: >> Destination Gateway Flags >> Netif Expire >> ::/96 ::1 UGRS lo0 >> ::1 link#5 UH lo0 >> ::ffff:0.0.0.0/96 ::1 UGRS >> lo0 >> fe80::/10 ::1 UGRS lo0 >> fe80::%lo0/64 link#5 U lo0 >> fe80::1%lo0 link#5 UHS lo0 >> ff01::%lo0/32 ::1 U lo0 >> ff02::/16 ::1 UGRS lo0 >> ff02::%lo0/32 ::1 U lo0 >> >>> Anything like ? >>> http://kb.juniper.net/InfoCenter/index?page=content&id=KB24639&actp=search >> >> Yes just like that. >> >> Regards, >> >> Nathan >> >>> On 19 Nov 2015, at 2:46 am, Ian Smith >> > wrote: >>> >>> On Wed, 18 Nov 2015 22:17:29 +0800, Julian Elischer wrote: >>>> On 11/18/15 8:40 AM, Nathan Aherne wrote: >>>>> For some reason hairpin (loopback nat or nat reflection) does >>>>> not seem to >>>>> be working, which is why I chose IPFW in the first place. >>> >>>> it would be good to see a diagram of what this actually means. >>> >>> Anything like ? >>> http://kb.juniper.net/InfoCenter/index?page=content&id=KB24639&actp=search >>> >>> Was this so one jail can only access service/s provided by other >>> jail/s, >>> both/all with internal NAT'd addresses, by using only the public >>> address >>> and port of the 'router', which IIRC this is a single system with >>> jails? >>> >>> If so, what sort of routing is setup on both host and jails? >>> >>> (blindfolded, no idea where I've pinned the donkey's tail :) >>> >>> cheers, Ian >> >