From owner-freebsd-ipfw@freebsd.org Mon Nov 23 07:02:21 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 36A54A35B3F for ; Mon, 23 Nov 2015 07:02:21 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 06337123C for ; Mon, 23 Nov 2015 07:02:20 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (50-196-156-133-static.hfc.comcastbusiness.net [50.196.156.133]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id tAN729dT060459 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Sun, 22 Nov 2015 23:02:14 -0800 (PST) (envelope-from julian@freebsd.org) Subject: Re: Kernel NAT issues To: Nathan Aherne , Ian Smith References: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> <20151013142301.B67283@sola.nimnet.asn.au> <20151014232026.S15983@sola.nimnet.asn.au> <9908EC22-344F-4D0B-8930-7D2C70B084A1@reddog.com.au> <32DEEFB3-E41F-40CD-8E1A-520FB261C572@reddog.com.au> <564C8879.8070307@freebsd.org> <20151119032200.T27669@sola.nimnet.asn.au> <9D81BDD4-200C-40AB-AB24-B1112881E43A@reddog.com.au> <3BF360A8-35E6-4043-8AFF-87D983F29C66@reddog.com.au> Cc: freebsd-ipfw@freebsd.org From: Julian Elischer Message-ID: <5652B9EB.10805@freebsd.org> Date: Mon, 23 Nov 2015 15:02:03 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <3BF360A8-35E6-4043-8AFF-87D983F29C66@reddog.com.au> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Nov 2015 07:02:21 -0000 On 21/11/2015 10:06 AM, Nathan Aherne wrote: > I had a bit of a think about how to describe what I am trying to achieve. > > I am treating each jail likes its own little "virtual machine”. The jail provides certain services, using things like nginx or nodejs, php-fpm, mysql or postgresql. The jails can control connections to themselves by configuring the firewall ports that are opened on the IP their IP (10.0.0.0/16 or a public IP). I know the jails have no firewall of their own, the firewall is configured from the host. > > I want each jail or “virtual machine” to be able to communicate with one another and the wider internet. When a jail does a DNS query for another App jail, it may get a public IP on its own Host (or it may get another host) and it has no issues being able to communicate with another jail on the same host. > > At the moment all of the above is working perfectly except for jail to jail communication on the same host (when the communication is not directly between 10.0.0.0/16 IP addresses). this is pretty much exactly when vimage/vnet jails could be used to great affect. Is there a reason you are not doing that? Each jail has it's own routing tables, addresses and (virtual) interfaces. here's how I'd do it with vimage +--------------+ +---------------+ | servers | +--------------+ | | +--------------+ | +--------+ | | | +--------------+ | | +--------+ +--+------+----+ | iface | | bridge | | +-----+ | +--------+ +----+---------+ | | | | | | +------------------------+---------------------+ | | | | | NAT jail router | | | | | +-------+--------+--------+-------+------------+ | | | | +--+--+ +--+--+ +--+--+ +--+--+ | | | | | | | | | | | | | | | | | | | | | | | | jails | | | | | | | | +-----+ +-----+ +-----+ +-----+ however the hairpin idea might still be useful even in that scenario if they don't know about each other's 'local' addresses, but do NAT'd machines need to talk to each other by externeal addresses? i Nathan >> On 21 Nov 2015, at 9:12 am, Nathan Aherne wrote: >> >> I am not exactly sure how to draw the setup so it doesn’t confuse the situation. The setup is extremely simple (I am not running vimage), jails running on the 10.0.0.0/16 (cloned lo1 interface) network or with public IPs. The jails with private IPs are the HTTP app jails. The Host runs a HTTP Proxy (nginx) and forwards traffic to each HTTP App jail based on the URL it receives. The jails with public IPs are things like database jails which cannot be proxied by the Host. >> >> I can happily communicate with any jail from my laptop (externally) but when I want one jail to communicate with another jail (for example an App Jail communicating with the database jail) the traffic shows as backwards (destination:port -> source:port) in the IPFW logs (tshark shows the traffic correctly source:port -> destination:port). The jail to jail traffic tries to go over the lo1 interface (backwards) and is blocked. Below is some IPFW logs of an App jail (10.0.0.25) communicating with the database jail (aaa.bbb.ccc.ddd) >> >> IPFW logs. The lines labelled UNKNOWN is the check-state rule (everything is labelled UNKNOWN even if it is KNOWN traffic) >> >> Nov 21 08:49:07 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> Nov 21 08:49:07 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> Nov 21 08:49:10 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> Nov 21 08:49:10 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> Nov 21 08:49:13 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> Nov 21 08:49:13 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> Nov 21 08:49:16 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> Nov 21 08:49:16 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> >> tshark output (loopback and wan interface capture for port 5432) >> >> Capturing on 'Loopback' and 'bce0' >> 1 0.000000 10.0.0.25 -> eee.fff.gg.hhh TCP 64 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142885525 TSecr=0 >> 2 3.013905 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142888539 TSecr=0 >> 3 6.241658 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142891767 TSecr=0 >> 4 9.451516 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142894976 TSecr=0 >> 5 12.654656 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142898180 TSecr=0 >> 6 15.863900 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142901389 TSecr=0 >> 7 22.076655 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142907602 TSecr=0 >> >> >>> If so, what sort of routing is setup on both host and jails? >> Routing is what would be added by default (whatever the host system adds when adding an IP), there is no custom routing. I have wondered if I need to modify the routing table to get this to work. >> >> Below is the output of netstat -rn >> >> www.xxx.yy .zzz is the gateway address >> eee.fff.gg.hhh is the database jail public IP >> aaa.bbb.cc.ddd is the public IP for NAT >> lll.mmm.nn.ooo is the Hosts public IP >> >> >> Routing tables >> >> Internet: >> Destination Gateway Flags Netif Expire >> default www.xxx.yy .zzz UGS bce0 >> 10.0.0.1 link#6 UH lo1 >> 10.0.0.2 link#6 UH lo1 >> 10.0.0.3 link#6 UH lo1 >> 10.0.0.4 link#6 UH lo1 >> 10.0.0.5 link#6 UH lo1 >> 10.0.0.6 link#6 UH lo1 >> 10.0.0.7 link#6 UH lo1 >> 10.0.0.8 link#6 UH lo1 >> 10.0.0.9 link#6 UH lo1 >> 10.0.0.10 link#6 UH lo1 >> 10.0.0.11 link#6 UH lo1 >> 10.0.0.12 link#6 UH lo1 >> 10.0.0.13 link#6 UH lo1 >> 10.0.0.14 link#6 UH lo1 >> 10.0.0.15 link#6 UH lo1 >> 10.0.0.16 link#6 UH lo1 >> 10.0.0.17 link#6 UH lo1 >> 10.0.0.18 link#6 UH lo1 >> 10.0.0.19 link#6 UH lo1 >> 10.0.0.20 link#6 UH lo1 >> 10.0.0.21 link#6 UH lo1 >> 10.0.0.22 link#6 UH lo1 >> 10.0.0.23 link#6 UH lo1 >> 10.0.0.24 link#6 UH lo1 >> 10.0.0.25 link#6 UH lo1 >> 10.0.0.26 link#6 UH lo1 >> www.xxx.yy.zzz/25 link#1 U bce0 >> eee.fff.gg.hhh link#1 UHS lo0 >> eee.fff.gg.hhh/32 link#1 U bce0 >> aaa.bbb.cc .ddd link#1 UHS lo0 >> aaa.bbb.cc.ddd/32 link#1 U bce0 >> lll.mmm.nn.ooo link#1 UHS lo0 >> 127.0.0.1 link#5 UH lo0 >> >> Internet6: >> Destination Gateway Flags Netif Expire >> ::/96 ::1 UGRS lo0 >> ::1 link#5 UH lo0 >> ::ffff:0.0.0.0/96 ::1 UGRS lo0 >> fe80::/10 ::1 UGRS lo0 >> fe80::%lo0/64 link#5 U lo0 >> fe80::1%lo0 link#5 UHS lo0 >> ff01::%lo0/32 ::1 U lo0 >> ff02::/16 ::1 UGRS lo0 >> ff02::%lo0/32 ::1 U lo0 >> >>> Anything like ? >>> http://kb.juniper.net/InfoCenter/index?page=content&id=KB24639&actp=search >> Yes just like that. >> >> Regards, >> >> Nathan >> >>> On 19 Nov 2015, at 2:46 am, Ian Smith > wrote: >>> >>> On Wed, 18 Nov 2015 22:17:29 +0800, Julian Elischer wrote: >>>> On 11/18/15 8:40 AM, Nathan Aherne wrote: >>>>> For some reason hairpin (loopback nat or nat reflection) does not seem to >>>>> be working, which is why I chose IPFW in the first place. >>>> it would be good to see a diagram of what this actually means. >>> Anything like ? >>> http://kb.juniper.net/InfoCenter/index?page=content&id=KB24639&actp=search >>> >>> Was this so one jail can only access service/s provided by other jail/s, >>> both/all with internal NAT'd addresses, by using only the public address >>> and port of the 'router', which IIRC this is a single system with jails? >>> >>> If so, what sort of routing is setup on both host and jails? >>> >>> (blindfolded, no idea where I've pinned the donkey's tail :) >>> >>> cheers, Ian > _______________________________________________ > freebsd-ipfw@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org" > > From owner-freebsd-ipfw@freebsd.org Wed Nov 25 05:00:22 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3E3F3A379AC for ; Wed, 25 Nov 2015 05:00:22 +0000 (UTC) (envelope-from graham@menhennitt.com.au) Received: from homiemail-a92.g.dreamhost.com (sub5.mail.dreamhost.com [208.113.200.129]) by mx1.freebsd.org (Postfix) with ESMTP id 1DAE81213 for ; Wed, 25 Nov 2015 05:00:21 +0000 (UTC) (envelope-from graham@menhennitt.com.au) Received: from homiemail-a92.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a92.g.dreamhost.com (Postfix) with ESMTP id E7ED93DC05E for ; Tue, 24 Nov 2015 21:00:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=menhennitt.com.au; h=to :from:subject:message-id:date:mime-version:content-type: content-transfer-encoding; s=menhennitt.com.au; bh=HbZ7fjFbwgktn EOmVjRDaRf4uIg=; b=e51VxR21yeXpw9Sm8T1iL9+zuOQ2pHic8N6MalJwFBLxU 1t5qJlWGXlRmfGVXEe0lhI4VqU6XHom34bXmVpQ86GamfXeZo0UqseIdk+lFGy/C vEpuDJ63WCvZDh1ZuqjbIxqI9TNQifYNXghVRB6Spo1x3ybJ61vK7jenBa9S2o= Received: from [203.2.73.68] (c122-107-214-88.mckinn3.vic.optusnet.com.au [122.107.214.88]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: graham@menhennitt.com.au) by homiemail-a92.g.dreamhost.com (Postfix) with ESMTPSA id 65DF53DC05B for ; Tue, 24 Nov 2015 21:00:14 -0800 (PST) To: freebsd-ipfw@freebsd.org From: Graham Menhennitt Subject: connecting a PS4 via IPFW X-Enigmail-Draft-Status: N1110 Message-ID: <5655405C.1060301@menhennitt.com.au> Date: Wed, 25 Nov 2015 16:00:12 +1100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Nov 2015 05:00:22 -0000 Hello IPFWers, I have a box running FreeBSD 10-stable that I use as a router/firewall/NAT. It runs IPFW and uses kernel NAT. My son is nagging me about playing multi-player online games on his Sony PS4. >From what I've read, I could enable UPnP. But I've tried compiling the net/miniupnpd port but it won't build for IPFW (and I don't want to convert to PF). Giving up on that, I'm now trying to enable port forwarding - apparently, this will fix it. I've allocated the PS4 a static IP address on my LAN. I need to port forward TCP ports 80, 443, 1935, 3478-3480, and UDP ports 3478-3479. I've tried the following command: ipfw nat 1 config \ redirect_port tcp ${PS4_LAN_ADDRESS}:1935 80 \ redirect_port tcp ${PS4_LAN_ADDRESS}:1935 443 \ redirect_port tcp ${PS4_LAN_ADDRESS}:1935 1935 \ redirect_port tcp ${PS4_LAN_ADDRESS}:3478 3478 \ redirect_port tcp ${PS4_LAN_ADDRESS}:3479 3479 \ redirect_port tcp ${PS4_LAN_ADDRESS}:3480 3480 \ redirect_port udp ${PS4_LAN_ADDRESS}:3478 3478 \ redirect_port udp ${PS4_LAN_ADDRESS}:3479 3479 but that completely kills my Internet connection for all other uses (and the PS4 still doesn't work!). Could somebody please give me some clues. Also, what are the risks of doing this port forwarding? Thanks, Graham From owner-freebsd-ipfw@freebsd.org Fri Nov 27 04:55:20 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A6842A3A14A for ; Fri, 27 Nov 2015 04:55:20 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from mail.7sq.com.au (mail.7sq.com.au [119.148.74.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 24B0B1B4F; Fri, 27 Nov 2015 04:55:19 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id A667D2C3305; Fri, 27 Nov 2015 14:50:56 +1000 (AEST) Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id Vqgulcc1sn8J; Fri, 27 Nov 2015 14:50:56 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id 6E1C92C330B; Fri, 27 Nov 2015 14:50:56 +1000 (AEST) X-Virus-Scanned: amavisd-new at mail.7sq.com.au Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id wdY-YqhWGFST; Fri, 27 Nov 2015 14:50:56 +1000 (AEST) Received: from [172.20.10.3] (unknown [1.132.29.186]) by mail.7sq.com.au (Postfix) with ESMTPSA id B92F12C3305; Fri, 27 Nov 2015 14:50:55 +1000 (AEST) Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Subject: Re: Kernel NAT issues From: Nathan Aherne In-Reply-To: <5652B9EB.10805@freebsd.org> Date: Fri, 27 Nov 2015 14:55:34 +1000 Cc: Ian Smith , freebsd-ipfw@freebsd.org Message-Id: References: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> <20151013142301.B67283@sola.nimnet.asn.au> <20151014232026.S15983@sola.nimnet.asn.au> <9908EC22-344F-4D0B-8930-7D2C70B084A1@reddog.com.au> <32DEEFB3-E41F-40CD-8E1A-520FB261C572@reddog.com.au> <564C8879.8070307@freebsd.org> <20151119032200.T27669@sola.nimnet.asn.au> <9D81BDD4-200C-40AB-AB24-B1112881E43A@reddog.com.au> <3BF360A8-35E6-4043-8AFF-87D983F29C66@reddog.com.au> <5652B9EB.10805@freebsd.org> To: Julian Elischer X-Mailer: Apple Mail (2.2104) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Nov 2015 04:55:20 -0000 Hi Julian, Thank you for replying. I was completely off grid for a while and only = got back on it today. I thought that Vimage was probably the way to achieve what I want. The = main reason I was staying away from Vimage was the reported bugs with = it, another reason was the extra overhead. I would like to be able to = shutdown jails quite regularly so was worried the kernel panic bug or = memory leak bug might be a problem here. Is there any version of = Vimage/FreeBSD which is stable? Regards, Nathan > On 23 Nov 2015, at 5:02 pm, Julian Elischer = wrote: >=20 > On 21/11/2015 10:06 AM, Nathan Aherne wrote: >> I had a bit of a think about how to describe what I am trying to = achieve. >>=20 >> I am treating each jail likes its own little "virtual machine=E2=80=9D.= The jail provides certain services, using things like nginx or nodejs, = php-fpm, mysql or postgresql. The jails can control connections to = themselves by configuring the firewall ports that are opened on the IP = their IP (10.0.0.0/16 or a public IP). I know the jails have no = firewall of their own, the firewall is configured from the host. >>=20 >> I want each jail or =E2=80=9Cvirtual machine=E2=80=9D to be able to = communicate with one another and the wider internet. When a jail does a = DNS query for another App jail, it may get a public IP on its own Host = (or it may get another host) and it has no issues being able to = communicate with another jail on the same host. >>=20 >> At the moment all of the above is working perfectly except for jail = to jail communication on the same host (when the communication is not = directly between 10.0.0.0/16 IP addresses). > this is pretty much exactly when vimage/vnet jails could be used to = great affect. > Is there a reason you are not doing that? Each jail has it's own = routing tables, addresses and (virtual) interfaces. >=20 > here's how I'd do it with vimage >=20 > +--------------+ > +---------------+ | servers > | +--------------+ > | > | +--------------+ > | +--------+ | > | | +--------------+ > | | > +--------+ +--+------+----+ > | iface | | bridge | > | +-----+ | > +--------+ +----+---------+ > | > | > | > | > | > | > +------------------------+---------------------+ > | | > | | > | NAT jail router | > | | > | | > +-------+--------+--------+-------+------------+ > | | | | > +--+--+ +--+--+ +--+--+ +--+--+ > | | | | | | | | > | | | | | | | | > | | | | | | | | jails > | | | | | | | | > +-----+ +-----+ +-----+ +-----+ >=20 >=20 >=20 > however the hairpin idea might still be useful even in that scenario = if they don't know about each other's 'local' addresses, but do NAT'd = machines need to talk to each other by externeal addresses? >=20 > i Nathan >>> On 21 Nov 2015, at 9:12 am, Nathan Aherne = wrote: >>>=20 >>> I am not exactly sure how to draw the setup so it doesn=E2=80=99t = confuse the situation. The setup is extremely simple (I am not running = vimage), jails running on the 10.0.0.0/16 (cloned lo1 interface) network = or with public IPs. The jails with private IPs are the HTTP app jails. = The Host runs a HTTP Proxy (nginx) and forwards traffic to each HTTP App = jail based on the URL it receives. The jails with public IPs are things = like database jails which cannot be proxied by the Host. >>>=20 >>> I can happily communicate with any jail from my laptop (externally) = but when I want one jail to communicate with another jail (for example = an App Jail communicating with the database jail) the traffic shows as = backwards (destination:port -> source:port) in the IPFW logs (tshark = shows the traffic correctly source:port -> destination:port). The jail = to jail traffic tries to go over the lo1 interface (backwards) and is = blocked. Below is some IPFW logs of an App jail (10.0.0.25) = communicating with the database jail (aaa.bbb.ccc.ddd) >>>=20 >>> IPFW logs. The lines labelled UNKNOWN is the check-state rule = (everything is labelled UNKNOWN even if it is KNOWN traffic) >>>=20 >>> Nov 21 08:49:07 host5 kernel: ipfw: 101 UNKNOWN TCP = eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >>> Nov 21 08:49:07 host5 kernel: ipfw: 65501 Deny TCP = eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >>> Nov 21 08:49:10 host5 kernel: ipfw: 101 UNKNOWN TCP = eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >>> Nov 21 08:49:10 host5 kernel: ipfw: 65501 Deny TCP = eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >>> Nov 21 08:49:13 host5 kernel: ipfw: 101 UNKNOWN TCP = eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >>> Nov 21 08:49:13 host5 kernel: ipfw: 65501 Deny TCP = eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >>> Nov 21 08:49:16 host5 kernel: ipfw: 101 UNKNOWN TCP = eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >>> Nov 21 08:49:16 host5 kernel: ipfw: 65501 Deny TCP = eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >>>=20 >>> tshark output (loopback and wan interface capture for port 5432) >>>=20 >>> Capturing on 'Loopback' and 'bce0' >>> 1 0.000000 10.0.0.25 -> eee.fff.gg.hhh TCP 64 42957=E2=86=92543= 2 [SYN] Seq=3D0 Win=3D65535 Len=3D0 MSS=3D16344 WS=3D64 SACK_PERM=3D1 = TSval=3D142885525 TSecr=3D0 >>> 2 3.013905 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142888539 TSecr=3D0 >>> 3 6.241658 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142891767 TSecr=3D0 >>> 4 9.451516 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142894976 TSecr=3D0 >>> 5 12.654656 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142898180 TSecr=3D0 >>> 6 15.863900 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142901389 TSecr=3D0 >>> 7 22.076655 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142907602 TSecr=3D0 >>>=20 >>>=20 >>>> If so, what sort of routing is setup on both host and jails? >>> Routing is what would be added by default (whatever the host system = adds when adding an IP), there is no custom routing. I have wondered if = I need to modify the routing table to get this to work. >>>=20 >>> Below is the output of netstat -rn >>>=20 >>> www.xxx.yy .zzz is the gateway address >>> eee.fff.gg.hhh is the database jail public IP >>> aaa.bbb.cc.ddd is the public IP for NAT >>> lll.mmm.nn.ooo is the Hosts public IP >>>=20 >>>=20 >>> Routing tables >>>=20 >>> Internet: >>> Destination Gateway Flags Netif Expire >>> default www.xxx.yy .zzz UGS = bce0 >>> 10.0.0.1 link#6 UH lo1 >>> 10.0.0.2 link#6 UH lo1 >>> 10.0.0.3 link#6 UH lo1 >>> 10.0.0.4 link#6 UH lo1 >>> 10.0.0.5 link#6 UH lo1 >>> 10.0.0.6 link#6 UH lo1 >>> 10.0.0.7 link#6 UH lo1 >>> 10.0.0.8 link#6 UH lo1 >>> 10.0.0.9 link#6 UH lo1 >>> 10.0.0.10 link#6 UH lo1 >>> 10.0.0.11 link#6 UH lo1 >>> 10.0.0.12 link#6 UH lo1 >>> 10.0.0.13 link#6 UH lo1 >>> 10.0.0.14 link#6 UH lo1 >>> 10.0.0.15 link#6 UH lo1 >>> 10.0.0.16 link#6 UH lo1 >>> 10.0.0.17 link#6 UH lo1 >>> 10.0.0.18 link#6 UH lo1 >>> 10.0.0.19 link#6 UH lo1 >>> 10.0.0.20 link#6 UH lo1 >>> 10.0.0.21 link#6 UH lo1 >>> 10.0.0.22 link#6 UH lo1 >>> 10.0.0.23 link#6 UH lo1 >>> 10.0.0.24 link#6 UH lo1 >>> 10.0.0.25 link#6 UH lo1 >>> 10.0.0.26 link#6 UH lo1 >>> www.xxx.yy.zzz/25 link#1 U = bce0 >>> eee.fff.gg.hhh link#1 UHS lo0 >>> eee.fff.gg.hhh/32 link#1 U bce0 >>> aaa.bbb.cc .ddd link#1 UHS = lo0 >>> aaa.bbb.cc.ddd/32 link#1 U bce0 >>> lll.mmm.nn.ooo link#1 UHS lo0 >>> 127.0.0.1 link#5 UH lo0 >>>=20 >>> Internet6: >>> Destination Gateway = Flags Netif Expire >>> ::/96 ::1 UGRS = lo0 >>> ::1 link#5 UH = lo0 >>> ::ffff:0.0.0.0/96 ::1 UGRS = lo0 >>> fe80::/10 ::1 UGRS = lo0 >>> fe80::%lo0/64 link#5 U = lo0 >>> fe80::1%lo0 link#5 UHS = lo0 >>> ff01::%lo0/32 ::1 U = lo0 >>> ff02::/16 ::1 UGRS = lo0 >>> ff02::%lo0/32 ::1 U = lo0 >>>=20 >>>> Anything like ? >>>> = http://kb.juniper.net/InfoCenter/index?page=3Dcontent&id=3DKB24639&actp=3D= search = >>> Yes just like that. >>>=20 >>> Regards, >>>=20 >>> Nathan >>>=20 >>>> On 19 Nov 2015, at 2:46 am, Ian Smith > wrote: >>>>=20 >>>> On Wed, 18 Nov 2015 22:17:29 +0800, Julian Elischer wrote: >>>>> On 11/18/15 8:40 AM, Nathan Aherne wrote: >>>>>> For some reason hairpin (loopback nat or nat reflection) does not = seem to >>>>>> be working, which is why I chose IPFW in the first place. >>>>> it would be good to see a diagram of what this actually means. >>>> Anything like ? >>>> = http://kb.juniper.net/InfoCenter/index?page=3Dcontent&id=3DKB24639&actp=3D= search = >>>>=20 >>>> Was this so one jail can only access service/s provided by other = jail/s, >>>> both/all with internal NAT'd addresses, by using only the public = address >>>> and port of the 'router', which IIRC this is a single system with = jails? >>>>=20 >>>> If so, what sort of routing is setup on both host and jails? >>>>=20 >>>> (blindfolded, no idea where I've pinned the donkey's tail :) >>>>=20 >>>> cheers, Ian >> _______________________________________________ >> freebsd-ipfw@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw >> To unsubscribe, send any mail to = "freebsd-ipfw-unsubscribe@freebsd.org" >>=20 >>=20 >=20 > _______________________________________________ > freebsd-ipfw@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > To unsubscribe, send any mail to = "freebsd-ipfw-unsubscribe@freebsd.org" From owner-freebsd-ipfw@freebsd.org Fri Nov 27 18:03:28 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E2A7CA3A292 for ; Fri, 27 Nov 2015 18:03:27 +0000 (UTC) (envelope-from thomas5th@gmail.com) Received: from mail-qg0-x232.google.com (mail-qg0-x232.google.com [IPv6:2607:f8b0:400d:c04::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9BE7F1AB2 for ; Fri, 27 Nov 2015 18:03:27 +0000 (UTC) (envelope-from thomas5th@gmail.com) Received: by qgec40 with SMTP id c40so77150845qge.2 for ; Fri, 27 Nov 2015 10:03:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to; bh=6mkR9mQgGj9Ul7rbKG14B38DdCSOQzstrvwjb5tuGkE=; b=gGW67G9fxtvkjSstcNDAsb8mmh/kjfN/glp9KMzZEup8vokO1qNW+LW3sQVQaqvZ3m XH5nfpK3Z+1jW3wG4XM98FVSN4j8+jYz5+eW4vyUyz15GxWsN7nw2eGvn1CsseHc6hK7 OfInucXVc8LX0Xm8RZvEcm4A9Ts1mslU7Ia3W/naa02E+bMl8nhtQX9Y/pdagsu1io6z uXOeFNHPGk6iKa5MuksoMaU1HdWQ9x1LQiHWrnV99pgniO+BeXSlFdlqJIEtNfgXllFY QsfH65H96VX9oE9t5p4Ytc8oJi62PNPrWNk1mHLtsJH72P9M17X8gwSc9To2RAjbRRxT FWrA== X-Received: by 10.140.174.68 with SMTP id u65mr60023942qhu.62.1448647406683; Fri, 27 Nov 2015 10:03:26 -0800 (PST) Received: from host ([201.47.217.187]) by smtp.gmail.com with ESMTPSA id o20sm9822962qki.42.2015.11.27.10.03.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 27 Nov 2015 10:03:25 -0800 (PST) Date: Fri, 27 Nov 2015 16:03:17 -0200 From: Thomas To: Graham Menhennitt Cc: freebsd-ipfw@freebsd.org Subject: Re: connecting a PS4 via IPFW Message-ID: <20151127180317.GA3661@host> References: <5655405C.1060301@menhennitt.com.au> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5655405C.1060301@menhennitt.com.au> X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Nov 2015 18:03:28 -0000 Aren't your regular NAT rules in NAT instance 1? That command will overwrite those and leave just the new ones. If that's the case, you can put those rules in a different NAT instance, or add them to the rest of your NAT rules. As to the risk, given the security history of UPnP, this kind of static port forwarding might be more secure. Redirect only the necessary ports and don't worry about it. Or isolate the PS4 in a DMZ, if you're concerned it could be used as a point of entry into your network. Cheers, Thomas Wed, Nov 25, 2015 at 04:00:12PM +1100, Graham Menhennitt: > Hello IPFWers, > > I have a box running FreeBSD 10-stable that I use as a > router/firewall/NAT. It runs IPFW and uses kernel NAT. My son is nagging > me about playing multi-player online games on his Sony PS4. > > From what I've read, I could enable UPnP. But I've tried compiling the > net/miniupnpd port but it won't build for IPFW (and I don't want to > convert to PF). > > Giving up on that, I'm now trying to enable port forwarding - > apparently, this will fix it. I've allocated the PS4 a static IP address > on my LAN. I need to port forward TCP ports 80, 443, 1935, 3478-3480, > and UDP ports 3478-3479. I've tried the following command: > > ipfw nat 1 config \ > redirect_port tcp ${PS4_LAN_ADDRESS}:1935 80 \ > redirect_port tcp ${PS4_LAN_ADDRESS}:1935 443 \ > redirect_port tcp ${PS4_LAN_ADDRESS}:1935 1935 \ > redirect_port tcp ${PS4_LAN_ADDRESS}:3478 3478 \ > redirect_port tcp ${PS4_LAN_ADDRESS}:3479 3479 \ > redirect_port tcp ${PS4_LAN_ADDRESS}:3480 3480 \ > redirect_port udp ${PS4_LAN_ADDRESS}:3478 3478 \ > redirect_port udp ${PS4_LAN_ADDRESS}:3479 3479 > > but that completely kills my Internet connection for all other uses (and > the PS4 still doesn't work!). > > Could somebody please give me some clues. > > Also, what are the risks of doing this port forwarding? > > Thanks, > Graham > _______________________________________________ > freebsd-ipfw@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org" From owner-freebsd-ipfw@freebsd.org Sat Nov 28 04:19:20 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2D07CA3A503 for ; Sat, 28 Nov 2015 04:19:20 +0000 (UTC) (envelope-from graham@menhennitt.com.au) Received: from homiemail-a82.g.dreamhost.com (sub5.mail.dreamhost.com [208.113.200.129]) by mx1.freebsd.org (Postfix) with ESMTP id 088A9195C for ; Sat, 28 Nov 2015 04:19:19 +0000 (UTC) (envelope-from graham@menhennitt.com.au) Received: from homiemail-a82.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a82.g.dreamhost.com (Postfix) with ESMTP id 14ABC282065; Fri, 27 Nov 2015 20:19:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=menhennitt.com.au; h= subject:to:references:cc:from:message-id:date:mime-version :in-reply-to:content-type:content-transfer-encoding; s= menhennitt.com.au; bh=SUfArlIMoj1F2Va0zKBlGg2a6z0=; b=oVoHx5Q7H5 Cq/pmzZdyDSBBiERWs18UOlOyPFJho917neVOYtmKXZDRN1XpBoXaX9a05Cfps+R aSJtohtUEYFYyftMPuN7zwHsrVZTXDJsuy8oWeX17ZdCUKbg360fhSaSRqmxEtaI /NHKdhKNC3QWEXyp52ayLWWIoYPP5Bl7w= Received: from [203.2.73.68] (c122-107-214-88.mckinn3.vic.optusnet.com.au [122.107.214.88]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: graham@menhennitt.com.au) by homiemail-a82.g.dreamhost.com (Postfix) with ESMTPSA id 4E85B282061; Fri, 27 Nov 2015 20:19:12 -0800 (PST) Subject: Re: connecting a PS4 via IPFW To: Thomas References: <5655405C.1060301@menhennitt.com.au> <20151127180317.GA3661@host> Cc: freebsd-ipfw@freebsd.org From: Graham Menhennitt X-Enigmail-Draft-Status: N1110 Message-ID: <56592B3D.5070800@menhennitt.com.au> Date: Sat, 28 Nov 2015 15:19:09 +1100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <20151127180317.GA3661@host> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Nov 2015 04:19:20 -0000 On 28/11/2015 05:03, Thomas wrote: > Aren't your regular NAT rules in NAT instance 1? That command will > overwrite those and leave just the new ones. > > If that's the case, you can put those rules in a different NAT instance, > or add them to the rest of your NAT rules. > > > Wed, Nov 25, 2015 at 04:00:12PM +1100, Graham Menhennitt: >> Hello IPFWers, >> >> I have a box running FreeBSD 10-stable that I use as a >> router/firewall/NAT. It runs IPFW and uses kernel NAT. My son is nagging >> me about playing multi-player online games on his Sony PS4. >> >> From what I've read, I could enable UPnP. But I've tried compiling the >> net/miniupnpd port but it won't build for IPFW (and I don't want to >> convert to PF). >> >> Giving up on that, I'm now trying to enable port forwarding - >> apparently, this will fix it. I've allocated the PS4 a static IP address >> on my LAN. I need to port forward TCP ports 80, 443, 1935, 3478-3480, >> and UDP ports 3478-3479. I've tried the following command: >> >> ... >> but that completely kills my Internet connection for all other uses (and >> the PS4 still doesn't work!). >> Thanks for responding, Thomas. You're right - that's what's killing my Internet connection. I've changed the command to: ipfw nat 1 config if re1 same_ports \ redirect_port tcp ${PS4_LAN_ADDRESS}:80 80 \ redirect_port tcp ${PS4_LAN_ADDRESS}:443 443 \ redirect_port tcp ${PS4_LAN_ADDRESS}:1935 1935 \ redirect_port tcp ${PS4_LAN_ADDRESS}:3478 3478 \ redirect_port tcp ${PS4_LAN_ADDRESS}:3479 3479 \ redirect_port tcp ${PS4_LAN_ADDRESS}:3480 3480 \ redirect_port udp ${PS4_LAN_ADDRESS}:3478 3478 \ redirect_port udp ${PS4_LAN_ADDRESS}:3479 3479 At least it doesn't kill anything now. But the PS4 still doesn't work. When I do "Test internet connection" on it, it says "NAT: Failed". For a DMZ, I need a separate network for the PS4 - is that right? That means a separate physical connection (i.e. WAP or cable). Thanks again, Graham From owner-freebsd-ipfw@freebsd.org Sat Nov 28 09:48:07 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D2985A3AC77 for ; Sat, 28 Nov 2015 09:48:07 +0000 (UTC) (envelope-from thomas5th@gmail.com) Received: from mail-qk0-x233.google.com (mail-qk0-x233.google.com [IPv6:2607:f8b0:400d:c09::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 910361621 for ; Sat, 28 Nov 2015 09:48:07 +0000 (UTC) (envelope-from thomas5th@gmail.com) Received: by qkao63 with SMTP id o63so43201464qka.2 for ; Sat, 28 Nov 2015 01:48:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to; bh=Va9zw8Pd23jMaf2haZ4UUcckHzG/3WLepgVnVKvlzJo=; b=INy5g1HiYSLxDNV2FOFPKZcXJf86kQAE0f3TSek56rLpiJudXUwG1cfHXH5oOaaYek Mk6WjXtxlfe7HUvwqEd8rRErsXQ6e8ypEztehW9PSp6rdPbMrjqk+nytr+Uaxhq7gU9c ld00/TPnps4MzYvRMgfaK3eg0XWvMW/Fz4JacLY8Loo/NnUP193Z/ayc928fJkjGnpbO FNl/FMuH8Ak4L5S/oEdjc3OGfwFrztCfHvrel5LlhXvaFDsiKXysIE3zkLMrX+HmUNUK BsoV9S84eXB69trfMOPYPZIcoef/qb4n6KAL1kR6w+k68QuUe3D0aRbhz4SglupKntlf MgpA== X-Received: by 10.55.23.170 with SMTP id 42mr58860340qkx.42.1448704086539; Sat, 28 Nov 2015 01:48:06 -0800 (PST) Received: from bloo ([201.47.217.187]) by smtp.gmail.com with ESMTPSA id x83sm10898171qka.14.2015.11.28.01.48.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 Nov 2015 01:48:06 -0800 (PST) Date: Sat, 28 Nov 2015 07:47:53 -0200 From: =?iso-8859-1?B?VGhvbeFzIFMu?= Bregolin To: Graham Menhennitt Cc: freebsd-ipfw@freebsd.org Subject: Re: connecting a PS4 via IPFW Message-ID: <20151128094753.GA5426@bloo> References: <5655405C.1060301@menhennitt.com.au> <20151127180317.GA3661@host> <56592B3D.5070800@menhennitt.com.au> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <56592B3D.5070800@menhennitt.com.au> X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Nov 2015 09:48:08 -0000 Hello, Besides the redirect_port option, you still need rules allowing traffic in to those ports. Excuse-me if you've done that already (I have no way of knowing). I'd suggest forwarding one or all of those ports to another host, and testing with netcat in order to rule out problems with the PS4 configuration or some other problem upstream in your network. Sniffing with tcpdump might help. There are "port checkers" out there if you don't have another external address to test from (like canyouseeme.org). To do proper DMZ, yes, you'd either use more physical interfaces and connections, or VLANS. That could be overkill in your situation, unless you're really keen on having a very secure home network. Best of luck, Thomas Sat, Nov 28, 2015 at 03:19:09PM +1100, Graham Menhennitt: > On 28/11/2015 05:03, Thomas wrote: > > Aren't your regular NAT rules in NAT instance 1? That command will > > overwrite those and leave just the new ones. > > > > If that's the case, you can put those rules in a different NAT instance, > > or add them to the rest of your NAT rules. > > > > > > Wed, Nov 25, 2015 at 04:00:12PM +1100, Graham Menhennitt: > >> Hello IPFWers, > >> > >> I have a box running FreeBSD 10-stable that I use as a > >> router/firewall/NAT. It runs IPFW and uses kernel NAT. My son is nagging > >> me about playing multi-player online games on his Sony PS4. > >> > >> From what I've read, I could enable UPnP. But I've tried compiling the > >> net/miniupnpd port but it won't build for IPFW (and I don't want to > >> convert to PF). > >> > >> Giving up on that, I'm now trying to enable port forwarding - > >> apparently, this will fix it. I've allocated the PS4 a static IP address > >> on my LAN. I need to port forward TCP ports 80, 443, 1935, 3478-3480, > >> and UDP ports 3478-3479. I've tried the following command: > >> > >> ... > >> but that completely kills my Internet connection for all other uses (and > >> the PS4 still doesn't work!). > >> > Thanks for responding, Thomas. You're right - that's what's killing my > Internet connection. I've changed the command to: > > ipfw nat 1 config if re1 same_ports \ > redirect_port tcp ${PS4_LAN_ADDRESS}:80 80 \ > redirect_port tcp ${PS4_LAN_ADDRESS}:443 443 \ > redirect_port tcp ${PS4_LAN_ADDRESS}:1935 1935 \ > redirect_port tcp ${PS4_LAN_ADDRESS}:3478 3478 \ > redirect_port tcp ${PS4_LAN_ADDRESS}:3479 3479 \ > redirect_port tcp ${PS4_LAN_ADDRESS}:3480 3480 \ > redirect_port udp ${PS4_LAN_ADDRESS}:3478 3478 \ > redirect_port udp ${PS4_LAN_ADDRESS}:3479 3479 > > At least it doesn't kill anything now. But the PS4 still doesn't work. > When I do "Test internet connection" on it, it says "NAT: Failed". > > For a DMZ, I need a separate network for the PS4 - is that right? That > means a separate physical connection (i.e. WAP or cable). > > Thanks again, > Graham From owner-freebsd-ipfw@freebsd.org Sat Nov 28 11:16:33 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B1252A3ACA4; Sat, 28 Nov 2015 11:16:33 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7D0E01393; Sat, 28 Nov 2015 11:16:30 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (ppp121-45-225-88.lns20.per1.internode.on.net [121.45.225.88]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id tASBGK0f085176 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Sat, 28 Nov 2015 03:16:23 -0800 (PST) (envelope-from julian@freebsd.org) Subject: Re: Kernel NAT issues To: Nathan Aherne References: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> <20151013142301.B67283@sola.nimnet.asn.au> <20151014232026.S15983@sola.nimnet.asn.au> <9908EC22-344F-4D0B-8930-7D2C70B084A1@reddog.com.au> <32DEEFB3-E41F-40CD-8E1A-520FB261C572@reddog.com.au> <564C8879.8070307@freebsd.org> <20151119032200.T27669@sola.nimnet.asn.au> <9D81BDD4-200C-40AB-AB24-B1112881E43A@reddog.com.au> <3BF360A8-35E6-4043-8AFF-87D983F29C66@reddog.com.au> <5652B9EB.10805@freebsd.org> Cc: freebsd-ipfw@freebsd.org, Ian Smith , "freebsd-net@freebsd.org" From: Julian Elischer Message-ID: <56598CFF.3060102@freebsd.org> Date: Sat, 28 Nov 2015 19:16:15 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Nov 2015 11:16:33 -0000 On 27/11/2015 12:55 PM, Nathan Aherne wrote: > Hi Julian, > > Thank you for replying. I was completely off grid for a while and only got back on it today. > > I thought that Vimage was probably the way to achieve what I want. The main reason I was staying away from Vimage was the reported bugs with it, another reason was the extra overhead. I would like to be able to shutdown jails quite regularly so was worried the kernel panic bug or memory leak bug might be a problem here. Is there any version of Vimage/FreeBSD which is stable? Generally vimage is stable. It has had problems with pf over the years becasue pf is imported from OpenBSD and has some pretty vimage-unfriendly assumptions in its design, but I hear that even some of thise have been ironed out. I know of vimage being used to run production virtual systems in some of the largest banks in the world processing amounts of trnasactions that would make your head spin so have a small play with it. Vimage overhead is negative in some situations. i.e. things work faster.. This is especially true when non vimage workloads contest a single lock heavily, but vimage splits it over many locks.. one for each VM. run up a virtualbox or amazon or whatever freebsd instance and play around with it. once realize how insanely powerful it is, you will wonder how you ever did jails without it. you can use bridges, epairs or netgraph to do your networking... your choice. > > Regards, > > Nathan > >> On 23 Nov 2015, at 5:02 pm, Julian Elischer wrote: >> >> On 21/11/2015 10:06 AM, Nathan Aherne wrote: >>> I had a bit of a think about how to describe what I am trying to achieve. >>> >>> I am treating each jail likes its own little "virtual machine”. The jail provides certain services, using things like nginx or nodejs, php-fpm, mysql or postgresql. The jails can control connections to themselves by configuring the firewall ports that are opened on the IP their IP (10.0.0.0/16 or a public IP). I know the jails have no firewall of their own, the firewall is configured from the host. >>> >>> I want each jail or “virtual machine” to be able to communicate with one another and the wider internet. When a jail does a DNS query for another App jail, it may get a public IP on its own Host (or it may get another host) and it has no issues being able to communicate with another jail on the same host. >>> >>> At the moment all of the above is working perfectly except for jail to jail communication on the same host (when the communication is not directly between 10.0.0.0/16 IP addresses). >> this is pretty much exactly when vimage/vnet jails could be used to great affect. >> Is there a reason you are not doing that? Each jail has it's own routing tables, addresses and (virtual) interfaces. >> >> here's how I'd do it with vimage >> >> +--------------+ >> +---------------+ | servers >> | +--------------+ >> | >> | +--------------+ >> | +--------+ | >> | | +--------------+ >> | | >> +--------+ +--+------+----+ >> | iface | | bridge | >> | +-----+ | >> +--------+ +----+---------+ >> | >> | >> | >> | >> | >> | >> +------------------------+---------------------+ >> | | >> | | >> | NAT jail router | >> | | >> | | >> +-------+--------+--------+-------+------------+ >> | | | | >> +--+--+ +--+--+ +--+--+ +--+--+ >> | | | | | | | | >> | | | | | | | | >> | | | | | | | | jails >> | | | | | | | | >> +-----+ +-----+ +-----+ +-----+ >> >> >> >> however the hairpin idea might still be useful even in that scenario if they don't know about each other's 'local' addresses, but do NAT'd machines need to talk to each other by externeal addresses? >> >> i Nathan >>>> On 21 Nov 2015, at 9:12 am, Nathan Aherne wrote: >>>> >>>> I am not exactly sure how to draw the setup so it doesn’t confuse the situation. The setup is extremely simple (I am not running vimage), jails running on the 10.0.0.0/16 (cloned lo1 interface) network or with public IPs. The jails with private IPs are the HTTP app jails. The Host runs a HTTP Proxy (nginx) and forwards traffic to each HTTP App jail based on the URL it receives. The jails with public IPs are things like database jails which cannot be proxied by the Host. >>>> >>>> I can happily communicate with any jail from my laptop (externally) but when I want one jail to communicate with another jail (for example an App Jail communicating with the database jail) the traffic shows as backwards (destination:port -> source:port) in the IPFW logs (tshark shows the traffic correctly source:port -> destination:port). The jail to jail traffic tries to go over the lo1 interface (backwards) and is blocked. Below is some IPFW logs of an App jail (10.0.0.25) communicating with the database jail (aaa.bbb.ccc.ddd) >>>> >>>> IPFW logs. The lines labelled UNKNOWN is the check-state rule (everything is labelled UNKNOWN even if it is KNOWN traffic) >>>> >>>> Nov 21 08:49:07 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >>>> Nov 21 08:49:07 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >>>> Nov 21 08:49:10 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >>>> Nov 21 08:49:10 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >>>> Nov 21 08:49:13 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >>>> Nov 21 08:49:13 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >>>> Nov 21 08:49:16 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >>>> Nov 21 08:49:16 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >>>> >>>> tshark output (loopback and wan interface capture for port 5432) >>>> >>>> Capturing on 'Loopback' and 'bce0' >>>> 1 0.000000 10.0.0.25 -> eee.fff.gg.hhh TCP 64 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142885525 TSecr=0 >>>> 2 3.013905 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142888539 TSecr=0 >>>> 3 6.241658 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142891767 TSecr=0 >>>> 4 9.451516 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142894976 TSecr=0 >>>> 5 12.654656 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142898180 TSecr=0 >>>> 6 15.863900 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142901389 TSecr=0 >>>> 7 22.076655 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142907602 TSecr=0 >>>> >>>> >>>>> If so, what sort of routing is setup on both host and jails? >>>> Routing is what would be added by default (whatever the host system adds when adding an IP), there is no custom routing. I have wondered if I need to modify the routing table to get this to work. >>>> >>>> Below is the output of netstat -rn >>>> >>>> www.xxx.yy .zzz is the gateway address >>>> eee.fff.gg.hhh is the database jail public IP >>>> aaa.bbb.cc.ddd is the public IP for NAT >>>> lll.mmm.nn.ooo is the Hosts public IP >>>> >>>> >>>> Routing tables >>>> >>>> Internet: >>>> Destination Gateway Flags Netif Expire >>>> default www.xxx.yy .zzz UGS bce0 >>>> 10.0.0.1 link#6 UH lo1 >>>> 10.0.0.2 link#6 UH lo1 >>>> 10.0.0.3 link#6 UH lo1 >>>> 10.0.0.4 link#6 UH lo1 >>>> 10.0.0.5 link#6 UH lo1 >>>> 10.0.0.6 link#6 UH lo1 >>>> 10.0.0.7 link#6 UH lo1 >>>> 10.0.0.8 link#6 UH lo1 >>>> 10.0.0.9 link#6 UH lo1 >>>> 10.0.0.10 link#6 UH lo1 >>>> 10.0.0.11 link#6 UH lo1 >>>> 10.0.0.12 link#6 UH lo1 >>>> 10.0.0.13 link#6 UH lo1 >>>> 10.0.0.14 link#6 UH lo1 >>>> 10.0.0.15 link#6 UH lo1 >>>> 10.0.0.16 link#6 UH lo1 >>>> 10.0.0.17 link#6 UH lo1 >>>> 10.0.0.18 link#6 UH lo1 >>>> 10.0.0.19 link#6 UH lo1 >>>> 10.0.0.20 link#6 UH lo1 >>>> 10.0.0.21 link#6 UH lo1 >>>> 10.0.0.22 link#6 UH lo1 >>>> 10.0.0.23 link#6 UH lo1 >>>> 10.0.0.24 link#6 UH lo1 >>>> 10.0.0.25 link#6 UH lo1 >>>> 10.0.0.26 link#6 UH lo1 >>>> www.xxx.yy.zzz/25 link#1 U bce0 >>>> eee.fff.gg.hhh link#1 UHS lo0 >>>> eee.fff.gg.hhh/32 link#1 U bce0 >>>> aaa.bbb.cc .ddd link#1 UHS lo0 >>>> aaa.bbb.cc.ddd/32 link#1 U bce0 >>>> lll.mmm.nn.ooo link#1 UHS lo0 >>>> 127.0.0.1 link#5 UH lo0 >>>> >>>> Internet6: >>>> Destination Gateway Flags Netif Expire >>>> ::/96 ::1 UGRS lo0 >>>> ::1 link#5 UH lo0 >>>> ::ffff:0.0.0.0/96 ::1 UGRS lo0 >>>> fe80::/10 ::1 UGRS lo0 >>>> fe80::%lo0/64 link#5 U lo0 >>>> fe80::1%lo0 link#5 UHS lo0 >>>> ff01::%lo0/32 ::1 U lo0 >>>> ff02::/16 ::1 UGRS lo0 >>>> ff02::%lo0/32 ::1 U lo0 >>>> >>>>> Anything like ? >>>>> http://kb.juniper.net/InfoCenter/index?page=content&id=KB24639&actp=search >>>> Yes just like that. >>>> >>>> Regards, >>>> >>>> Nathan >>>> >>>>> On 19 Nov 2015, at 2:46 am, Ian Smith > wrote: >>>>> >>>>> On Wed, 18 Nov 2015 22:17:29 +0800, Julian Elischer wrote: >>>>>> On 11/18/15 8:40 AM, Nathan Aherne wrote: >>>>>>> For some reason hairpin (loopback nat or nat reflection) does not seem to >>>>>>> be working, which is why I chose IPFW in the first place. >>>>>> it would be good to see a diagram of what this actually means. >>>>> Anything like ? >>>>> http://kb.juniper.net/InfoCenter/index?page=content&id=KB24639&actp=search >>>>> >>>>> Was this so one jail can only access service/s provided by other jail/s, >>>>> both/all with internal NAT'd addresses, by using only the public address >>>>> and port of the 'router', which IIRC this is a single system with jails? >>>>> >>>>> If so, what sort of routing is setup on both host and jails? >>>>> >>>>> (blindfolded, no idea where I've pinned the donkey's tail :) >>>>> >>>>> cheers, Ian >>> _______________________________________________ >>> freebsd-ipfw@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw >>> To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org" >>> >>> >> _______________________________________________ >> freebsd-ipfw@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw >> To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-ipfw@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org" > > From owner-freebsd-ipfw@freebsd.org Sat Nov 28 15:02:19 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 64401A3BBEB for ; Sat, 28 Nov 2015 15:02:19 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C2A9E121C for ; Sat, 28 Nov 2015 15:02:17 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id tASErGHb027241; Sun, 29 Nov 2015 01:53:17 +1100 (EST) (envelope-from smithi@nimnet.asn.au) Date: Sun, 29 Nov 2015 01:53:16 +1100 (EST) From: Ian Smith To: Graham Menhennitt cc: Thomas , freebsd-ipfw@freebsd.org Subject: Re: connecting a PS4 via IPFW In-Reply-To: <56592B3D.5070800@menhennitt.com.au> Message-ID: <20151129003238.F69136@sola.nimnet.asn.au> References: <5655405C.1060301@menhennitt.com.au> <20151127180317.GA3661@host> <56592B3D.5070800@menhennitt.com.au> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Nov 2015 15:02:19 -0000 On Sat, 28 Nov 2015 15:19:09 +1100, Graham Menhennitt wrote: > On 28/11/2015 05:03, Thomas wrote: > > Aren't your regular NAT rules in NAT instance 1? That command will > > overwrite those and leave just the new ones. > > > > If that's the case, you can put those rules in a different NAT instance, > > or add them to the rest of your NAT rules. > > > > > > Wed, Nov 25, 2015 at 04:00:12PM +1100, Graham Menhennitt: > >> Hello IPFWers, > >> > >> I have a box running FreeBSD 10-stable that I use as a > >> router/firewall/NAT. It runs IPFW and uses kernel NAT. My son is nagging > >> me about playing multi-player online games on his Sony PS4. > >> > >> From what I've read, I could enable UPnP. But I've tried compiling the > >> net/miniupnpd port but it won't build for IPFW (and I don't want to > >> convert to PF). > >> > >> Giving up on that, I'm now trying to enable port forwarding - > >> apparently, this will fix it. I've allocated the PS4 a static IP address > >> on my LAN. I need to port forward TCP ports 80, 443, 1935, 3478-3480, > >> and UDP ports 3478-3479. I've tried the following command: > >> > >> ... > >> but that completely kills my Internet connection for all other uses (and > >> the PS4 still doesn't work!). > >> > Thanks for responding, Thomas. You're right - that's what's killing my > Internet connection. I've changed the command to: > > ipfw nat 1 config if re1 same_ports \ > redirect_port tcp ${PS4_LAN_ADDRESS}:80 80 \ > redirect_port tcp ${PS4_LAN_ADDRESS}:443 443 \ > redirect_port tcp ${PS4_LAN_ADDRESS}:1935 1935 \ > redirect_port tcp ${PS4_LAN_ADDRESS}:3478 3478 \ > redirect_port tcp ${PS4_LAN_ADDRESS}:3479 3479 \ > redirect_port tcp ${PS4_LAN_ADDRESS}:3480 3480 \ > redirect_port udp ${PS4_LAN_ADDRESS}:3478 3478 \ > redirect_port udp ${PS4_LAN_ADDRESS}:3479 3479 > > At least it doesn't kill anything now. But the PS4 still doesn't work. > When I do "Test internet connection" on it, it says "NAT: Failed". To add a little to what Thomas has contributed .. Not knowing what ipfw rules you're running apart from NAT, it should be rewarding adding 'log' to rules that could block access, and/or adding 'count log' rules to highlight the traffic to/from $PS4_LAN_ADDRESS, which should illustrate connection problems, NAT or other ipfw issues. Firstly I'd just run tcpdump on both outside and inside interfaces over some test sessions. Again, shouldn't take too long to spot the issue/s. Often these sort of things only want to use port 80 or 443 if they can't connect on their more favoured ports and/or protocol, perhaps as a last resort behind some corporate firewalls. Personally I'd only expose 80 or 443 inbound if running host services on them; too many drive-by bots. Does the PS4 have any setting for favoured protocol/port/s? Perhaps you can limit it to a smaller range than all the above, fewer being better? For safety, you should deny and log accesses from the PS4 to unexpected hosts (including yours) and/or ports, rightly concerned about security - of which upnp is the antithesis! > For a DMZ, I need a separate network for the PS4 - is that right? That > means a separate physical connection (i.e. WAP or cable). Separate network connection, whether physical or virtual. Testing with tcpdump is a lot easier to find out how you can limit port forwarding, to see if it's wild enough to make a DMZ advisable for your use case. And if PS4 usage becomes too heavy, dummynet becomes your best mate :) cheers, Ian From owner-freebsd-ipfw@freebsd.org Sat Nov 28 17:55:00 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C983BA3723C for ; Sat, 28 Nov 2015 17:55:00 +0000 (UTC) (envelope-from dewaynegeraghty@gmail.com) Received: from mail-lf0-x230.google.com (mail-lf0-x230.google.com [IPv6:2a00:1450:4010:c07::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 65C241A37; Sat, 28 Nov 2015 17:55:00 +0000 (UTC) (envelope-from dewaynegeraghty@gmail.com) Received: by lfdl133 with SMTP id l133so156074161lfd.2; Sat, 28 Nov 2015 09:54:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=9rVUJohWhyx658CiLKLxyh9FiILS6AxyOWem7m76Lu4=; b=wy43I0kNCeRAtFLrVTi554MLmubZ+orwxZGC+3jpwS79B6LRkKvDErTrGCVX1bJbIu ucC7F6qsRHZMsa4QkwMZ7OdgyqvZD2zWzE2XCVFFZyGHQ3cExEGG/qYhpSBlAZYAKA5a WoG4o/Y6K1IEeOYaE2HAU913C7rYJo6G9hayAnLzLwNRzYU+SN5aUq9L1+9CPHflhnG9 earqSM/8Ar6oHPZO25X3pn7EYwCdc0Bx2wHT2T0r+QofacQhSYUy6acLivYl1/2f1Ubs 4VRVb5iKZGx3IEUQdq0VJ29AxWJYBlCbbXjgAWr19dBcRKw30MyaGerCa8MKlfcJ38bU 3OjA== MIME-Version: 1.0 X-Received: by 10.112.199.194 with SMTP id jm2mr17231244lbc.109.1448733298441; Sat, 28 Nov 2015 09:54:58 -0800 (PST) Received: by 10.25.41.193 with HTTP; Sat, 28 Nov 2015 09:54:58 -0800 (PST) In-Reply-To: References: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> <20151013142301.B67283@sola.nimnet.asn.au> <20151014232026.S15983@sola.nimnet.asn.au> <9908EC22-344F-4D0B-8930-7D2C70B084A1@reddog.com.au> <32DEEFB3-E41F-40CD-8E1A-520FB261C572@reddog.com.au> <564C8879.8070307@freebsd.org> <20151119032200.T27669@sola.nimnet.asn.au> <9D81BDD4-200C-40AB-AB24-B1112881E43A@reddog.com.au> <3BF360A8-35E6-4043-8AFF-87D983F29C66@reddog.com.au> <5652B9EB.10805@freebsd.org> Date: Sun, 29 Nov 2015 04:54:58 +1100 Message-ID: Subject: Re: Kernel NAT issues From: Dewayne Geraghty To: Nathan Aherne Cc: Julian Elischer , "freebsd-ipfw@freebsd.org" , Ian Smith Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Nov 2015 17:55:01 -0000 Nathan, I've gone the same way that you have, ie bunch of jails that are individually providing services& kernel Nat. It takes careful planning and the knowledge that the default route will be the first IP in your jail.conf list for each jail. Getting jails to play nice means fiddling around with all interfaces. If you can take ipfw out of the equation until you can see tcpdump traffic doing what you want; the challenge hasn't been ipfw in my experience. (& yes initially I've had three tcpdumps going at once too; along with old friends: raw ip & ping ) Enjoy the fun of getting it to work, it's well worth the effort. (And be thankful that you aren't using pf, another level of complexity but suits my needs perfectly) ;) Dewayne