From owner-freebsd-ipfw@freebsd.org Sun Aug 30 02:59:24 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A0AA69C5FF7 for ; Sun, 30 Aug 2015 02:59:24 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from mail.7sq.com.au (mail.7sq.com.au [119.148.74.199]) by mx1.freebsd.org (Postfix) with ESMTP id E02C2E43 for ; Sun, 30 Aug 2015 02:59:22 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id 382EF2C0008 for ; Sun, 30 Aug 2015 12:48:45 +1000 (EST) Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id iGz_6V4fURI1 for ; Sun, 30 Aug 2015 12:48:45 +1000 (EST) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id 04D542C18EC for ; Sun, 30 Aug 2015 12:48:45 +1000 (EST) X-Virus-Scanned: amavisd-new at mail.7sq.com.au Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 6iuCnOqiS-Xb for ; Sun, 30 Aug 2015 12:48:44 +1000 (EST) Received: from [172.20.10.2] (unknown [1.129.28.125]) by mail.7sq.com.au (Postfix) with ESMTPSA id 247142C0008 for ; Sun, 30 Aug 2015 12:48:44 +1000 (EST) From: Nathan Aherne Message-Id: Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Subject: Re: IPFW and in kernel nat Date: Sun, 30 Aug 2015 12:49:32 +1000 References: To: freebsd-ipfw@freebsd.org In-Reply-To: X-Mailer: Apple Mail (2.2102) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Aug 2015 02:59:24 -0000 I have done some further work on this and I have been able to get kernel = nat working in ipfw but some functionality I am after didn=E2=80=99t = work. As I said in the below email I have a couple of jails with public ips, = those jails are unable to communicate with the jails on the private = network behind nat. The traffic seems to go via the loopback interface, = which is then blocked because its not going through nat. I have also found that while jails can communicate with each other = (hairpin nat is working) some of the jails cannot. The jails that cannot = communicate with each other are the ones which have ports redirected to = them. For instance, I have a jail on 10.1.0.9 and it is unable to = communicate with the jail on 10.1.0.2 via port 65432. The traffic seems = to be going over the loopback interface and not via nat. Any traffic = between jails where there is no redirected ports (via nat) work fine. I = have no custom routes, only what the system sets up when adding ips via = /etc/rc.conf. It seems this may be a bug with ipfw and kernel nat? Regards, Nathan Aherne nathan@reddog.com.au +61 402 555 080 > On 22 Aug 2015, at 1:19 pm, Nathan Aherne = wrote: >=20 > Hi Everyone, >=20 > First time message to this list, so I am sorry if I do something = against the rules. >=20 > I have posted this questions on the FreeBSD forums in two different = places but have not had a single response in several days, so thought I = might get more success here. I have spent many days solely reading about = IPFW and in kernel nat in the hopes that I would be able to get what I = want to achieve working without having to ask for help. I have found it = extremely hard to find what I would think is a regular use use case for = IPFW and nat. There are examples but there are just so many that are bad = or hard to follow, including the handbook and IPFW that its extremely = easy to get confused, as I have. Because of this, I will be posting my = eventual solution back to the forums in the hopes that it stops someone = from wasting days. >=20 > I have a single wan interface (bce0) that has two public IPs attached = to it. I would like to use one of these IPs for the host (wanip1) and = jails and another for a jail (wanip2) that requires a public IP. I have = a cloned lo0 interface, lo1 with a subnet of 10.1.0.0/24 which all the = jails reside on. I would like to forward ports 80,443 coming in on = wanip1 to the jail proxy, which then forwards the traffic off to the = correct jail. I would like for the host server to be able to have its = own firewall and also receive traffic on port 65222. I have a couple of = other jails that require some other ports as you will see in my = ipfw.rules script. I would also like each jail to be able to connect to = the internet. >=20 > I would like the jails to be able to have their own firewalls, which I = currently allow by including a firewall script from within each jail. >=20 > I have net.inet.ip.fw.one_pass=3D0 set in /etc/sysctl.conf >=20 > Below is my ipfw.rules script. I would super appreciate it if someone = could show me where I have gone wrong. >=20 > = **************************************************************************= **************************************************************************= ************************* > #!/bin/sh >=20 > ###################################################### > # Configuration > wif=3D"bce0" # WAN interface > wip=3D"119.111.111.111" # WAN IP >=20 > j1if=3D"lo1" # Jails Interface > j1net=3D"10.1.0.0/24" # Jails Network > skip=3D"skipto 30000" > jcmd=3D"ipfw -q add 10000=E2=80=9D # Jails rules are inserted here > ###################################################### > # IPFW variables > cmd=3D"ipfw -q add" > ks=3D"keep-state" > sks=3D"setup keep-state" > ###################################################### > ipfw -q -f flush # Flush all rules > ###################################################### > # NAT on Jail1 WAN IP > ipfw nat 1 config ip $wip same_ports unreg_only reset \ > redirect_port tcp 10.1.0.1:80 80 \ > redirect_port tcp 10.1.0.1:443 443 \ > redirect_port tcp 10.1.0.2:65432 65432 \ > redirect_port tcp 10.1.0.3:65444 65444 \ > redirect_port tcp 10.1.0.3:65333 65333 >=20 > # Jail1 Network - allow all traffic > $cmd 10 allow ip from any to any via $j1if > ###################################################### > # Allow all traffic on Loopback > $cmd 999 allow ip from any to any via lo0 >=20 > # NAT Rule for incoming packets on WAN IP > $cmd 1000 nat 1 ip4 from any to any in via $wif >=20 > # Check stage table > $cmd 2000 check-state > ###################################################### > # HOST ONLY > # Ping > $cmd 2100 allow icmp from $wip to any out $ks > $cmd 2101 allow icmp from any to $wip in $ks > # DNS > $cmd 2102 allow tcp from $wip to any 53 out $sks > $cmd 2103 allow udp from $wip to any 53 out $ks > # Ports > $cmd 2104 allow tcp from $wip to any 80 out $ks > $cmd 2105 allow tcp from $wip to any 433 out $ks > # SSH > $cmd 2106 allow tcp from $wip to any 22 out $ks > $cmd 2107 allow tcp from $wip to any 65222 out $ks > $cmd 2108 allow tcp from any to $wip 65222 in $ks > # OpenNTP > $cmd 2109 allow udp from $wip to any 123 out $ks > ###################################################### > # Jails =20 > # Out > $cmd 3004 $skip ip4 from any to any out xmit $wif $ks > # In > $cmd 3000 $skip tcp from any to any dst-port 80 in via $wif $sks > $cmd 3001 $skip tcp from any to any dst-port 443 in via $wif $sks > $cmd 3002 $skip tcp from any to any dst-port 65444 in via $wif $sks > $cmd 3003 $skip tcp from any to any dst-port 65432 in via $wif $sks > $cmd 3004 $skip tcp from any to any dst-port 65333 in via $wif $sks > ###################################################### >=20 > # jail.example.com > . /usr/jails/jail.example.com/ipfw.rules = >=20 > $cmd 30000 nat 1 ip from $jnet to any out >=20 > ###################################################### > # Deny Remainder and Log > $cmd deny log all from any to any > ###################################################### >=20 > = **************************************************************************= **************************************************************************= ************************* >=20 > Regards, >=20 > Nathan >=20 From owner-freebsd-ipfw@freebsd.org Fri Sep 4 14:28:19 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CF5289C9715 for ; Fri, 4 Sep 2015 14:28:19 +0000 (UTC) (envelope-from s.tyshchenko@identika.pro) Received: from scale222.ru (scale222.ru [51.254.99.22]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7E7FCA8 for ; Fri, 4 Sep 2015 14:28:19 +0000 (UTC) (envelope-from s.tyshchenko@identika.pro) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=scale222.ru; s=default; h=Content-Type:List-Unsubscribe:Message-ID:Sender:From:Date:MIME-Version:Subject:To; bh=xBLH1LKXQ30CQmhrT21aLlSu1GMFomEOC3/YCLuUrN0=; b=hX0u8fqzV2nJtU5v3HGjvy6ttVTSKD24kBqQ7YDnJ3u8MvbW25FijiWU1xKX9i0h1XDmOlp8S7k2Dc8Oj80080r0FZTE13dMt8Uey8id1+UUaebBGV0DkYJAThJ7V1rmA/ePkAa69mv14u4JB44HfQBp+Q5lWWcRNk+fqiZPFVs=; Received: from root by scale222.ru with local (Exim 4.80) (envelope-from ) id 1ZXryT-0003Da-Jq for freebsd-ipfw@freebsd.org; Fri, 04 Sep 2015 16:28:17 +0200 To: freebsd-ipfw@freebsd.org Subject: Plastic ProductS MIME-Version: 1.0 Date: Fri, 4 Sep 2015 16:28:17 +0200 From: Sergey Tyshchenko Sender: s.tyshchenko@identika.pro Message-ID: <243146758.27121@scale222.ru> X-Priority: 3 X-Mailer: scale222.ru mailer. Ver. 1.1. Precedence: bulk Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Sep 2015 14:28:19 -0000 TWFudWZhY3R1cmUgb2YgwqBwcmludGVkIHByb2R1Y3RzIGZyb20gQUJTIHBsYXN0aWMsIGFjcnls aWMsIFBFVCBtZXRob2Qgb2YgdmFjdXVtIGZvcm1pbmcuIFNlcmllcyBwcm9kdWN0aW9uIG9mIExl dHRlcnMsIHNpZ25zLCBsaWdodCBib3hlcyAobGlnaHRib3gpLCBQT1MgbWF0ZXJpYWwgZm9yIHJl dGFpbCBjaGFpbnMuRXhhbXBsZXMgb2Ygb3VyIHdvcms6wqBodHRwOi8vaWRlbnRpa2EucHJvL2Nv dW50ZXJfbGluay9wcmVzZW50YXRpb25fZW4ucGRm4oCLDQoJCQkJCQkJCQkJCQkJCQkJCQkJDQoJ CQkJCQkJCQkJCQkJCQkJCQkJDQoJCQkJCQkJCQkJCQkJCQkJCQkJCQ0KCQkJCQkJCQkJCQkJCQkJ CQkJCQkNCgkJCQkJCQkJCQkJCQkJCQkJCQkJDQoJCQkJCQkJCQkJCQkJCQkJCQkJCQ0KCQkJCQkJ CQkJCQkJCQkJCQkJCQkNCgkJCQkJCQkJCQkJCQkJCQkNCgkJCQkJCQkJCQkJCQkJCQkNCgkJCQkJ CQkJCQkJCQkJCQkNCgkJCQkJCQkJCQkJCQkJCQkNCgkJCQkJCQkJCQkJCQkJCQkJDQoJCQkJCQkJ CQkJCQkJCQkJCQkNCgkJCQkJCQkJCQkJCQkJCQkJCQkNCgkJCQkJCQkJCQkJCQkJCQkJCQkJU2Vy Z2V5IFR5c2hjaGVua29DRU8gfMKgSURFTlRJS0EuUFJPVmliZXI6wqArMzgwNTA1NTY2OTY1wqB8 IFdoYXRzQXBwOsKgKzM4MDUwNTU2Njk2NVNreXBlOiB0LnNlcmdleS5tcy50eXNoY2hlbmtvQGlk ZW50aWthLnBybyB8wqBpZGVudGlrYS5wcm8wMzA0MCB8IEdvbG9zaWl2c2t5aSBBdmUuIDcwIHwg b2ZmaWNlIDUwMiB8IEtpZXbCoA==