Date: Sat, 22 Aug 2015 07:03:48 +0300 From: wishmaster <artemrts@ukr.net> To: Nathan Aherne <nathan@reddog.com.au> Cc: freebsd-ipfw@freebsd.org Subject: Re: IPFW and in kernel nat Message-ID: <1440215094.231702740.aw1gnxg5@frv34.fwdcdn.com> In-Reply-To: <C513D7E5-D890-4230-A4EE-B94180444396@reddog.com.au> References: <C513D7E5-D890-4230-A4EE-B94180444396@reddog.com.au>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi,
--- Original message ---
From: "Nathan Aherne" <nathan@reddog.com.au>
Date: 22 August 2015, 06:28:51
Hi Everyone,
First time message to this list, so I am sorry if I do something against the rules.
I have posted this questions on the FreeBSD forums in two different places but have not had a single response in several days, so thought I might get more success here. I have spent many days solely reading about IPFW and in kernel nat in the hopes that I would be able to get what I want to achieve working without having to ask for help. I have found it extremely hard to find what I would think is a regular use use case for IPFW and nat. There are examples but there are just so many that are bad or hard to follow, including the handbook and IPFW that its extremely easy to get confused, as I have. Because of this, I will be posting my eventual solution back to the forums in the hopes that it stops someone from wasting days.
I have a single wan interface (bce0) that has two public IPs attached to it. I would like to use one of these IPs for the host (wanip1) and jails and another for a jail (wanip2) that requires a public IP. I have a cloned lo0 interface, lo1 with a subnet of 10.1.0.0/24 which all the jails reside on. I would like to forward ports 80,443 coming in on wanip1 to the jail proxy, which then forwards the traffic off to the correct jail. I would like for the host server to be able to have its own firewall and also receive traffic on port 65222. I have a couple of other jails that require some other ports as you will see in my ipfw.rules script. I would also like each jail to be able to connect to the internet.
I would like the jails to be able to have their own firewalls, which I currently allow by including a firewall script from within each jail.
I have net.inet.ip.fw.one_pass=0 set in /etc/sysctl.conf
Below is my ipfw.rules script. I would super appreciate it if someone could show me where I have gone wrong.
*****************************************************************************************************************************************************************************
#!/bin/sh
######################################################
# Configuration
wif="bce0" # WAN interface
wip="119.111.111.111" # WAN IP
j1if="lo1" # Jails Interface
j1net="10.1.0.0/24" # Jails Network
skip="skipto 30000"
jcmd="ipfw -q add 10000” # Jails rules are inserted here
######################################################
# IPFW variables
cmd="ipfw -q add"
ks="keep-state"
sks="setup keep-state"
######################################################
ipfw -q -f flush # Flush all rules
######################################################
# NAT on Jail1 WAN IP
ipfw nat 1 config ip $wip same_ports unreg_only reset \
redirect_port tcp 10.1.0.1:80 80 \
redirect_port tcp 10.1.0.1:443 443 \
redirect_port tcp 10.1.0.2:65432 65432 \
redirect_port tcp 10.1.0.3:65444 65444 \
redirect_port tcp 10.1.0.3:65333 65333
# Jail1 Network - allow all traffic
$cmd 10 allow ip from any to any via $j1if
######################################################
# Allow all traffic on Loopback
$cmd 999 allow ip from any to any via lo0
# NAT Rule for incoming packets on WAN IP
$cmd 1000 nat 1 ip4 from any to any in via $wif
# Check stage table
$cmd 2000 check-state
######################################################
# HOST ONLY
# Ping
$cmd 2100 allow icmp from $wip to any out $ks
$cmd 2101 allow icmp from any to $wip in $ks
# DNS
$cmd 2102 allow tcp from $wip to any 53 out $sks
$cmd 2103 allow udp from $wip to any 53 out $ks
# Ports
$cmd 2104 allow tcp from $wip to any 80 out $ks
$cmd 2105 allow tcp from $wip to any 433 out $ks
# SSH
$cmd 2106 allow tcp from $wip to any 22 out $ks
$cmd 2107 allow tcp from $wip to any 65222 out $ks
$cmd 2108 allow tcp from any to $wip 65222 in $ks
# OpenNTP
$cmd 2109 allow udp from $wip to any 123 out $ks
######################################################
# Jails
# Out
$cmd 3004 $skip ip4 from any to any out xmit $wif $ks
# In
$cmd 3000 $skip tcp from any to any dst-port 80 in via $wif $sks
$cmd 3001 $skip tcp from any to any dst-port 443 in via $wif $sks
$cmd 3002 $skip tcp from any to any dst-port 65444 in via $wif $sks
$cmd 3003 $skip tcp from any to any dst-port 65432 in via $wif $sks
$cmd 3004 $skip tcp from any to any dst-port 65333 in via $wif $sks
######################################################
# jail.example.com
. /usr/jails/jail.example.com/ipfw.rules
$cmd 30000 nat 1 ip from $jnet to any out
######################################################
# Deny Remainder and Log
$cmd deny log all from any to any
#####################################################
As for me, ipfw is quite complex to configure in not trivial configurations where there are many interfaces. So my advise, you should use per-interface ACL and VIMAGE for jails for better traffic control. Small example below from the real machine.
# tables with interfaces, in
ipfw -fq table 10 flush
ipfw table 10 add nfe0 3000
ipfw table 10 add ue0 3500
ipfw table 10 add lo0 4000
ipfw table 10 add sk0 5000
ipfw table 10 add epair1a 6000
ipfw table 10 add epair2a 6000
ipfw table 10 add epair3a 6000
ipfw table 10 add epair4a 6000
ipfw table 10 add epair999a 6000
ipfw table 10 add tun1 6100
# out
ipfw -fq table 11 flush
ipfw table 11 add nfe0 13000
ipfw table 11 add ue0 13500
ipfw table 11 add lo0 14000
ipfw table 11 add sk0 15000
ipfw table 11 add epair1a 16000
ipfw table 11 add epair2a 16000
ipfw table 11 add epair3a 16000
ipfw table 11 add epair4a 16000
ipfw table 11 add epair999a 16000
ipfw table 11 add tun1 16100
#nat rules .....
$cmd 100 skipto tablearg log all from any to any in recv "table(10)"
$cmd 110 skipto tablearg log all from any to any out xmit "table(11)"
#rules for interfaces
# rl0
$cmd 1000 allow log ip4 from any to any
$cmd 1099 deny log all from any to any
$cmd 11000 allow log ip4 from any to any
$cmd 11099 deny log all from any to any
# epair[0-9]a in (from jail)
$cmd 6000 nat 2 log ip4 from "table(12)" 80,81,443,5555 to me
$cmd 6001 allow log ip4 from "table(12)" 80,81,443,5555 to $nonroute 1025-65535
$cmd 6003 fwd 127.0.0.1,25 log ip4 from any to me 25
$cmd 6010 check-state log
$cmd 6020 allow log ip4 from 192.168.254.10 to not $nonroute $ks
$cmd 6030 allow log ip4 from 192.168.254.2,192.168.254.254 to not $nonroute $ks # allows requests to wild world from basejail & j1 only!
$cmd 6099 deny log all from any to any
# epair[0-9]a out (to jail)
$cmd 16000 nat 2 log ip4 from $nonroute 1025-65535 to "table(12)" 80,81,443,5555
$cmd 16010 allow log ip4 from me to "table(12)" $ks # for www redirect ext_if -> jail_if and for $lan -> www jail
$cmd 16090 check-state log
$cmd 16099 deny log all from any to any
### IN ext_if sk0
$cmd 5000 call 30000 log all from any to any
$cmd 5010 nat 1 log ip4 from any to me in
$cmd 5011 call 25000 log all from any to any
$cmd 5020 check-state log
$cmd 5100 allow log tcp from any to me 10001 setup limit src-addr 5
$cmd 5122 allow log tcp from any to me 25 setup limit src-addr 100
$cmd 5127 allow log tcp from any to me 2112 setup limit src-addr 10
$cmd 5128 allow log tcp from any to me 49152-65535 limit src-addr 10
$cmd 5150 allow log udp from any to me 11944 limit src-addr 5
$cmd 5152 allow log ip4 from any to me 67,68
$cmd 5200 allow log tcp from any to "table(12)" 80,81,443,5555 setup limit src-addr 20
$cmd 5210 allow log tcp from any to 192.168.10.2 55551 setup limit src-addr 100 # my torrent
$cmd 5211 allow log udp from any to 192.168.10.2 55551 limit src-addr 100 # my torrent
$cmd 5215 allow log udp from any to 192.168.254.10 11945 limit src-addr 2
$cmd 5310 allow log icmp from any to any icmptypes 0,3,4,8,11
$cmd 5999 deny log all from any to any
...
and so on
.....
I massively use stateful feature and call subrequests. Hope this helps.
Cheers,
Vitaliy
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1440215094.231702740.aw1gnxg5>