Date: Sat, 25 Jun 2016 17:17:53 -0600 From: Alan Somers <asomers@freebsd.org> To: org.freebsd.security@io7m.com Cc: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: Filtering outbound traffic for private address jails? Message-ID: <CAOtMX2gHpw-WKtFQdXcPEOc4W%2BrBjW4WDgfZuS2kefjUXmfvFA@mail.gmail.com> In-Reply-To: <20160625220137.1ed8de16@copperhead.int.arc7.info> References: <20160625220137.1ed8de16@copperhead.int.arc7.info>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Jun 25, 2016 at 4:01 PM, <org.freebsd.security@io7m.com> wrote:
> Hello.
>
> I have been searching for the best part of a day for a solution to this
> problem and quite frankly cannot believe that I've spent this long on
> something that appears to be so simple and that used to be fairly easy
> to achieve. Many years ago, I solved this problem on FreeBSD 6, but the
> way I did it there seems to no longer work on modern releases.
>
> The problem is this: I have a single public IP address. I want to run
> multiple jails.
>
> Back in the days of FreeBSD 6.*, the accepted way to do this seemed to
> be to create a new loopback device:
>
> # ifconfig lo1 create
>
> ... and then add a lot of private 127.0.0.* addresses, one per jail.
>
> Then, the real network adapter and the new loopback device were both
> added to a bridge (if_bridge). Unfortunately, I can't remember the exact
> details, but I believe that NAT was then enabled on the real interface.
> In order to filter traffic to, from, and between jails, pf rules were
> written that filtered the bridge device.
>
> This meant that jails could correctly send outbound traffic and
> receive responses (via pf states), could correctly receive specific
> inbound traffic (via rdr rules), and traffic in both directions could be
> filtered based on packets entering and leaving the bridge.
>
> However (see my other mailing list post), it seems that now with
> FreeBSD 10, you just can't add loopback devices to bridges. I can find
> no evidence of anyone online doing this, or even using the old bridge
> method that I just described! I can find one post in russian that seems
> to have the same error that I encounter, but nobody has any idea why
> it's happening.
>
> I can find dozens of blog posts describing how to set up jails on
> private IP addresses. They all follow the same pattern:
>
> 1. Create a loopback device.
> 2. Create a 127.0.0.* address on the loopback device.
> 3. Create a jail using the address you just added.
> 4. Set up pf and enable NAT between the real network adapter and
> the new loopback device.
>
> Unfortunately, at this point, you completely lose the ability to filter
> outbound jail traffic; All packets sent from a jail will obviously have
> their source address changed to that of the host and therefore it's not
> possible to distinguish between outbound host traffic and outbound jail
> traffic in filter rules.
>
> As far as I can tell, people are just not filtering outbound traffic,
> which seems insane!
>
> Is it really impossible to do this with FreeBSD 10?
>
> M
I'm filtering outbound traffic, but I'm not using NAT on the jail
host. Instead, I have a dedicated router doing NAT, and my jail host
has multiple IP addresses. At first I tried using traditional
shared-address jails, but the firewall rules quickly got very
complicated, especially for dealing with IPv6 and other non-IPv4
traffic. So I switched to using vimage jails. I use iocage to setup
my jails, and pf to filter them. A simplified version of my pf.conf
follows:
www_services = "{ http, https, 8080 }"
host_iface = "em0"
dmz_iface = "em1"
www_jail_iface = "vnet0:1"
www_ip = "192.168.0.40"
set state-policy if-bound
scrub in
block in all
block out all
pass in on $host_iface
pass out on $host_iface
set skip on lo0
# Allow all traffic to the DMZ. Filtering happens on individual vnet
# interfaces
pass in on $dmz_iface
pass out on $dmz_iface
# Put the www jail in a DMZ. Don't allow outgoing traffic from it except for
# the webserver
pass out on $www_jail_iface proto tcp to $www_ip port $www_services keep state
# Uncomment next line to allow outbound traffice from www jail
# pass in on $www_jail_iface
-Alan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2gHpw-WKtFQdXcPEOc4W%2BrBjW4WDgfZuS2kefjUXmfvFA>