Date: Fri, 02 Aug 2013 09:05:21 -0400 From: Fbsd8 <fbsd8@a1poweruser.com> To: Josh Beard <josh@signalboxes.net> Cc: freebsd-jail@freebsd.org Subject: Re: Starting jail breaks routing / multi-network jail Message-ID: <51FBAE91.7030205@a1poweruser.com> In-Reply-To: <CAHDrHStCng%2Bzg=_RThWysgRm5wD=DxxzJQz=%2BoZL8JwbX%2BXh7w@mail.gmail.com> References: <CAHDrHStCng%2Bzg=_RThWysgRm5wD=DxxzJQz=%2BoZL8JwbX%2BXh7w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Josh Beard wrote: > Hello, > > I posted this on forums.freebsd.org ( > http://forums.freebsd.org/showthread.php?t=41135), but figured I may have > better luck here. > > I'm trying to setup a host that will accommodate two networks for its jails > - with two NICs. > > One of this NICs (igb0) is connected to our LAN and the other (igb1) is > connected to a public WAN switch. > For the WAN side, I'll actually have two different gateways with two > completely different set of addresses due to IP exhaustion - same network, > however. > > I'm not sure if the problem I'm having is a bug, a misconfiguration, or a > limitation. Whenever starting a a test jail that has an address on the LAN > and one on the WAN, my hosts's routing gets changed and I'm unable to reach > the public address I have for the jail. > > Here's a snip of what the host /etc/rc.conf looks like (addresses > obfuscated for privacy): > ifconfig_igb0="inet 172.30.112.196 netmask 255.255.240.0" > ifconfig_igb0_alias0="inet 172.30.112.192 netmask 255.255.240.0" # (I tried > a recommended 255.255.255.255, too) > > ifconfig_igb1="inet 96.2.192.A netmask 255.255.255.240 broadcast 96.2.192.BA > " > ifconfig_igb1_alias0="inet 24.111.1.B netmask 255.255.255.240 broadcast > 24.111.1.BB" > > defaultrouter="24.111.1.BR" > > I'm using ezjail and in the jail's config, I have: > export jail_jailedhost_ip="igb0|172.30.112.192,igb1|24.111.1.a" > export jail_jailedhost_fib="1" > > Before starting the jail, I can ping any of the addresses in question. > After starting, the public addresses stop responding. > --------- > default 24.111.1.b UGS 0 4 igb1 > 24.111.1.x/28 link#3 U 0 43 igb1 > 24.111.1.a link#3 UHS 0 0 lo0 > (and the routes for the LAN) > --------- > When I start the jail, my hosts's routes change: > > --------- > default 24.111.1.b UGS 0 236 igb1 > 24.111.1.a link#3 UHS 0 0 lo0 => > 24.111.1.a/32 link#3 U 0 0 igb1 > (routes for the LAN - routes for each address /32) > --------- > The broadcast for each interface also changes to its own address (/32). > > I can "fix" this by doing this on the host system, but this isn't > desirable. If I have to, I guess I could have this executed on startup > (but cycling a jail will break the routing table again): > > service netif restart > > service routing restart > > set fib 1 route add -host 24.111.1.BR -iface igb1 > > set fib 1 route add default 24.111.1.BR > > > I'm not sure where to go from here. I've tried using setfib to take care > of this (as you see there), but the results are the same. > > > TL;DR: > > Starting a jail with a LAN and public address changes the host's routing > table and will not talk over the public network. Cycling the netif and > routing services resolves it. > > > Any insight? Anything is much appreciated. > > > Josh Let me start of by saying I an no network expert. This is my understanding of how jail works. 1. There are 2 ways to define jails, the legacy rc.d-script method where the jail description parameters are in /etc/rc.conf and the jail(8) method that finally has all the bugs fixed in 9.2 where the jail description parameters are in /etc/jail.conf. These 2 methods can not be mixed together. 2. By design normal jails defined using either method ONLY access an single NIC having a single or multiple IPv4/IPv6 ip address/addresses. 3. The only way to assign multiple NICs to a jail is by using the highly experimental vimage software that has to be compiled into the hosts kernel which limits the host to only using IPFW firewall. PF and IPF filewalls on the host with vimage will case a hang. 4. fib's are only configured on the host, it takes an boot option or the kernel has to be recompiled to increase the number of system fibs available to the host before you can assign a second one to a jail. 5. This is incorrect syntax ip="igb0|172.30.112.192,igb1|24.111.1.a" should be ip="172.30.112.192,24.111.1.a" No nic device name. Not issuing a error does not mean its correct. My jail system has 4 LAN only jails that have outbound access to the public internet and 2 public accessible jails for my web and email servers using the same public routable dynamic IPv4 IP address assigned by my ISP without the need for special host firewall port redirection. I use the qjail version 3.1 utility to admin my jail system. Due to the 9.2-BETA port freeze qjail-3.2 which adds IPv6 support has not been committed to the port system yet. The port-make-files can be downloaded from here http://sourceforge.net/projects/qjail/files/Port%20make%20files/ Good luck.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?51FBAE91.7030205>