Date: Thu, 25 Nov 2010 14:50:22 -0600 From: Brandon Gooch <jamesbrandongooch@gmail.com> To: Kevin Mai <kma@mrecic.gov.ar> Cc: "freebsd-current@freebsd.org" <freebsd-current@freebsd.org> Subject: Re: Issues with Jails/Routes/FIBs Message-ID: <739DC3FD-84F4-4ECC-AD44-C9328B206884@gmail.com> In-Reply-To: <1832177924.3260.1290713938422.JavaMail.root@mrelmx10.mrec.ar> References: <1832177924.3260.1290713938422.JavaMail.root@mrelmx10.mrec.ar>
next in thread | previous in thread | raw e-mail | index | archive | help
On Nov 25, 2010, at 1:38 PM, Kevin Mai <kma@mrecic.gov.ar> wrote: > Hi folks!=20 >=20 > I'm facing an issue here while trying to define separate routing tables fo= r each jail and host.=20 >=20 > Let me show you briefly how it's done:=20 >=20 > The server has 3 physical NICs, each one connected to a different network (= say, public network A, public network B, and LAN).=20 >=20 > Currently, the default gateway is set to be the LAN gateway, even though t= he two jails can see their own public network subnet.=20 >=20 > Internet:=20 > Destination Gateway Flags Refs Use Netif Expire=20 > default 172.16.2.1 UGS 1 3935 bce2=20 > 127.0.0.1 link#5 UH 0 0 lo0=20 > 172.16.2.0/24 link#3 U 0 0 bce2=20 > 172.16.2.127 link#3 UHS 0 0 lo0=20 > 100.16.97.0/24 link#1 U 0 0 bce0=20 > 100.16.97.5 link#1 UHS 0 0 lo0=20 > 100.16.98.0/24 link#2 U 0 0 bce1=20 > 100.16.98.5 link#2 UHS 0 0 lo0=20 >=20 > 100.16.97.0/24 and 100.16.98.0/24 are the two public networks and 172.16.2= .0/24 is the LAN.=20 >=20 > I have already tried removing devfs rules from the jails, setting securele= vel to -1 but I'm still out of luck..=20 >=20 > I know setfib can define alternate routing tables, and I even created a de= fault gateway for two fibs, 1 & 2:=20 >=20 > [root@mrefns09 ~]# setfib 2 netstat -rn=20 > Routing tables=20 >=20 > Internet:=20 > Destination Gateway Flags Refs Use Netif Expire=20 > default 100.16.98.100 UGS 14 906 bce1=20 > 127.0.0.1 link#5 UH 0 0 lo0=20 > 172.16.2.0/24 link#3 U 0 0 bce2=20 > 100.16.97.0/24 link#1 U 0 39 bce0=20 > 100.16.98.0/24 link#2 U 0 0 bce1=20 >=20 > [root@mrefns09 ~]# setfib 1 netstat -rn=20 > Routing tables=20 >=20 > Internet:=20 > Destination Gateway Flags Refs Use Netif Expire=20 > default 100.16.97.100 UGS 0 1758 bce0=20 > 127.0.0.1 link#5 UH 0 0 lo0=20 > 172.16.2.0/24 link#3 U 0 0 bce2=20 > 100.16.97.0/24 link#1 U 0 44 bce0=20 > 100.16.98.0/24 link#2 U 0 4 bce1=20 >=20 > And i've added the proper settings in rc.conf..=20 >=20 > jail_athea97_ip=3D"100.16.97.5 netmask 255.255.255.0"=20 > jail_athea97_fib=3D1=20 >=20 >=20 > jail_athea98_ip=3D"100.16.98.5 netmask 255.255.255.0"=20 > jail_athea98_fib=3D2=20 >=20 > Am I missing something? because once I get into the jail the routing table= is the same:=20 >=20 > [root@athea97 /]# netstat -rn=20 > Routing tables=20 >=20 > Internet:=20 > Destination Gateway Flags Refs Use Netif Expire=20 > default 172.16.2.1 UGS 13 6175 bce2=20 > 127.0.0.1 link#5 UH 0 0 lo0=20 > 172.16.2.0/24 link#3 U 0 0 bce2=20 > 172.16.2.127 link#3 UHS 0 0 lo0=20 > 100.16.97.0/24 link#1 U 0 0 bce0=20 > 100.16.97.5 link#1 UHS 0 0 lo0=20 > 100.16.98.0/24 link#2 U 0 0 bce1=20 > 100.16.98.5 link#2 UHS 0 0 lo0=20 >=20 > [root@athea97 /]# setfib 1 netstat -rn=20 > Routing tables=20 >=20 > Internet:=20 > Destination Gateway Flags Refs Use Netif Expire=20 > default 100.16.97.100 UGS 15 1814 bce0=20 > 127.0.0.1 link#5 UH 0 0 lo0=20 > 172.16.2.0/24 link#3 U 0 0 bce2=20 > 100.16.97.0/24 link#1 U 0 44 bce0=20 > 100.16.98.0/24 link#2 U 0 4 bce1=20 >=20 > The other jail is acting the same way. I know that since I'm doing a jexec= , the shell will have the host's route because, but, how can I know if it's g= etting the alternate routing table?=20 >=20 > Thanks,=20 >=20 > Kevin Try ssh'ing into one of the jails from the public side. The jail should hono= r the FIB configuration from that perspective. Are things behaving as you ex= pect in the jail at that point? As you've figured out, when jexec'ing into the jail from the host machine, y= ou inherit the FIB of your current shell. I think this due to the design of FreeBSD's multiple routing tables -- and n= ot a bug :) -Brandon=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?739DC3FD-84F4-4ECC-AD44-C9328B206884>