Date: Sun, 06 Apr 2014 22:06:17 -0400 From: Jason Unovitch <jason.unovitch@gmail.com> To: freebsd-net@freebsd.org, chris@nevermind.co.nz Subject: Re: freebsd-net Digest, Vol 574, Issue 9 Message-ID: <53420819.8070806@gmail.com> In-Reply-To: <mailman.68.1396699201.9931.freebsd-net@freebsd.org> References: <mailman.68.1396699201.9931.freebsd-net@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
> Date: Sat, 05 Apr 2014 15:22:39 +1300 > From: Chris Smith <chris@nevermind.co.nz> > To: freebsd-net@freebsd.org > Subject: Multihomed system with jails routing issues > Message-ID: <533F68EF.8060607@nevermind.co.nz> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Hi All, > > I have a system with 1 network interface with 2 extra VLANs off it and > I'm having some trouble getting the routing working correctly with it > and jails. > > bge0 - management - 10.71.100.0/24 > bge0.101 - LAN - 10.71.101.0/24 > bge0.103 - DMZ - 10.71.101.0/24 Do you mean .102.0/24 for the DMZ subnet? An overlap is bound to cause issues. Just so I understand, is the router side configured as a trunk port with a native vlan set for that management address? > Here's what I want to achieve... > > Host: > I want the host system to only listen on one interface, bge0. I want NO > ip addresses of the host on the vlan interfaces. The only service it > will be exposing is its sshd. The management address for this system is > 10.71.100.50. Just make sure every daemon running on the host is bound to an address. ListenAddress in /etc/ssh/sshd_config. Use '-b' flags for a bunch of parameters in rc.conf. Unfortunately, keep in mind that not everything supports being bound to a specific address. > Jails: > The system will also host a variety of jails, each with an IP either on > the LAN or DMZ. I am using ezjail to manage the jails. > > Router: > There is a router at the .254 address of every subnet that can route > between each network. > > I set up jail1 on bge0.101 with the IP 10.71.101.51. Since the host does > not have an address configured on bge0.101, I configured the jail > address as /24 instead of the default /32. > > My issues: > > * If I do not configure the jail as a /24 (e.g. /32), the LAN cannot > communicate with the jail. I can't comment on the difference between /24 and /32 here as I've only set my jails to a /32. I use DHCP on my em0 with a bunch of /32 aliases on a DMZ fib and static address on my em1 with a bunch of /32 aliases on a LAN fib. All my host services have the bind address set to the static IP on my LAN interface. > * When the jail is up and 10.71.101.51/24 is active, SSHing from the LAN > to the mgmt interface via the router fails, as the host tries to send > return traffic via the bge0.101 interface, even though traffic arrived > via the bge0 interface. It doesn't matter which interface the traffic arrived on, it's going to use the directly connected on when it has the whole routing table. Is the router using a source address of 10.71.101.254? > So I did a whole lot of research for people having these apparently > problems, and decided to try the multiple routing table/fib approach. So > I recompiled my kernel, configured fib 1 with the LAN interface route > (setfib route add 10.71.101.0/24 -iface bge0.101), set the jail fib and > set the tunable net.addr_all_fibs = 0. I still can't get this working > correctly. ezjail still seems to add the interface route to fib 0 by > default (but it won't if i run ezjail with the setfib 1 command). Here is a short version of what I do. I set the sysctl as you pointed out so that directly connected subnets don't automatically get added to every fib. echo 'net.add_addr_allfibs=0' >> /boot/loader.conf I set the alias and a static route for both the network and gateway in my rc.conf. ifconfig_em0="DHCP" ifconfig_em0_alias0="inet 192.168.102.11/32 fib 2" static_routes="fibnetwork fibdefault" route_fibnetwork="-net 192.168.102.0/24 -interface em0 -fib 2" route_fibdefault="default 192.168.102.1 -fib 2" Finally I set ezjail to use the fib. With all this I can run tcpdump on my router and see that connections from my jail behave as expected. ezjail-admin config -f 2 jail.example.com > Using FIB 1 and trying to ping hosts on the LAN gives an error like: > sendto failed: invalid argument. > > Does anybody have any best practices for doing this, or anything else I > can try? I'm happy to share/pastebin any configuration and I've tried > most things I've found on the internet. I'm using FreeBSD 10.0 with a > custom kernel for multiple routing tables. FYI, you don't need a custom kernel for this. It's set at boot time. Example for 4 fibs: echo 'net.fibs=4' >> /boot/loader.conf > Thanks in advance! > Chris. > I had to toy around with this quite a bit before I got the behaviour I was looking for. I hope this helps. Jason
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?53420819.8070806>