From owner-freebsd-jail@freebsd.org Wed May 18 15:08:10 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3BE46B41711 for ; Wed, 18 May 2016 15:08:10 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [185.24.122.233]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E68AA1D8C for ; Wed, 18 May 2016 15:08:09 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from crayon2.yoonka.com (crayon2.yoonka.com [192.168.1.20]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id u4IF87RJ012284 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Wed, 18 May 2016 15:08:07 GMT (envelope-from list1@gjunka.com) Subject: Re: jails in different private subnets on the same host References: From: Grzegorz Junka To: freebsd-jail@freebsd.org Message-ID: <07d67bd5-206c-edd8-7f47-ef2b5c538e01@gjunka.com> Date: Wed, 18 May 2016 15:08:07 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.1.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 May 2016 15:08:10 -0000 On 18/05/2016 14:11, Bjoern A. Zeeb wrote: > >> On 18 May 2016, at 14:00 , Grzegorz Junka >> wrote: >> >> Is it possible to have two jails on the same host each one in a >> different private subnet, e.g. 192.168.1.0 and 10.33.1.0, and have >> routing between them working without issues? >> >> I know it's possible to run jails with IPs in those two subnets >> but it seems there is no routing and I am not sure if it's because >> I can't configure my router properly or there is a more >> fundamental problem. One issue I see is that the jail can't have a >> different default gateway than the host, and that for now is >> 192.168.1.1, but I don't see a reason why 10.33.1.0 wouldn't be >> able to use 192.168.1.1 as it's default gateway provided there is >> routing between those two subnets. > > Given they are both on the same base system host, both addresses > are connected locally and thus the kernel knows where to deliver > these packets. If that doesn’t work, there is a bug somewhere. > > If you want different default gateways then you may want to look > into using different FIBs for different jails. See route(8) and > jail(8) for parameters to set and tune. > > /bz > I can ping both jails from the main host, however when in the 10.33.1.0 jail I can't access any jail in the 192.168.1.0 network. This is what netstat -r shows: --------------------------------- root@dns1:/ # ifconfig em0: flags=8843 metric 0 mtu 1500 options=4219b ether 00:25:90:ae:e8:bc media: Ethernet autoselect (1000baseT ) status: active em1: flags=8843 metric 0 mtu 1500 options=4219b ether 00:25:90:ae:e8:bc media: Ethernet autoselect (1000baseT ) status: active lo0: flags=8049 metric 0 mtu 16384 options=600003 lagg0: flags=8843 metric 0 mtu 1500 options=4219b ether 00:25:90:ae:e8:bc inet 192.168.1.60 netmask 0xffffffff broadcast 192.168.1.60 media: Ethernet autoselect status: active laggproto lacp lagghash l2,l3,l4 laggport: em0 flags=1c laggport: em1 flags=1c root@dns1:/ # netstat -r Routing tables Internet: Destination Gateway Flags Netif Expire dns1 link#4 UHS lo0 --------------------------------- root@pjp1:/ # ifconfig em0: flags=8843 metric 0 mtu 1500 options=4219b ether 00:25:90:ae:e8:bc media: Ethernet autoselect (1000baseT ) status: active em1: flags=8843 metric 0 mtu 1500 options=4219b ether 00:25:90:ae:e8:bc media: Ethernet autoselect (1000baseT ) status: active lo0: flags=8049 metric 0 mtu 16384 options=600003 lagg0: flags=8843 metric 0 mtu 1500 options=4219b ether 00:25:90:ae:e8:bc inet 10.33.1.40 netmask 0xffffffff broadcast 10.33.1.40 media: Ethernet autoselect status: active laggproto lacp lagghash l2,l3,l4 laggport: em0 flags=1c laggport: em1 flags=1c root@pjp1:/ # netstat -r netstat: kvm not available: /dev/mem: No such file or directory Routing tables rt_tables: symbol not in namelist --------------------------------- On the main host: root@somehost:~ # netstat -r Routing tables Internet: Destination Gateway Flags Netif Expire default 192.168.1.1 UGS lagg0 pjp1.somehost.somedomain. link#4 UHS lo0 10.33.1.40/32 link#4 U lagg0 localhost link#3 UH lo0 192.168.1.0 link#4 U lagg0 somehost link#4 UHS lo0 web1.somehost.somedomain. link#4 UHS lo0 192.168.1.50/32 link#4 U lagg0 dns1.somehost.somedomain. link#4 UHS lo0 192.168.1.60/32 link#4 U lagg0 (... other jails) Internet6: Destination Gateway Flags Netif Expire :: localhost UGRS lo0 localhost link#3 UH lo0 ::ffff:0.0.0.0 localhost UGRS lo0 fe80:: localhost UGRS lo0 fe80::%lo0 link#3 U lo0 fe80::1%lo0 link#3 UHS lo0 ff01::%lo0 localhost U lo0 ff02:: localhost UGRS lo0 ff02::%lo0 localhost U lo0 --------------------------------- I would rather not set up different FIBs for different jails, unless required. First of all I would like to establish what's wrong. I just tried telnet 192.168.1.50 80 from the main host and from the 10.33.1.40 jail. From the main host it works without issues. From the jail it eventually connected after 15 or so seconds of waiting. Grzegorz