From owner-freebsd-jail@freebsd.org Tue Mar 12 19:47:05 2019 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9E6D3153C630 for ; Tue, 12 Mar 2019 19:47:05 +0000 (UTC) (envelope-from jamie@gritton.org) Received: from gritton.org (gritton.org [199.192.165.131]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8A8E082E3E for ; Tue, 12 Mar 2019 19:47:04 +0000 (UTC) (envelope-from jamie@gritton.org) Received: from gritton.org ([127.0.0.131]) by gritton.org (8.15.2/8.15.2) with ESMTP id x2CJJLIs012906; Tue, 12 Mar 2019 13:19:21 -0600 (MDT) (envelope-from jamie@gritton.org) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Tue, 12 Mar 2019 13:19:21 -0600 From: James Gritton To: freebsd-jail@freebsd.org Subject: Re: exec.fib and a jail in two subnets In-Reply-To: References: Message-ID: <6a245a1f51270c71d1da07c55ef51113@gritton.org> X-Sender: jamie@gritton.org User-Agent: Roundcube Webmail/1.3.8 X-Greylist: inspected by milter-greylist-4.6.2 (gritton.org [127.0.0.131]); Tue, 12 Mar 2019 13:19:22 -0600 (MDT) for IP:'127.0.0.131' DOMAIN:'[127.0.0.131]' HELO:'gritton.org' FROM:'jamie@gritton.org' RCPT:'' X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (gritton.org [127.0.0.131]); Tue, 12 Mar 2019 13:19:22 -0600 (MDT) X-Rspamd-Queue-Id: 8A8E082E3E X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of jamie@gritton.org designates 199.192.165.131 as permitted sender) smtp.mailfrom=jamie@gritton.org X-Spamd-Result: default: False [-0.90 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.05)[-0.051,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:199.192.165.128/28]; NEURAL_HAM_LONG(-0.86)[-0.856,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[gritton.org]; NEURAL_SPAM_SHORT(0.33)[0.333,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[alt1.aspmx.l.google.com,aspmx.l.google.com,aspmx5.googlemail.com,alt2.aspmx.l.google.com,aspmx2.googlemail.com]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(-0.01)[country: US(-0.07)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:30247, ipnet:199.192.164.0/22, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Mar 2019 19:47:05 -0000 On 2019-03-10 13:40, Grzegorz Junka wrote: > Hi, > > I am not sure if this question fits better to net or jail list so > please delete one crosspost when replying. > > I have two routers in separate subnets (say 10.0.0.0/16 and > 172.16.0.0/16). I have enabled multiple fibs on the host and I am > trying to setup a jail so that packets from one router are returned to > the same router. The second subnet is configured like this: > > setfib 1 route add -net 172.16.0.0/16 -iface lagg0 > setfib 1 route add default 172.16.0.1 > > When the jail configuration is (differences in red): > > mta { >   exec.fib=1; >   ip4.addr = 172.16.0.2; >   interface = lagg0; > } > > router 172.16.0.1 is able to send to and receive packets from the jail > as expected. > > When the jail configuration is: > > mta { >   ip4.addr = 10.0.0.2,172.16.0.2; >   interface = lagg0; > } > > then router 10.0.0.1 is also able to send and receive packets from the > jail as expected. > > However, when the configuration is: > > mta { > exec.fib=1; >   ip4.addr = 10.0.0.2,172.16.0.2; >   interface = lagg0; > } > > then router 172.16.0.1 is no longer able to receive a response from > the jail. The router's event log shows entry similar to the following > two about 2 minutes apart: > > IN: ACCEPT [54] Connection opened (Port Forwarding: TCP > [172.16.0.2]:80 <-​-​> [212.159.95.213]:80 -​ -​ -​ > [111.202.101.2]:34172 CLOSED/SYN_SENT ppp3 NAPT) > IN: ACCEPT [57] Connection closed (Port Forwarding: TCP > [172.16.0.2]:80 <-​-​> [212.159.95.213]:80 -​ -​ -​ > [111.202.101.2]:34172 CLOSED/SYN_SENT ppp3 NAPT) > > My question is why the 10.0.0.1 router is able to communicate with the > jail in the second configuration but 172.16.0.1 is not able to > communicate with the jail in the third configuration. Is it because of > order of IPs in ip4.addr? > > When the jail is started jls shows only the first IP from either of > the configuration list above (i.e. 10.0.0.2 even if exec.fib is set to > 1). So my guess is that the first IP is somehow a default IP? > > Then my additional question is if it's possible for a jail to be in > two subnets at the same time, i.e. so that when the jail responds to a > packet received from router 10.0.0.1 it sends it to the default route > from fib0 and when it responds to a packet received from 172.16.0.1 it > sends it to the default route from fib1. What exec.fib should be in > such a case? > > Any help would be greatly appreciated. Thanks! You're correct in your assumption that a jail's first IP address is its default: in the absence of binding a particular address for an outgoing connection, the first-listed address will be used. So then the problem with the third jail is you have a packing being sent from 10.0.0.2 with only the routing table that doesn't include 10.0/16. I can't say exactly why your second example *does* work, but at least from the jail side it has a default address that's reachable in its routing table. I'm thinking you're saying that the second jail works not only with 10.0 but also with 172.16 (it's the 172.16 part I'm unsure about). To answer your last question: sure, a jail can be in two subnets - but it will still use its first address by default for any outbound packets. Note that the FIB associated with the jail isn't *really* associated with the jail, but with the processes jail(8) starts for it - the reason for the "exec" in "exec.fib". You're still free to call setfib from inside the jail to access a different table. I haven't tried using two different routing tables in one jail at the same time; the closest I've come is one jail that routed on the non-default network. Outside of the jail world, I believe multiple routing tables implies multiple instances of servers, and that would be the same for inside a jail. Your router log shows port 80, so that would imply two different apache (or whatever) processes running the jail, each pointing to its own address, and rung under its own routing table. - Jamie