From owner-freebsd-jail@freebsd.org Sun Mar 10 19:40:30 2019 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EC46A153F57A; Sun, 10 Mar 2019 19:40:29 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [88.98.225.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id DD0E28B5FD; Sun, 10 Mar 2019 19:40:28 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from crayon2.yoonka.com (crayon2.yoonka.com [10.70.7.20]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id x2AJeKYW059964 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Sun, 10 Mar 2019 19:40:20 GMT (envelope-from list1@gjunka.com) To: freebsd-net@freebsd.org, freebsd-jail@freebsd.org From: Grzegorz Junka Subject: exec.fib and a jail in two subnets Message-ID: Date: Sun, 10 Mar 2019 19:40:20 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.3.2 MIME-Version: 1.0 Content-Language: en-GB-large X-Rspamd-Queue-Id: DD0E28B5FD X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of list1@gjunka.com designates 88.98.225.149 as permitted sender) smtp.mailfrom=list1@gjunka.com X-Spamd-Result: default: False [-6.77 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:88.98.225.149]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[gjunka.com]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE(-3.64)[ip: (-9.53), ipnet: 88.98.192.0/18(-4.77), asn: 56478(-3.81), country: GB(-0.09)]; MX_GOOD(-0.01)[gjunka.com]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.81)[-0.815,0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; ASN(0.00)[asn:56478, ipnet:88.98.192.0/18, country:GB]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Mar 2019 19:40:30 -0000 Hi, I am not sure if this question fits better to net or jail list so please delete one crosspost when replying. I have two routers in separate subnets (say 10.0.0.0/16 and 172.16.0.0/16). I have enabled multiple fibs on the host and I am trying to setup a jail so that packets from one router are returned to the same router. The second subnet is configured like this: setfib 1 route add -net 172.16.0.0/16 -iface lagg0 setfib 1 route add default 172.16.0.1 When the jail configuration is (differences in red): mta {   exec.fib=1;   ip4.addr = 172.16.0.2;   interface = lagg0; } router 172.16.0.1 is able to send to and receive packets from the jail as expected. When the jail configuration is: mta {   ip4.addr = 10.0.0.2,172.16.0.2;   interface = lagg0; } then router 10.0.0.1 is also able to send and receive packets from the jail as expected. However, when the configuration is: mta { exec.fib=1;   ip4.addr = 10.0.0.2,172.16.0.2;   interface = lagg0; } then router 172.16.0.1 is no longer able to receive a response from the jail. The router's event log shows entry similar to the following two about 2 minutes apart: IN: ACCEPT [54] Connection opened (Port Forwarding: TCP [172.16.0.2]:80 <-​-​> [212.159.95.213]:80 -​ -​ -​ [111.202.101.2]:34172 CLOSED/SYN_SENT ppp3 NAPT) IN: ACCEPT [57] Connection closed (Port Forwarding: TCP [172.16.0.2]:80 <-​-​> [212.159.95.213]:80 -​ -​ -​ [111.202.101.2]:34172 CLOSED/SYN_SENT ppp3 NAPT) My question is why the 10.0.0.1 router is able to communicate with the jail in the second configuration but 172.16.0.1 is not able to communicate with the jail in the third configuration. Is it because of order of IPs in ip4.addr? When the jail is started jls shows only the first IP from either of the configuration list above (i.e. 10.0.0.2 even if exec.fib is set to 1). So my guess is that the first IP is somehow a default IP? Then my additional question is if it's possible for a jail to be in two subnets at the same time, i.e. so that when the jail responds to a packet received from router 10.0.0.1 it sends it to the default route from fib0 and when it responds to a packet received from 172.16.0.1 it sends it to the default route from fib1. What exec.fib should be in such a case? Any help would be greatly appreciated. Thanks! GrzegorzJ