From owner-freebsd-jail@freebsd.org Fri Sep 29 08:33:07 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 07C3DE26F87 for ; Fri, 29 Sep 2017 08:33:07 +0000 (UTC) (envelope-from marko.cupac@mimar.rs) Received: from mail.mimar.rs (tazar.mimar.rs [193.53.106.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B585D82F94 for ; Fri, 29 Sep 2017 08:33:06 +0000 (UTC) (envelope-from marko.cupac@mimar.rs) Received: from tazar.mimar.rs (localhost [127.0.2.132]) by mail.mimar.rs (Postfix) with ESMTP id 0042B620BF12 for ; Fri, 29 Sep 2017 10:33:03 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=mimar.rs; h= content-transfer-encoding:content-type:content-type:mime-version :x-mailer:organization:message-id:subject:subject:from:from:date :date:received:received; s=mimar-0901; t=1506673979; x= 1508488380; bh=ywGM/l5+gaiQf+YLVJmEv1+ReAAmAJjT7jtnbyi1Jz0=; b=0 8EcuOUlOXHJuiPv3MrmDKyeztIji99fq8ypaRa8Mbx0O/1FQ+3iBUvb8fg6nVk3K Gmj9kii0MV9yOzxBxgisuuMvGyzlipzHI7Q68kTVeXTNl0dLDq8Pht5JAA7c4oh4 Ja5owSo3nwaoU5oSQnKWnbo88DEy9qwH2RpwnvCRrM= X-Virus-Scanned: amavisd-new at mimar.rs Received: from mail.mimar.rs ([127.0.2.132]) by tazar.mimar.rs (amavis.mimar.rs [127.0.2.132]) (amavisd-new, port 10026) with LMTP id 4xMP8Lz6hTIU for ; Fri, 29 Sep 2017 10:32:59 +0200 (CEST) Received: from efreet-freebsd.kappastar.com (nat-nat.kappastar.com [193.53.106.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: marko.cupac) by mail.mimar.rs (Postfix) with ESMTPSA id A9844620BEF8 for ; Fri, 29 Sep 2017 10:32:59 +0200 (CEST) Date: Fri, 29 Sep 2017 10:32:58 +0200 From: Marko =?UTF-8?B?Q3VwYcSH?= To: freebsd-jail@freebsd.org Subject: setfib (ez)jails and wierd routing Message-ID: <20170929103258.2f912308@efreet-freebsd.kappastar.com> Organization: Mimar X-Mailer: Claws Mail 3.15.1 (GTK+ 2.24.31; amd64-portbld-freebsd11.1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Sep 2017 08:33:07 -0000 Hi, I notice wierd routing in my setfib (ez)jails setup. I have a server with multiple NICs. setfib should ensure that LAN jails (setfib 1) can not talk to DMZ jails (setfib 2) over loopbacks, but need to go through firewalls as though they were physical boxes. pacija@warden3:~ % sudo setfib 1 netstat -rn Routing tables (fib: 1) Internet: Destination Gateway Flags Netif Expire default 10.30.19.190 UGS bce0 10.30.19.160/27 00:1c:c4:de:0a:86 US bce0 127.0.0.1 lo0 UHS lo0 127.0.1.0/24 lo1 US lo1 pacija@warden3:~ % sudo setfib 2 netstat -rn Routing tables (fib: 2) Internet: Destination Gateway Flags Netif Expire default 193.53.106.254 UGS bce1 127.0.0.1 lo0 UHS lo0 127.0.2.0/24 lo2 US lo2 193.53.106.0/24 00:1c:c4:de:0a:84 US bce1 Host has the same default route as fib 1: pacija@warden3:~ % sudo netstat -rn Routing tables Internet: Destination Gateway Flags Netif Expire default 10.30.19.190 UGS bce0 ... If I ssh from the Internet into DMZ jail, everything works as expected. But if I ping DMZ jail from the Internet, I see reply packets leaving not the interface they came from (bce1, public address space, DMZ), but another one (bce0, private address space, LAN). This is kinda understandable, because jail on fib2 does not have ICMP enabled, so it is not DMZ jail, but the host (which is in fib 0) who replies to packets via its default gateway (router on a private LAN). Is there an easy and elegant way to solve this? Like binding IP address to fib? I wouldn't like to have to fire up pf on host and meddle with reply-to rules in order to achieve this, I'd rather revert to old setup of separate physical servers for each network. Thank you in advance, --=20 Before enlightenment - chop wood, draw water. After enlightenment - chop wood, draw water. Marko Cupa=C4=87 https://www.mimar.rs/