From owner-freebsd-questions@freebsd.org Sun Aug 2 17:48:16 2020 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0E86F3A37C1; Sun, 2 Aug 2020 17:48:16 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: from mail-qt1-x844.google.com (mail-qt1-x844.google.com [IPv6:2607:f8b0:4864:20::844]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BKT5B1s5nz3Tw7; Sun, 2 Aug 2020 17:48:13 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: by mail-qt1-x844.google.com with SMTP id v22so20780226qtq.8; Sun, 02 Aug 2020 10:48:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:user-agent:mime-version:to:subject :content-transfer-encoding; bh=xwFe9guNwA1dGgL48tMXyF8rwowlvw2p6UvPFVpArMM=; b=QfMdyleWvQx1Jf8Rk9m88SCW2i4o1t1QUkCuwLVQRr45DyHs7iLAZg/ExJZPetu3nq TEQdu6PxM+ym9RplV9rR4Bgs5DBLFv1ahHJjcFSTYSKSuzucLYymUpSjHT/JYLYGga9K YGLsBCUQLbHe+nHLTDxbrVr6Kakesy+uIFaGvIQlUCRTYconMeqii4G+PV74moJ2fAz4 bl8AIralwVK/KcFJGjMw42nWobVE/pC/avwxfAWB+B7lyskUPsDtZnNKV+R1zJOxTy5V +hl9FhwZK22eWU/2hY3g7AoHTiJey8Y/cV10F3iaCGmZtY/YT1lAwdgIaGixoc+n34qB DZFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:content-transfer-encoding; bh=xwFe9guNwA1dGgL48tMXyF8rwowlvw2p6UvPFVpArMM=; b=ZnRxHF3QBd/WNv5Zlju/rcZKQ/ujMx7/RS1bDdp4qlDLpInGA/K3n9T8U1XQ8ZfZyt TEkA3n5nRglrLumQ+CSe9FdUKfLo2L80mrzNDssg+iVhO8ZyoN0MoaZgkRemPNlBLH19 tsPkMvz1qBVJMzIcjXlTY8WZeSddaln34aXS4CPk/MAMmubGZo/m1HIu0Nter1PZ8SVy zKU/kzLBQBAW0HRru0G/xWDxhB6RLbrA257fcPRYaMaTh2eiZqRxOpl4MzxuJcTYQa7W EHlqByWn5BEBd5EaJHdzCfhMNZnPBQJsoYwBNrzQccxHiWnS4VkFc+mMykddmqj7XKJE Z0aQ== X-Gm-Message-State: AOAM530IbLQwSg0Z7vW7ecuPO7T4v7X99Lr5OSudo11X2nbR51y+sUtM Nhb3ODOVngIb4lLMatw3vYAGR5MH X-Google-Smtp-Source: ABdhPJz4MyCz1Pk/dMBtybrKZqzAzVVeWYA9UcJzmdHh8K+hYYa9Ri+LKyRlEOA+h56oqkBS2hkziw== X-Received: by 2002:ac8:7383:: with SMTP id t3mr13020575qtp.160.1596390492663; Sun, 02 Aug 2020 10:48:12 -0700 (PDT) Received: from [10.0.10.8] (cpe-65-25-51-0.neo.res.rr.com. [65.25.51.0]) by smtp.googlemail.com with ESMTPSA id c205sm18018744qkg.98.2020.08.02.10.48.11 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 02 Aug 2020 10:48:11 -0700 (PDT) Message-ID: <5F26FC5B.6030706@gmail.com> Date: Sun, 02 Aug 2020 13:48:11 -0400 From: Ernie Luzar User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: "freebsd-questions@freebsd.org" , freebsd-jail@FreeBSD.org Subject: jail(8) bug with vnet & non-vnet jails running at same time? Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4BKT5B1s5nz3Tw7 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=QfMdyleW; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of luzar722@gmail.com designates 2607:f8b0:4864:20::844 as permitted sender) smtp.mailfrom=luzar722@gmail.com X-Spamd-Result: default: False [-2.46 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.53)[-0.534]; RECEIVED_SPAMHAUS_PBL(0.00)[65.25.51.0:received]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.97)[-0.971]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.96)[-0.959]; MIME_GOOD(-0.10)[text/plain]; SUBJECT_ENDS_QUESTION(1.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::844:from]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Aug 2020 17:48:16 -0000 Hello list; Please review configuration looking for something I may have missed. Hopping someone can suggest something that will change the behavior eliminating the problem. Equipment. Real hardware, 12.1 release, amd64 dual cpu. Description; non-vnet jails and vnet jails using the bridge/epair method can ping the public internet when only non-vnet jails are started at a time or when only vnet jails are started at a time. But when both non-vnet jails and vnet jails are started together then neither one can ping the public internet. The order of the jails definitions in the jail.conf file has no effect on changing what is happening. Bug description: When non-vnet jails are started their ip addresses are added to the NIC facing the public AFTER the public ip address and the non-vnet jail has access to the public internet. But when both non-vnet jails and vnet jails are started at the same time then the non-vnet jails ip addresses gets added before the public ip address of the NIC facing the public internet causing the host to lose all access to the public internet. This seems to be a jail(8) bug. It makes no difference which command method is used to start and stop the jails. Service jail onestart jailname or jail –cv jailname The following is a capture of the command sequence showing this bug. Follow the re0 NIC public ip address xx.25.51.0 in the ifconfig -a listing. Before any jails are started. /root >ifconfig -a snip ... re0: flags=8943 metric 0 mtu 1500 options=8209b ether 50:3e:aa:06:11:22 inet xx.25.51.0 netmask 0xfffff000 broadcast 255.255.255.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=29 bridge0: flags=8843 metric 0 mtu 1500 ether 02:3e:ba:a7:58:00 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: re0 flags=143 ifmaxaddr 0 port 2 priority 128 path cost 20000 groups: bridge nd6 options=1 /root >cat /etc/jail.conf # non-vnet jail zdir20 { host.hostname = "zdir20"; path = "/usr/jails/zdir20"; mount.fstab = "/usr/local/etc/fstab/zdir20"; exec.consolelog = "/var/log/zdir20.console.log"; mount.devfs; ip4.addr = 10.0.22.5; interface = "re0"; allow.raw_sockets; devfs_ruleset = "4"; exec.start = "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown"; } # vnet jail using the bridge/epair method v0jail1 { host.hostname = "v0jail1"; path = "/usr/jails/v0jail1"; mount.fstab = "/usr/local/etc/fstab/v0jail1"; exec.consolelog = "/var/log/v0jail1.console.log"; mount.devfs; devfs_ruleset = "4"; vnet = "new"; vnet.interface = "epair55b"; exec.prestart = "ifconfig epair55 create up"; exec.prestart += "ifconfig bridge0 addm epair55a"; exec.prestart += "ifconfig epair55a descr vnet-v0jail1"; exec.prestart += "ifconfig bridge0 inet 10.0.48.2 netmask 255.255.255.0 alias"; exec.start = "/bin/sh /etc/rc"; exec.start += "ifconfig epair55b inet 10.0.48.1 netmask 255.255.255.0"; exec.start += "route add default 10.0.48.2"; exec.prestop = "ifconfig epair55b -vnet v0jail1"; exec.stop = "/bin/sh /etc/rc.shutdown"; exec.poststop = "ifconfig bridge0 deletem epair55a"; exec.poststop += "sleep 2"; exec.poststop += "ifconfig epair55a destroy"; exec.poststop += "ifconfig bridge0 inet 10.0.48.2 -alias"; } /root >jls JID IP Address Hostname Path # start only the non-vnet jail /root >service jail onestart zdir20 Starting jails: zdir20. /root >jls JID IP Address Hostname Path 18 10.0.22.5 zdir20 /usr/jails/zdir20 # Take notice that the non-vnet jails ip address follows the nic’s # public ip address. /root >ifconfig -a re0: flags=8943 metric 0 mtu 1500 options=8209b ether 50:3e:aa:06:11:22 inet xx.25.51.0 netmask 0xfffff000 broadcast 255.255.255.255 inet 10.0.22.5 netmask 0xffffffff broadcast 10.0.22.5 media: Ethernet autoselect (1000baseT ) status: active nd6 options=29 bridge0: flags=8843 metric 0 mtu 1500 ether 02:3e:ba:a7:58:00 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: re0 flags=143 ifmaxaddr 0 port 2 priority 128 path cost 20000 groups: bridge nd6 options=1 # login to the non-vnet jail and ping the public /root >jexec zdir20 login -f root Last login: Sun Aug 2 11:30:40 on pts/0 FreeBSD 12.1-RELEASE-p6 GENERIC Welcome to your FreeBSD jail. zdir20 /root > zdir20 /root >ping -c 2 freebsd.org PING freebsd.org (96.47.72.84): 56 data bytes 64 bytes from 96.47.72.84: icmp_seq=0 ttl=48 time=44.426 ms 64 bytes from 96.47.72.84: icmp_seq=1 ttl=48 time=44.481 ms --- freebsd.org ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 44.426/44.453/44.481/0.027 ms zdir20 /root >exit logout # stop the non-vnet jail and show that the network is back to # starting condition. /root >service jail onestop zdir20 Stopping jails: zdir20. /root >jls JID IP Address Hostname Path /root >ifconfig -a re0: flags=8943 metric 0 mtu 1500 options=8209b ether 50:3e:aa:06:11:22 inet xx.25.51.0 netmask 0xfffff000 broadcast 255.255.255.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=29 bridge0: flags=8843 metric 0 mtu 1500 ether 02:3e:ba:a7:58:00 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: re0 flags=143 ifmaxaddr 0 port 2 priority 128 path cost 20000 groups: bridge nd6 options=1 # start only the vnet jail and see the bridge0 /root >service jail onestart v0jail1 Starting jails: v0jail1. /root >jls JID IP Address Hostname Path 19 v0jail1 /usr/jails/v0jail1 /root >ifconfig -a re0: flags=8943 metric 0 mtu 1500 options=82099 ether 50:3e:aa:06:11:22 inet xx.25.51.0 netmask 0xfffff000 broadcast 255.255.255.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=29 bridge0: flags=8843 metric 0 mtu 1500 ether 02:3e:ba:a7:58:00 inet 10.0.48.2 netmask 0xffffff00 broadcast 10.0.48.255 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: epair55a flags=143 ifmaxaddr 0 port 5 priority 128 path cost 2000 member: re0 flags=143 ifmaxaddr 0 port 2 priority 128 path cost 20000 groups: bridge nd6 options=1 epair55a: flags=8943 metric 0 mtu 1500 description: vnet-v0jail1 options=8 ether 02:eb:be:f5:15:0a inet6 fe80::eb:beff:fef5:150a%epair55a prefixlen 64 scopeid 0x5 groups: epair media: Ethernet 10Gbase-T (10Gbase-T ) status: active nd6 options=21 # login to the vnet jail and ping the public internet. /root >jexec v0jail1 login -f root Last login: Sun Aug 2 11:29:41 on pts/0 FreeBSD 12.1-RELEASE-p6 GENERIC Welcome to your FreeBSD jail. v0jail1 /root >ping -c 2 freebsd.org PING freebsd.org (96.47.72.84): 56 data bytes 64 bytes from 96.47.72.84: icmp_seq=0 ttl=47 time=46.745 ms 64 bytes from 96.47.72.84: icmp_seq=1 ttl=47 time=43.930 ms --- freebsd.org ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 43.930/45.337/46.745/1.407 ms v0jail1 /root >exit logout # close the vnet jail and return to starting condition. /root >service jail onestop v0jail1 Stopping jails: v0jail1. /root >jls JID IP Address Hostname Path /root >ifconfig -a re0: flags=8943 metric 0 mtu 1500 options=8209b ether 50:3e:aa:06:11:22 inet xx.25.51.0 netmask 0xfffff000 broadcast 255.255.255.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=29 bridge0: flags=8843 metric 0 mtu 1500 ether 02:3e:ba:a7:58:00 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: re0 flags=143 ifmaxaddr 0 port 2 priority 128 path cost 20000 groups: bridge nd6 options=1 # Start both the non-vnet jail and the vnet jail together. /root >service jail onestart Starting jails: zdir20 v0jail1. # login to the non-vnet jail and it has no public access. /root >jexec zdir20 login -f root [K Last login: Sun Aug 2 11:36:34 on pts/0 FreeBSD 12.1-RELEASE-p6 GENERIC Welcome to your FreeBSD jail. zdir20 /root >ping -c 2 freebsd.org ping: cannot resolve freebsd.org: Host name lookup failure zdir20 /root >exit logout # login to the vnet jail and it has no public access. /root >jexec v0jail1 login -f root Last login: Sun Aug 2 11:38:56 on pts/0 FreeBSD 12.1-RELEASE-p6 GENERIC Welcome to your FreeBSD jail. v0jail1 /root >ping -c 2 freebsd.org ping: cannot resolve freebsd.org: Host name lookup failure v0jail1 /root >exit logout /root >jls JID IP Address Hostname Path 20 10.0.22.5 zdir20 /usr/jails/zdir20 21 v0jail1 /usr/jails/v0jail1 # Here is the bug. See that the non-vnet jail ip address comes before the # public address causing the host to lose access to the public internet. /root >ifconfig -a re0: flags=8943 metric 0 mtu 1500 options=82099 ether 50:3e:aa:06:11:22 inet 10.0.22.5 netmask 0xffffffff broadcast 10.0.22.5 inet xx.25.51.0 netmask 0xfffff000 broadcast 255.255.255.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=29 bridge0: flags=8843 metric 0 mtu 1500 ether 02:3e:ba:a7:58:00 inet 10.0.48.2 netmask 0xffffff00 broadcast 10.0.48.255 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: epair55a flags=143 ifmaxaddr 0 port 5 priority 128 path cost 2000 member: re0 flags=143 ifmaxaddr 0 port 2 priority 128 path cost 20000 groups: bridge nd6 options=1 epair55a: flags=8943 metric 0 mtu 1500 description: vnet-v0jail1 options=8 ether 02:77:b8:5f:e4:0a inet6 fe80::77:b8ff:fe5f:e40a%epair55a prefixlen 64 scopeid 0x5 groups: epair media: Ethernet 10Gbase-T (10Gbase-T ) status: active nd6 options=21 # stop both jails and return to starting condition. /root >service jail onestop Stopping jails: zdir20 v0jail1. /root >jls JID IP Address Hostname Path /root >ifconfig -a re0: flags=8943 metric 0 mtu 1500 options=8209b ether 50:3e:aa:06:11:22 inet xx.25.51.0 netmask 0xfffff000 broadcast 255.255.255.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=29 bridge0: flags=8843 metric 0 mtu 1500 ether 02:3e:ba:a7:58:00 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: re0 flags=143 ifmaxaddr 0 port 2 priority 128 path cost 20000 groups: bridge nd6 options=1