From owner-freebsd-questions@freebsd.org Thu Dec 10 09:30:29 2020 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 09EAF47AA5F for ; Thu, 10 Dec 2020 09:30:29 +0000 (UTC) (envelope-from jacques+freebsd@foucry.net) Received: from mail.foucry.net (fournil.foucry.net [95.217.83.231]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Cs7tr03WXz3PS4 for ; Thu, 10 Dec 2020 09:30:27 +0000 (UTC) (envelope-from jacques+freebsd@foucry.net) Received: from mail.foucry.net (unknown [192.168.12.17]) by mail.foucry.net (Postfix) with ESMTP id 8EC3130060 for ; Thu, 10 Dec 2020 09:30:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at foucry.net Received: from mail.foucry.net ([192.168.12.17]) by mail.foucry.net (mail.foucry.net [192.168.12.17]) (amavisd-new, port 10024) with ESMTP id uxdkOzjqdNCR for ; Thu, 10 Dec 2020 09:30:07 +0000 (UTC) Received: by mail.foucry.net (Postfix, from userid 58) id 3268E3004F; Thu, 10 Dec 2020 09:30:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=foucry.net; s=dkim; t=1607592607; bh=rhcinSAmRyjrXiG1ESlsk1PaX4Sh66FZ5U7+Dc4kjME=; h=Date:From:To:Subject; b=BhXznH2uJjvhjRYfNKF03wRaiwq7BcgD98Da7kP6kQZ3Ewr9V0dmmvvcLdNfSk1Nc TBXuV6bWiulcIOP2ULZKGWmyh3L72K56SEe8pLRxsSR2WZIC2TvF5CO5M0+NirWxg2 nVwo3M5SZCi2gAm+CYz/wcn0Gm5378okxZjNOGwE= Received: from mithril.localdomain (lfbn-dij-1-1138-109.w90-125.abo.wanadoo.fr [90.125.86.109]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mail.foucry.net (Postfix) with ESMTPSA id E8E9F3004B for ; Thu, 10 Dec 2020 09:30:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=foucry.net; s=dkim; t=1607592605; bh=rhcinSAmRyjrXiG1ESlsk1PaX4Sh66FZ5U7+Dc4kjME=; h=Date:From:To:Subject; b=dIw6SA4LCgSDwmlCPYnqjjdZkQ+l2ufvphPJdhFXFizl01unLEfm0d1dI6Ips7xpF 3KZKvdACxBBPtK+bhgYUX+uAbwwosuv7n/YTuxBcEOaHUgbC03F2ys9V3CKK25CR9D EKP0q27fSdWwADtoIRcJ/sp0grs17SLxdbTUBhUo= Received: from mithril (localhost [IPv6:::1]) by mithril.localdomain (Postfix) with ESMTP id 4993C70CE9 for ; Thu, 10 Dec 2020 10:30:04 +0100 (CET) Date: Thu, 10 Dec 2020 10:30:04 +0100 From: Jacques Foucry To: freebsd-questions@freebsd.org Subject: Jail, VNET and IPv6 Message-ID: Mail-Followup-To: freebsd-questions@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4Cs7tr03WXz3PS4 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=foucry.net header.s=dkim header.b=BhXznH2u; dkim=pass header.d=foucry.net header.s=dkim header.b=dIw6SA4L; dmarc=pass (policy=none) header.from=foucry.net; spf=pass (mx1.freebsd.org: domain of jacques@foucry.net designates 95.217.83.231 as permitted sender) smtp.mailfrom=jacques@foucry.net X-Spamd-Result: default: False [-3.50 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[95.217.83.231:from]; RCVD_COUNT_FIVE(0.00)[6]; R_DKIM_ALLOW(-0.20)[foucry.net:s=dkim]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[95.217.83.231:from:127.0.2.255]; MID_RHS_NOT_FQDN(0.50)[]; DKIM_TRACE(0.00)[foucry.net:+]; DMARC_POLICY_ALLOW(-0.50)[foucry.net,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; ARC_NA(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:24940, ipnet:95.217.0.0/16, country:DE]; TAGGED_FROM(0.00)[freebsd]; MAILMAN_DEST(0.00)[freebsd-questions] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Dec 2020 09:30:29 -0000 Hello folks, I manage on a hosted server many « clasical » jail with ip adresses as alias of em0. I would like to make a new jail, but using VNET and ipv6. All my tries failed :-( IPv4 work great but IPv6 not. em0: ptions=810099 ether b4:2e:99:6a:80:9d inet6 2a01:4f9:4a:1fd8::2 prefixlen 64 inet6 fe80::b62e:99ff:fe6a:809d%em0 prefixlen 64 scopeid 0x1 inet6 2a01:4f9:4a:1fd8::16 prefixlen 64 inet6 2a01:4f9:4a:1fd8::21 prefixlen 64 inet6 2a01:4f9:4a:1fd8::12 prefixlen 64 inet6 2a01:4f9:4a:1fd8::29 prefixlen 64 inet6 2a01:4f9:4a:1fd8::15 prefixlen 64 inet6 2a01:4f9:4a:1fd8::11 prefixlen 64 inet6 2a01:4f9:4a:1fd8::22 prefixlen 64 inet6 2a01:4f9:4a:1fd8::17 prefixlen 64 inet6 2a01:4f9:4a:1fd8::28 prefixlen 64 inet6 2a01:4f9:4a:1fd8::18 prefixlen 64 inet6 2a01:4f9:4a:1fd8::19 prefixlen 64 inet6 2a01:4f9:4a:1fd8::25 prefixlen 64 inet6 2a01:4f9:4a:1fd8::5 prefixlen 64 inet6 2a01:4f9:4a:1fd8::14 prefixlen 64 inet 95.217.83.231 netmask 0xffffffc0 broadcast 95.217.83.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=21 netstat -rn Routing tables Internet: Destination Gateway Flags Netif Expire default 95.217.83.193 UGS em0 10.0.0.0/24 link#5 U bridge0 10.0.0.1 link#5 UHS lo0 95.217.83.192/26 link#1 U em0 95.217.83.231 link#1 UHS lo0 127.0.0.1 link#2 UH lo0 127.0.1.5 link#2 UH lo0 127.0.1.11 link#2 UH lo0 127.0.1.12 link#2 UH lo0 127.0.1.14 link#2 UH lo0 127.0.1.15 link#2 UH lo0 127.0.1.16 link#2 UH lo0 127.0.1.17 link#2 UH lo0 127.0.1.18 link#2 UH lo0 127.0.1.19 link#2 UH lo0 127.0.1.21 link#2 UH lo0 127.0.1.22 link#2 UH lo0 127.0.1.25 link#2 UH lo0 127.0.1.28 link#2 UH lo0 127.0.1.29 link#2 UH lo0 127.0.12.1 link#2 UH lo0 192.168.12.1 link#4 UH lo1 192.168.12.5 link#4 UH lo1 192.168.12.11 link#4 UH lo1 192.168.12.12 link#4 UH lo1 192.168.12.14 link#4 UH lo1 192.168.12.15 link#4 UH lo1 192.168.12.16 link#4 UH lo1 192.168.12.17 link#4 UH lo1 192.168.12.18 link#4 UH lo1 192.168.12.19 link#4 UH lo1 192.168.12.21 link#4 UH lo1 192.168.12.22 link#4 UH lo1 192.168.12.25 link#4 UH lo1 192.168.12.28 link#4 UH lo1 192.168.12.29 link#4 UH lo1 Internet6: Destination Gateway Flags Netif Expire ::/96 ::1 UGRS lo0 default fe80::1%em0 UGS em0 ::1 link#2 UH lo0 ::ffff:0.0.0.0/96 ::1 UGRS lo0 2a01:4f9:4a:1fd8::/64 link#1 U em0 2a01:4f9:4a:1fd8::2 link#1 UHS lo0 2a01:4f9:4a:1fd8::5 link#1 UHS lo0 2a01:4f9:4a:1fd8::11 link#1 UHS lo0 2a01:4f9:4a:1fd8::12 link#1 UHS lo0 2a01:4f9:4a:1fd8::14 link#1 UHS lo0 2a01:4f9:4a:1fd8::15 link#1 UHS lo0 2a01:4f9:4a:1fd8::16 link#1 UHS lo0 2a01:4f9:4a:1fd8::17 link#1 UHS lo0 2a01:4f9:4a:1fd8::18 link#1 UHS lo0 2a01:4f9:4a:1fd8::19 link#1 UHS lo0 2a01:4f9:4a:1fd8::21 link#1 UHS lo0 2a01:4f9:4a:1fd8::22 link#1 UHS lo0 2a01:4f9:4a:1fd8::25 link#1 UHS lo0 2a01:4f9:4a:1fd8::28 link#1 UHS lo0 2a01:4f9:4a:1fd8::29 link#1 UHS lo0 fe80::/10 ::1 UGRS lo0 fe80::%em0/64 link#1 U em0 fe80::b62e:99ff:fe6a:809d%em0 link#1 UHS lo0 fe80::%lo0/64 link#2 U lo0 fe80::1%lo0 link#2 UHS lo0 ff02::/16 ::1 UGRS lo0 ifconfig em0: flags=8943 metric 0 mtu 1500 options=81009b ether b4:2e:99:6a:80:9d inet6 2a01:4f9:4a:1fd8::2 prefixlen 64 inet6 fe80::b62e:99ff:fe6a:809d%em0 prefixlen 64 scopeid 0x1 inet6 2a01:4f9:4a:1fd8::16 prefixlen 64 inet6 2a01:4f9:4a:1fd8::21 prefixlen 64 inet6 2a01:4f9:4a:1fd8::12 prefixlen 64 inet6 2a01:4f9:4a:1fd8::29 prefixlen 64 inet6 2a01:4f9:4a:1fd8::15 prefixlen 64 inet6 2a01:4f9:4a:1fd8::11 prefixlen 64 inet6 2a01:4f9:4a:1fd8::22 prefixlen 64 inet6 2a01:4f9:4a:1fd8::17 prefixlen 64 inet6 2a01:4f9:4a:1fd8::28 prefixlen 64 inet6 2a01:4f9:4a:1fd8::18 prefixlen 64 inet6 2a01:4f9:4a:1fd8::19 prefixlen 64 inet6 2a01:4f9:4a:1fd8::25 prefixlen 64 inet6 2a01:4f9:4a:1fd8::5 prefixlen 64 inet6 2a01:4f9:4a:1fd8::14 prefixlen 64 inet 95.217.83.231 netmask 0xffffffc0 broadcast 95.217.83.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=21 lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 inet 127.0.12.1 netmask 0xff000000 inet 127.0.1.16 netmask 0xffffffff inet 127.0.1.21 netmask 0xffffffff inet 127.0.1.12 netmask 0xffffffff inet 127.0.1.29 netmask 0xffffffff inet 127.0.1.15 netmask 0xffffffff inet 127.0.1.11 netmask 0xffffffff inet 127.0.1.22 netmask 0xffffffff inet 127.0.1.17 netmask 0xffffffff inet 127.0.1.28 netmask 0xffffffff inet 127.0.1.18 netmask 0xffffffff inet 127.0.1.19 netmask 0xffffffff inet 127.0.1.25 netmask 0xffffffff inet 127.0.1.5 netmask 0xffffffff inet 127.0.1.14 netmask 0xffffffff groups: lo nd6 options=21 pflog0: flags=0<> metric 0 mtu 33160 groups: pflog lo1: flags=8049 metric 0 mtu 16384 options=680003 inet 192.168.12.1 netmask 0xffffff00 inet 192.168.12.16 netmask 0xffffff00 inet 192.168.12.21 netmask 0xffffff00 inet 192.168.12.12 netmask 0xffffff00 inet 192.168.12.29 netmask 0xffffff00 inet 192.168.12.15 netmask 0xffffff00 inet 192.168.12.11 netmask 0xffffff00 inet 192.168.12.22 netmask 0xffffff00 inet 192.168.12.17 netmask 0xffffff00 inet 192.168.12.28 netmask 0xffffff00 inet 192.168.12.18 netmask 0xffffff00 inet 192.168.12.19 netmask 0xffffff00 inet 192.168.12.25 netmask 0xffffff00 inet 192.168.12.5 netmask 0xffffffff inet 192.168.12.14 netmask 0xffffff00 groups: lo nd6 options=29 bridge0: flags=8843 metric 0 mtu 1500 description: vnet-jail-bridge ether 02:36:b3:c1:8a:00 inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: em0 flags=143 ifmaxaddr 0 port 1 priority 128 path cost 20000 groups: bridge nd6 options=1 As you can see thereis a bridge (bridg0) with an IPv4 10.0.0.1/24. PF assume the nat fonction for this range to 10.0.010/24 the new jail IPv4. /etc/jail.conf jitsi{ $id="10"; $ipaddr="10.0.0.${id}"; $mask="255.255.255.0"; $gw="10.0.0.1"; host.hostname="${name}.exemple.net"; path="/jails/${name}"; exec.clean; exec.consolelog = "/var/log/consolelog$(name}.log"; vnet = "new"; vnet.interface = "epair${id}b"; exec.prestart = "ifconfig epair${id} create up"; exec.prestart += "ifconfig epair${id}a up descr vnet-${name}"; exec.prestart += "ifconfig bridge0 addm epair${id}a"; exec.start = "/bin/sh /etc/rc"; exec.start += "/sbin/ifconfig epair${id}b ${ipaddr} netmask ${mask} up"; exec.start += "/sbin/route add default ${gw}"; exec.start += "/sbin/ifconfig epair${id}b inet6 2a01:4f9:4a:1fd8::27"; exec.start += "route add -inet6 default 2a01:4f9:4a:1fd8::2"; exec.stop = "/bin/sh /etc/rc.shutdown"; exec.poststop = "ifconfig epair${id}b -vnet ${name}"; exec.poststop += "ifconfig bridge0 deletem epair${id}a"; exec.poststop += "sleep 2"; exec.poststop += "ifconfig epair${id}a destroy"; allow.mount.fusefs=1; mount.fstab="/etc/fstab.${name}"; devfs_ruleset="5"; allow.raw_sockets; persist; } The jail statup seems to be ok: # jail -cv jitsi jitsi: run command as root: /sbin/mount -t nullfs -o rw,late /usr/local/etc/letsencrypt /jails/jitsi/usr/local/etc/letsencrypt jitsi: run command as root: /sbin/mount -t nullfs -o rw,late /usr/local/www/certbot /jails/jitsi/usr/local/www/certbot jitsi: run command as root: /sbin/mount -t devfs -oruleset=5 . /jails/jitsi/dev jitsi: run command as root: ifconfig epair10 create up epair10a jitsi: run command as root: ifconfig epair10a up descr vnet-jitsi jitsi: run command as root: ifconfig bridge0 addm epair10a jitsi: jail_set(JAIL_CREATE) name=jitsi allow.mount=true allow.mount.devfs allow.mount.zfs=true devfs_ruleset=5 enforce_statfs=1 sysvshm=new host.hostname=jitsi.foucry.net path=/jails/jitsi vnet=new allow.mount.fusefs=true allow.raw_sockets persist jitsi: created jitsi: run command as root: /sbin/ifconfig epair10b vnet jitsi jitsi: run command in jail as root: /bin/sh /etc/rc jitsi: run command in jail as root: /sbin/ifconfig epair10b 10.0.0.10 netmask 255.255.255.0 up jitsi: run command in jail as root: /sbin/route add default 10.0.0.1 jitsi: run command in jail as root: /sbin/ifconfig epair10b inet6 2a01:4f9:4a:1fd8::27 jitsi: run command in jail as root: route add -inet6 default 2a01:4f9:4a:1fd8::2 epair10a on the host: epair10a: flags=8943 metric 0 mtu 1500 description: vnet-jitsi options=8 ether 02:dc:c8:b1:ac:0a inet6 fe80::dc:c8ff:feb1:ac0a%epair10a prefixlen 64 scopeid 0x6 groups: epair media: Ethernet 10Gbase-T (10Gbase-T ) status: active nd6 options=21 Once connected to the jail (through ssh/IPv4), ifconfig looks ok: ifconfig lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21 pflog0: flags=0<> metric 0 mtu 33160 groups: pflog epair10b: flags=8843 metric 0 mtu 1500 options=8 ether 02:dc:c8:b1:ac:0b inet6 fe80::dc:c8ff:feb1:ac0b%epair10b prefixlen 64 scopeid 0x3 inet6 2a01:4f9:4a:1fd8::27 prefixlen 64 inet 10.0.0.10 netmask 0xffffff00 broadcast 10.0.0.255 groups: epair media: Ethernet 10Gbase-T (10Gbase-T ) status: active nd6 options=8021 But form my work machine to the jail the ping666666 does not work: ping6 2a01:4f9:4a:1fd8::27 PING6(56=40+8+8 bytes) 2a01:cb10:8e64:fe00:4aa4:72ff:fe9e:65a1 --> 2a01:4f9:4a:1fd8::27 ^C --- 2a01:4f9:4a:1fd8::27 ping6 statistics --- 3 packets transmitted, 0 packets received, 100.0% packet loss Notice that a ping6 to another (old school jail) work perfectly: ping6 2a01:4f9:4a:1fd8::25 PING6(56=40+8+8 bytes) 2a01:cb10:8e64:fe00:4aa4:72ff:fe9e:65a1 --> 2a01:4f9:4a:1fd8::25 16 bytes from 2a01:4f9:4a:1fd8::25, icmp_seq=0 hlim=52 time=43.882 ms 16 bytes from 2a01:4f9:4a:1fd8::25, icmp_seq=1 hlim=52 time=43.731 ms 16 bytes from 2a01:4f9:4a:1fd8::25, icmp_seq=2 hlim=52 time=42.906 ms I must miss something, or misunderstood something… Any advices are welcome. Regards, -- Jacques Foucry