Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 10 Dec 2020 10:30:04 +0100
From:      Jacques Foucry <jacques+freebsd@foucry.net>
To:        freebsd-questions@freebsd.org
Subject:   Jail, VNET and IPv6
Message-ID:  <X9HqnHRReRE34Nw5@mithril>

Next in thread | Raw E-Mail | Index | Archive | Help
Hello folks,

I manage on a hosted server many « clasical » jail with ip adresses as alias of
em0.

I would like to make a new jail, but using VNET and ipv6. All my tries failed
:-( IPv4 work great but IPv6 not.

em0:

ptions=810099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
	ether b4:2e:99:6a:80:9d
	inet6 2a01:4f9:4a:1fd8::2 prefixlen 64
	inet6 fe80::b62e:99ff:fe6a:809d%em0 prefixlen 64 scopeid 0x1
	inet6 2a01:4f9:4a:1fd8::16 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::21 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::12 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::29 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::15 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::11 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::22 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::17 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::28 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::18 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::19 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::25 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::5 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::14 prefixlen 64
	inet 95.217.83.231 netmask 0xffffffc0 broadcast 95.217.83.255
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>


netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            95.217.83.193      UGS         em0
10.0.0.0/24        link#5             U       bridge0
10.0.0.1           link#5             UHS         lo0
95.217.83.192/26   link#1             U           em0
95.217.83.231      link#1             UHS         lo0
127.0.0.1          link#2             UH          lo0
127.0.1.5          link#2             UH          lo0
127.0.1.11         link#2             UH          lo0
127.0.1.12         link#2             UH          lo0
127.0.1.14         link#2             UH          lo0
127.0.1.15         link#2             UH          lo0
127.0.1.16         link#2             UH          lo0
127.0.1.17         link#2             UH          lo0
127.0.1.18         link#2             UH          lo0
127.0.1.19         link#2             UH          lo0
127.0.1.21         link#2             UH          lo0
127.0.1.22         link#2             UH          lo0
127.0.1.25         link#2             UH          lo0
127.0.1.28         link#2             UH          lo0
127.0.1.29         link#2             UH          lo0
127.0.12.1         link#2             UH          lo0
192.168.12.1       link#4             UH          lo1
192.168.12.5       link#4             UH          lo1
192.168.12.11      link#4             UH          lo1
192.168.12.12      link#4             UH          lo1
192.168.12.14      link#4             UH          lo1
192.168.12.15      link#4             UH          lo1
192.168.12.16      link#4             UH          lo1
192.168.12.17      link#4             UH          lo1
192.168.12.18      link#4             UH          lo1
192.168.12.19      link#4             UH          lo1
192.168.12.21      link#4             UH          lo1
192.168.12.22      link#4             UH          lo1
192.168.12.25      link#4             UH          lo1
192.168.12.28      link#4             UH          lo1
192.168.12.29      link#4             UH          lo1

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
default                           fe80::1%em0                   UGS         em0
::1                               link#2                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
2a01:4f9:4a:1fd8::/64             link#1                        U           em0
2a01:4f9:4a:1fd8::2               link#1                        UHS         lo0
2a01:4f9:4a:1fd8::5               link#1                        UHS         lo0
2a01:4f9:4a:1fd8::11              link#1                        UHS         lo0
2a01:4f9:4a:1fd8::12              link#1                        UHS         lo0
2a01:4f9:4a:1fd8::14              link#1                        UHS         lo0
2a01:4f9:4a:1fd8::15              link#1                        UHS         lo0
2a01:4f9:4a:1fd8::16              link#1                        UHS         lo0
2a01:4f9:4a:1fd8::17              link#1                        UHS         lo0
2a01:4f9:4a:1fd8::18              link#1                        UHS         lo0
2a01:4f9:4a:1fd8::19              link#1                        UHS         lo0
2a01:4f9:4a:1fd8::21              link#1                        UHS         lo0
2a01:4f9:4a:1fd8::22              link#1                        UHS         lo0
2a01:4f9:4a:1fd8::25              link#1                        UHS         lo0
2a01:4f9:4a:1fd8::28              link#1                        UHS         lo0
2a01:4f9:4a:1fd8::29              link#1                        UHS         lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%em0/64                     link#1                        U           em0
fe80::b62e:99ff:fe6a:809d%em0     link#1                        UHS         lo0
fe80::%lo0/64                     link#2                        U           lo0
fe80::1%lo0                       link#2                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0


ifconfig
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=81009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
	ether b4:2e:99:6a:80:9d
	inet6 2a01:4f9:4a:1fd8::2 prefixlen 64
	inet6 fe80::b62e:99ff:fe6a:809d%em0 prefixlen 64 scopeid 0x1
	inet6 2a01:4f9:4a:1fd8::16 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::21 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::12 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::29 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::15 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::11 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::22 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::17 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::28 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::18 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::19 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::25 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::5 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::14 prefixlen 64
	inet 95.217.83.231 netmask 0xffffffc0 broadcast 95.217.83.255
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
	inet 127.0.0.1 netmask 0xff000000
	inet 127.0.12.1 netmask 0xff000000
	inet 127.0.1.16 netmask 0xffffffff
	inet 127.0.1.21 netmask 0xffffffff
	inet 127.0.1.12 netmask 0xffffffff
	inet 127.0.1.29 netmask 0xffffffff
	inet 127.0.1.15 netmask 0xffffffff
	inet 127.0.1.11 netmask 0xffffffff
	inet 127.0.1.22 netmask 0xffffffff
	inet 127.0.1.17 netmask 0xffffffff
	inet 127.0.1.28 netmask 0xffffffff
	inet 127.0.1.18 netmask 0xffffffff
	inet 127.0.1.19 netmask 0xffffffff
	inet 127.0.1.25 netmask 0xffffffff
	inet 127.0.1.5 netmask 0xffffffff
	inet 127.0.1.14 netmask 0xffffffff
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
	groups: pflog
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet 192.168.12.1 netmask 0xffffff00
	inet 192.168.12.16 netmask 0xffffff00
	inet 192.168.12.21 netmask 0xffffff00
	inet 192.168.12.12 netmask 0xffffff00
	inet 192.168.12.29 netmask 0xffffff00
	inet 192.168.12.15 netmask 0xffffff00
	inet 192.168.12.11 netmask 0xffffff00
	inet 192.168.12.22 netmask 0xffffff00
	inet 192.168.12.17 netmask 0xffffff00
	inet 192.168.12.28 netmask 0xffffff00
	inet 192.168.12.18 netmask 0xffffff00
	inet 192.168.12.19 netmask 0xffffff00
	inet 192.168.12.25 netmask 0xffffff00
	inet 192.168.12.5 netmask 0xffffffff
	inet 192.168.12.14 netmask 0xffffff00
	groups: lo
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: vnet-jail-bridge
	ether 02:36:b3:c1:8a:00
	inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 1 priority 128 path cost 20000
	groups: bridge
	nd6 options=1<PERFORMNUD>


As you can see thereis a bridge (bridg0) with an IPv4 10.0.0.1/24. PF assume
the nat fonction for this range to 10.0.010/24 the new jail IPv4.


/etc/jail.conf

jitsi{
   $id="10";
   $ipaddr="10.0.0.${id}";
   $mask="255.255.255.0";
   $gw="10.0.0.1";

   host.hostname="${name}.exemple.net";
   path="/jails/${name}";
   exec.clean;
   exec.consolelog  = "/var/log/consolelog$(name}.log";

   vnet             = "new";
   vnet.interface   = "epair${id}b";

   exec.prestart     = "ifconfig epair${id} create up";
   exec.prestart    += "ifconfig epair${id}a up descr vnet-${name}";
   exec.prestart    += "ifconfig bridge0 addm epair${id}a";

   exec.start       = "/bin/sh /etc/rc";
   exec.start       += "/sbin/ifconfig epair${id}b ${ipaddr} netmask ${mask} up";
   exec.start       += "/sbin/route add default ${gw}";
   exec.start       += "/sbin/ifconfig epair${id}b inet6 2a01:4f9:4a:1fd8::27";
   exec.start       += "route add -inet6 default 2a01:4f9:4a:1fd8::2";

   exec.stop         = "/bin/sh /etc/rc.shutdown";

   exec.poststop     = "ifconfig epair${id}b -vnet ${name}";
   exec.poststop    += "ifconfig bridge0 deletem epair${id}a";
   exec.poststop    += "sleep 2";
   exec.poststop    += "ifconfig epair${id}a destroy";

   allow.mount.fusefs=1;
   mount.fstab="/etc/fstab.${name}";
   devfs_ruleset="5";
   allow.raw_sockets;
   persist;
}

The jail statup seems to be ok:
# jail -cv jitsi
jitsi: run command as root: /sbin/mount -t nullfs -o rw,late /usr/local/etc/letsencrypt /jails/jitsi/usr/local/etc/letsencrypt
jitsi: run command as root: /sbin/mount -t nullfs -o rw,late /usr/local/www/certbot /jails/jitsi/usr/local/www/certbot
jitsi: run command as root: /sbin/mount -t devfs -oruleset=5 . /jails/jitsi/dev
jitsi: run command as root: ifconfig epair10 create up
epair10a
jitsi: run command as root: ifconfig epair10a up descr vnet-jitsi
jitsi: run command as root: ifconfig bridge0 addm epair10a
jitsi: jail_set(JAIL_CREATE) name=jitsi allow.mount=true allow.mount.devfs allow.mount.zfs=true devfs_ruleset=5 enforce_statfs=1 sysvshm=new host.hostname=jitsi.foucry.net path=/jails/jitsi vnet=new allow.mount.fusefs=true allow.raw_sockets persist
jitsi: created
jitsi: run command as root: /sbin/ifconfig epair10b vnet jitsi
jitsi: run command in jail as root: /bin/sh /etc/rc
jitsi: run command in jail as root: /sbin/ifconfig epair10b 10.0.0.10 netmask 255.255.255.0 up
jitsi: run command in jail as root: /sbin/route add default 10.0.0.1
jitsi: run command in jail as root: /sbin/ifconfig epair10b inet6 2a01:4f9:4a:1fd8::27
jitsi: run command in jail as root: route add -inet6 default 2a01:4f9:4a:1fd8::2

epair10a on the host:

epair10a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: vnet-jitsi
	options=8<VLAN_MTU>
	ether 02:dc:c8:b1:ac:0a
	inet6 fe80::dc:c8ff:feb1:ac0a%epair10a prefixlen 64 scopeid 0x6
	groups: epair
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Once connected to the jail (through ssh/IPv4), ifconfig looks ok:

ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
	groups: pflog
epair10b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 02:dc:c8:b1:ac:0b
	inet6 fe80::dc:c8ff:feb1:ac0b%epair10b prefixlen 64 scopeid 0x3
	inet6 2a01:4f9:4a:1fd8::27 prefixlen 64
	inet 10.0.0.10 netmask 0xffffff00 broadcast 10.0.0.255
	groups: epair
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	nd6 options=8021<PERFORMNUD,AUTO_LINKLOCAL,DEFAULTIF>


But form my work machine to the jail the ping666666 does not work:

ping6 2a01:4f9:4a:1fd8::27   
PING6(56=40+8+8 bytes) 2a01:cb10:8e64:fe00:4aa4:72ff:fe9e:65a1 --> 2a01:4f9:4a:1fd8::27
^C
--- 2a01:4f9:4a:1fd8::27 ping6 statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss


Notice that a ping6 to another (old school jail) work perfectly:

ping6 2a01:4f9:4a:1fd8::25    
PING6(56=40+8+8 bytes) 2a01:cb10:8e64:fe00:4aa4:72ff:fe9e:65a1 --> 2a01:4f9:4a:1fd8::25
16 bytes from 2a01:4f9:4a:1fd8::25, icmp_seq=0 hlim=52 time=43.882 ms
16 bytes from 2a01:4f9:4a:1fd8::25, icmp_seq=1 hlim=52 time=43.731 ms
16 bytes from 2a01:4f9:4a:1fd8::25, icmp_seq=2 hlim=52 time=42.906 ms


I must miss something, or misunderstood something…

Any advices are welcome. 


Regards,
-- 
Jacques Foucry



Want to link to this message? Use this URL: <http://docs.FreeBSD.org/cgi/mid.cgi?X9HqnHRReRE34Nw5>