Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 17 Aug 2021 15:55:23 +0200
From:      Mario Marietto <marietto2008@gmail.com>
To:        freebsd-net@freebsd.org, freebsd-virtualization@freebsd.org
Subject:   Re: Confirm subscription to freebsd-net@FreeBSD.org
Message-ID:  <CA%2B1FSigbnNg08Q3GTLG59xtepzg-uQB=QPQoAt_u2DiMkU_avg@mail.gmail.com>
In-Reply-To: <CA%2B1FSii8M2E4ghOEenPkTw3HzyWj7a4Dm6=c7A7KruoMf%2BKg7w@mail.gmail.com>
References:  <1629141950-13176-mlmmj-4abd3c89@FreeBSD.org> <CA%2B1FSii8M2E4ghOEenPkTw3HzyWj7a4Dm6=c7A7KruoMf%2BKg7w@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000074620f05c9c1ad86
Content-Type: text/plain; charset="UTF-8"

Hello.

I'm trying to configure the bridging and the tap interface on freebsd 13
because I want to boot and use a bhyve VM based on Ubuntu 21.04 and Windows
10,but unfortunately,after several readings and help requests on :

      0.
https://www.reddit.com/r/Ubuntu/comments/p38ru1/ubuntu_2104_cant_connect_to_internet_when_it_is/


   1.


   https://forums.freebsd.org/threads/how-to-duplicate-the-tap0-interface-in-a-new-tap1-network-interface-that-it-is-not-used.81671/page-2#post-527355
   2.


   https://forums.freebsd.org/threads/ubuntu-budgie-emulated-with-bhyve-doesnt-connect-to-internet-because-the-error-network-unreachable-n-2.81340/#post-524303
   3.


   https://forums.freebsd.org/threads/im-tryng-to-emulate-ubuntu-budgie-with-bhyve-but-it-doesnt-connect-to-internet-because-the-error-network-unreachable.80916/#post-517845

     4.
https://askubuntu.com/questions/1357670/ubuntu-21-04-cant-connect-to-internet-when-it-is-emulated-with-bhyve-on-my-free

     5.
https://unix.stackexchange.com/questions/664466/ubuntu-21-04-cant-connect-to-internet-when-it-is-emulated-with-bhyve-on-my-free?noredirect=1#comment1250440_664466

     6.
https://serverfault.com/questions/1074745/my-bridging-configuration-on-freebsd-13-inst-properly-configured-because-any-bh

     7.
https://www.linuxquestions.org/questions/linux-virtualization-and-cloud-90/my-bridging-configuration-on-freebsd-13-ins%27t-properly-configured-because-any-bhyve-vm-based-on-ubuntu-21-04-and-windows-10-can%27t-connect-4175699240/


I haven't been able to understand why Ubuntu and Windows 10 VM can't
connect to the internet,while if I use vm-bhyve,it configures the network
in some different way and the vm can connect to the internet. Since I'm not
a coder,I'm not able to look inside the vm-bhyve code and I'm not able to
understand what's the secret key. Below I want to give you some useful
information for understanding how I have configured the bridge and the tap
interface on FreeBSD and on Ubuntu.

/etc/resolv.conf

search homenet.telecomitalia.it
nameserver 127.0.0.1
options edns0

my /etc/rc.conf is like this :

hostname="marietto"
keymap="it.kbd"
ifconfig_em0="DHCP"
local_unbound_enable="YES"
sshd_enable="YES"
powerd_enable="YES"
ntpdate_enable="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
dumpdev="NO"
dbus_enable="YES"
slim_enable="NO"
libvirt_enable="YES"
linux_enable="YES"
linux_mounts_enable="YES"
gateway_enable="YES"
kld_list="fusefs"
lightdm_enable="NO"
sendmail_enable="NO"
microcode_update_enable="YES"
fsck_y_enable="YES"
tcp_drop_synfin="NO"
telnet_enable="NO"
cupsd_enable="NO"
portmap_enable="NO"
rlogin_enable="NO"
inetd_enable="NO"
webcamd_enable="NO"
samba_enable="NO"
lpd_enable="NO"
winbindd_enable="NO"
nfs_server_enable="NO"
nfs_client_enable="NO"
vm_enable="YES"
vm_dir="/mnt/da1p1/vms/os/"

my /boot/loader.conf is like this :

vmm_load="YES"
nmdm_load="YES"
if_tap_load="YES"
if_bridge_load="YES"
fusefs_load="YES"
linprocfs_load="YES"
linsysfs_load="YES"
tmpfs_load="YES"
verbose_loading="YES"
vboxdrv_load="YES"
console=vga,com1
boot_multicons="YES"
boot_serial="YES"
console="comconsole,vidconsole"

my file /etc/sysctl.conf is like this :

net.link.tap.up_on_open=1
net.inet.ip.forwarding=1

I've read here to get some orientation about what to do :
https://wiki.freebsd.org/bhyve/UEFI
<https://wiki.freebsd.org/bhyve/UEFI,specially>; specially in this section :

kldload vmm
ifconfig tap0 create up
ifconfig bridge0 create up
ifconfig bridge0 addm em0

As a first attempt,with the suggestions of Vasily Postnicov,I tried to
configure the network interfaces of freebsd and of linux in a different
way,to be able to give the internet connection to ubuntu emulated with
bhyve. This is what I did :

root@marietto:/home/marietto # ifconfig em0 delete


root@marietto:/home/marietto # dhclient bridge0


DHCPDISCOVER on bridge0 to 255.255.255.255 port 67 interval 4
DHCPOFFER from 192.168.1.1
DHCPREQUEST on bridge0 to 255.255.255.255 port 67
DHCPACK from 192.168.1.1
bound to 192.168.1.2 -- renewal in 10800 seconds.


root@marietto:/home/marietto # ifconfig tap0 create up

root@marietto:/home/marietto # ifconfig bridge0 addm tap0

root@marietto:/usr/home/marietto # ifconfig


em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 1500
options=4810099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER,NOMAP>
ether e0:d5:5e:e2:1f:22
inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>


bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 58:9c:fc:10:ff:b7
inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 2000000
member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 2000000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>


vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether b6:74:ff:a8:4d:4e
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
groups: bridge vm-switch viid-4c918@
nd6 options=9<PERFORMNUD,IFDISABLED>



tap0: flags=8903<UP,BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
ether 58:9c:fc:10:ff:8e
groups: tap
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Unfortunately as soon as the vm boots,the em0 interface on freebsd gets
again the old IP number : 192.168.1.6 and I lose the connection to the net
on FreeBSD.

As a further try this is what I did : (take in consideration that I'm
trying to make a parallel network configuration on Freebsd,since I want to
keep vm-bhyve installed. Anyway,I've removed vm-bhyve several times and I
tried to configure the bridge network for bhyve,but even in this case it
didn't work. In short terms,it worked only the first time,but I haven't
been able to repeat succesfully the steps anymore. It's working only if I
let vm-bhyve configure the network by itself. This is the initial state of
the network / bridge configuration on my FreeBSD with vm-bhyve installed
but without starting a virtual machine.

marietto@marietto:~ $ ifconfig

em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 1500
options=481009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER,NOMAP>
ether e0:d5:5e:e2:1f:22
inet 192.168.1.6 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 0a:77:93:94:d5:77
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
       ifmaxaddr 0 port 1 priority 128 path cost 20000
groups: bridge vm-switch viid-4c918@
nd6 options=9<PERFORMNUD,IFDISABLED>

Since I already have a bridged interface (vm-public) it does not make any
sense to create another one (bridge0),so on the terminal I do :

root@marietto:/home/marietto # kldload vmm
kldload: can't load vmm: module already loaded or in kernel

root@marietto:/home/marietto # ifconfig tap0 create up

root@marietto:/home/marietto # ifconfig vm-public addm tap0

so,this is how appear the network configuration on FreeBSD after having
configured it like above :

root@marietto:/home/marietto # ifconfig

em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 1500
options=4810099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER,NOMAP>
ether e0:d5:5e:e2:1f:22
inet 192.168.1.6 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 0a:77:93:94:d5:77
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
       ifmaxaddr 0 port 4 priority 128 path cost 2000000
member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
       ifmaxaddr 0 port 1 priority 128 path cost 20000
groups: bridge vm-switch viid-4c918@
nd6 options=9<PERFORMNUD,IFDISABLED>

tap0: flags=8903<UP,BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
ether 58:9c:fc:10:ff:8e
groups: tap
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Then,I launch the bhyve VM with this commands :

bhyve -c 4 -m 8G -w -H \
-s 0,hostbridge \
-s 4,virtio-blk,/mnt/da1p1/vms/os/ubuntu-budgie-gpu/ubuntu-2104-gpu.img \
-s 5,virtio-net,tap0 \
-s 29,fbuf,tcp=0.0.0.0:5900,w=1440,h=900,wait \
-s 30,xhci,tablet \
-s 31,lpc -l com1,stdio \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-s 20,hda,play=/dev/dsp8,rec=/dev/dsp8 \
vm0

As I said Ubuntu VM can't connect to the internet. Below I want to show you
how I have configured Ubuntu,instead. On Ubuntu my /etc/resolv.conf file is
like this :

nameserver 127.0.0.53
options edns0 trust-ad
search homenet.telecomitalia.it

In addition I've created a netplan file inside the folder /etc/netplan,with
this content inside :

network
    version: 2
    renderer: NetworkManager
    ethernets:
       enp0s5:
           addresses:
                - 192.168.1.7/24
           gateway4: 192.168.1.1
           nameservers:
                     search : [homenet.telecomitalia.it]
                     addresses: 192.168.1.1

netplan apply and this is what happens :

the network interface enp0s5 gets the IP number 192.168.1.7 ping
192.168.1.7 : OK ping 192.168.1.6 (FreeBSD IP) = destination host
unreachable ping 192.168.1.1 : destination host unreachable PING google.it
(172.217.21.67) : host unreachable

tcpdump :

listening on enp0s5,link-type EN10MB (Ethernet),capture size 262144 bytes

20:58:04:688913 ARP, request who-has 192.168.1.1 tell 192.168.1.7,length 28

route -n

destination     gateway        genmask              flags metric ref  use  iface

0.0.0.0            192.168.1.1   0.0.0.0                UG     100
0    0     enp0s5
169.254.0.0    0.0.0.0           255.255.0.0        U       1000    0
  0    enp0s5
192.168.1.0    0.0.0.0           255.255.255.0    U       100      0
 0    enp0s5

ip route list

default via 192.168.1.1 dev enp0s5 proto static metric
100169.254.0.0/16 dev enp0s5 scope link metric 100192.168.1.0/24 dev
enp0s5 proto kernel scope link src 192.168.1.7 metric 100

ping google.it = temporary error in name resolution

firewall on ubuntu = not installed firewall on router = down

instead,from freebsd to ubuntu :

ping 192.168.1.7 : host is down

netstar -r

routing tables

internet:

destination         gateway        flags     Netif Expire

default                modemtim    UGS      em0
localhost             link#2           UH        lo0192.168.1.0/24
link#1           U          em0
192.168.1.6        link#1           UHS      lo0

I forgot. This is what happens if I remove the netplan conf file and I let
ubuntu choose the IP via DHCP :

/etc/netplan.conf

network:
    version: 2
    renderer: networkd
    ethernets:
        enp3s0:
            dhcp4: true

ping PING google.it (172.217.21.67) : network unreachable. What else ?

PS : Since I'm not a real system administrator,but I do it as a hobby,don't
give me generic suggestions because probably I will not understant what to
do. If you want to be a kind person,try to be exhaustive and precise. You
will help me a lot to understand how to fix the problem. thanks.

PS : more failed attempt to fix the problem are listed here :
https://forums.freebsd.org/threads/how-to-duplicate-the-tap0-interface-in-a-new-tap1-network-interface-that-it-is-not-used.81671/page-3

--00000000000074620f05c9c1ad86--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B1FSigbnNg08Q3GTLG59xtepzg-uQB=QPQoAt_u2DiMkU_avg>