Date: Thu, 26 Apr 2018 07:33:07 -0700 (PDT) From: "Rodney W. Grimes" <freebsd-rwg@pdx.rh.CN85.dnsmgr.net> To: Paul Esson <paul.esson@redstor.com> Cc: Harry Schmalzbauer <freebsd@omnilan.de>, "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org> Subject: Re: bhyve networking Message-ID: <201804261433.w3QEX77v001811@pdx.rh.CN85.dnsmgr.net> In-Reply-To: <HE1PR0102MB2588FD30B37F0255952BD3579E8E0@HE1PR0102MB2588.eurprd01.prod.exchangelabs.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[ Charset ISO-8859-1 unsupported, converting... ] > Hi Folks, > > I've just realised that the igb1 interface is not up in any of the output I shared. So I took the switch out of the equation and created tap and bridge interfaces manually, then added igb1 and tap0 to bridge0 and brought the bridge up. Finally, I brought igb1 and tap0 up. Once all the interfaces were up I amended the guest configuration to replace network0_switch="public" with network0_device="tap0". Now when I start my guest I have network connectivity on the guest VLAN. > > I'd really like to try and use the switch approach if possible and had thought that creating the switch and adding the igb1 interface would have brought igb1 up automatically. Is that the expected behaviour? No, the expected behavior is to not alter the state of igb1, that would be doing automagic stuff behind your back, you should add ifconfig_igb1="up" to the hosts /etc/rc.conf file. And I think all your issues well resolve and things shall work as you wanted. > Regards, > > Paul Esson??|??Redstor Limited > t??+44 (0)118 951 5235??|???m??+44 (0)776 690 6514 > e??paul.esson@redstor.com > www.redstor.com > > > > > > -----Original Message----- > From: Paul Esson > Sent: 26 April 2018 13:41 > To: Harry Schmalzbauer <freebsd@omnilan.de> > Cc: freebsd-virtualization@freebsd.org > Subject: RE: bhyve networking > > Hi Folks, > > Apologies for the lack of detail on my first post. To recap, I am attempting to set-up a guest using vm-bhyve. I have a Dell PER730xd server with quad-port INTEL 350 NIC. The first two ports have been configured on a) a management LAN for the host and b) an application LAN for the guests. > > FreeBSD 11.1-RELEASE-p9 > Dell PowerEdge R730xd - INTEL i350 NICs > > NIC-1 igb0 24:6E:96:B4:61:CC VLAN92 ge-6/0/11 (Host) > NIC-2 igb1 24:6E:96:B4:61:CD VLAN101 ge-6/0/18 (Guests) - not a trunk > > Both interfaces are active as viewed from the host, but I have only assigned an ipv4 address to igb0 for management of the host > > igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> > ether 24:6e:96:b4:61:cc > hwaddr 24:6e:96:b4:61:cc > inet 172.16.92.20 netmask 0xffffff00 broadcast 172.16.92.255 > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > media: Ethernet autoselect (1000baseT <full-duplex>) > status: active > > igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> > ether 24:6e:96:b4:61:cd > hwaddr 24:6e:96:b4:61:cd > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > media: Ethernet autoselect (1000baseT <full-duplex>) > status: active > > If I assign a temporary address to igb1 I can then ping other computers on the guests subnet - I've had to hide the address as the network is restricted. > > # ifconfig igb1 inet xx.xxx.xxx.xx/25 up # ping xx.xxx.xxx.xx PING xx.xxx.xxx.xx (xx.xxx.xxx.xx): 56 data bytes > 64 bytes from xx.xxx.xxx.xx: icmp_seq=0 ttl=64 time=0.145 ms > 64 bytes from xx.xxx.xxx.xx: icmp_seq=1 ttl=64 time=0.080 ms > 64 bytes from xx.xxx.xxx.xx: icmp_seq=2 ttl=64 time=0.078 ms > 64 bytes from xx.xxx.xxx.xx: icmp_seq=3 ttl=64 time=0.077 ms > 64 bytes from xx.xxx.xxx.xx: icmp_seq=4 ttl=64 time=0.076 ms > > I then used the "vm" command to create a virtual switch and add interface igb1 to it. This automatically created the bridge interface. > > root@dc1-olbp-hn-01:~ # vm switch create public root@dc1-olbp-hn-01:~ # vm switch add public igb1 root@dc1-olbp-hn-01:~ # vm switch info public > ------------------------ > Virtual Switch: public > ------------------------ > type: auto > ident: bridge0 > vlan: - > nat: - > physical-ports: igb1 > bytes-in: 0 (0.000B) > bytes-out: 0 (0.000B) > > Finally, I created a guest VM and gave its NIC the same ipv4 address details I used previously to test igb1 from the host. This automatically created the tap interface. > > igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> > ether 24:6e:96:b4:61:cc > hwaddr 24:6e:96:b4:61:cc > inet 172.16.92.20 netmask 0xffffff00 broadcast 172.16.92.255 > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > media: Ethernet autoselect (1000baseT <full-duplex>) > status: active > > igb1: flags=8d02<BROADCAST,PROMISC,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> > ether 24:6e:96:b4:61:cd > hwaddr 24:6e:96:b4:61:cd > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > media: Ethernet autoselect (1000baseT <full-duplex>) > status: active > > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 > options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 > inet 127.0.0.1 netmask 0xff000000 > nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> > groups: lo > > bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > description: vm-public > ether 02:ee:ce:b0:6a:00 > nd6 options=1<PERFORMNUD> > groups: bridge > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> > ifmaxaddr 0 port 7 priority 128 path cost 2000000 > member: igb1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> > ifmaxaddr 0 port 2 priority 128 path cost 20000 > > tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 > description: vmnet-testvm-0-public > options=80000<LINKSTATE> > ether 00:bd:dd:51:0a:00 > hwaddr 00:bd:dd:51:0a:00 > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > media: Ethernet autoselect > status: active > groups: tap > Opened by PID 1791 > > >From the guest VM I can see that the interface vtnet0 is up and has the relevant ipv4 address information. However, I cannot communicate with any other computer on the guest subnet or beyond. > > vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 > options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE> > ether 58:9c:fc:08:4a:20 > hwaddr 58:9c:fc:08:4a:20 > inet xx.xxx.xxx.xx netmask 0xffffff80 broadcast xx.xxx.xxx.xx > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > media: Ethernet 10Gbase-T <full-duplex> > status: active > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 > options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 > inet 127.0.0.1 netmask 0xff000000 > nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> > groups: lo > > The guest configuration file has the following network details network0_type="virtio-net" > network0_switch="public" > > >From the vm-bhyve.log I see the following Apr 26 07:59:23: initialising Apr 26 07:59:23: [loader: bhyveload] Apr 26 07:59:23: [uefi: no] Apr 26 07:59:23: [cpu: 1] Apr 26 07:59:23: [memory: 256M] Apr 26 07:59:23: [hostbridge: standard] Apr 26 07:59:23: [com ports: com1] Apr 26 07:59:23: [uuid: 417cfb63-491f-11e8-949b-246e96b461cc] > Apr 26 07:59:23: [utctime: no] > Apr 26 07:59:23: [debug mode: no] > Apr 26 07:59:23: [primary disk: disk0] > Apr 26 07:59:23: [primary disk dev: sparse-zvol] Apr 26 07:59:23: generated static mac 58:9c:fc:08:4a:20 (based on 'testvm:0:1524725963:0') Apr 26 07:59:23: initialising network device tap0 Apr 26 07:59:23: adding tap0 -> bridge0 (public) Apr 26 07:59:23: booting > > Should I have to supply ipv4 details anywhere other than the guest's own vtnet0 interface? If I re-configure the switch to remove the igb1 interface and add igb0 instead, then change the guest ipv4 address details to the management network (172.16.92.0/24), I can connect to other computers on that subnet and beyond. > > vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 > options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE> > ether 58:9c:fc:08:4a:20 > hwaddr 58:9c:fc:08:4a:20 > inet 172.16.92.21 netmask 0xffffff80 broadcast 172.16.92.127 > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > media: Ethernet 10Gbase-T <full-duplex> > status: active > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 > options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 > inet 127.0.0.1 netmask 0xff000000 > nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> > groups: lo > $ ping 172.16.92.11 > PING 172.16.92.11 (172.16.92.11): 56 data bytes > 64 bytes from 172.16.92.11: icmp_seq=0 ttl=64 time=0.416 ms > 64 bytes from 172.16.92.11: icmp_seq=1 ttl=64 time=0.371 ms > 64 bytes from 172.16.92.11: icmp_seq=2 ttl=64 time=0.369 ms > --- 172.16.92.11 ping statistics --- > 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.369/0.385/0.416/0.022 ms > > $ telnet bbc.co.uk 443 > Trying 151.101.192.81... > Connected to bbc.co.uk. > Escape character is '^]'. > > > Regards, > > > Paul Esson | Redstor Limited > t +44 (0)118 951 5235 | m +44 (0)776 690 6514 > e paul.esson@redstor.com > www.redstor.com > > > > > > -----Original Message----- > From: Rodney W. Grimes <freebsd-rwg@pdx.rh.CN85.dnsmgr.net> > Sent: 25 April 2018 22:31 > To: Paul Esson <paul.esson@redstor.com> > Cc: freebsd-virtualization@freebsd.org > Subject: Re: bhyve networking > > > Hi Rod, > > Can you share a command line for that? > Its not just a single command, but you want these in /etc/rc.conf of the GUEST: > network_interfaces="lo0" > cloned_interfaces="vlan48" > ifconfig_lo0=" inet 127.0.0.1 netmask 0xff000000" > ifconfig_vtnet0=" up" > ifconfig_vlan48=" inet 192.168.48.38 netmask 0xffffff00 vlan 48 vlandev vtnet0" > > That may be your issue... is your vtnet0 "up" in the guest. > > It would help a whole lot to share more of the info about your system, from commands, not from "vm-bhyve" settings. > Like > ifconfig -a > on the host and the guest would be a starting point. > > > I also tried presenting an access port from my switch on a specific VLAN - not trimmed. > Trimmed? You mean you set the switch port to untagged mode, and had the switch tag/untag the packets to a specific vlan. > Be sure you also set the default incoming tag at the switch if you did this, some switches do not follow the vlan setting. > > > Would I still have to tag the interface on the guest in that scenario? > No. If I understand what I think you meant by trimmed. > > > Regards, > > > > > > Paul Esson | Redstor Limited > > t +44 (0)118 951 5235 > > m +44 (0)776 690 6514 > > e paul.esson@redstor.com > > > > ________________________________ > > From: Rodney W. Grimes <freebsd-rwg@pdx.rh.CN85.dnsmgr.net> > > Sent: Wednesday, April 25, 2018 9:33:57 PM > > To: Paul Esson > > Cc: Harry Schmalzbauer; freebsd-virtualization@freebsd.org > > Subject: Re: bhyve networking > > > > [ Charset windows-1252 unsupported, converting... ] > > > Hi Harry, > > > I?m simply using the ?vm? utility as in > > > > > > vm switch create public > > > vm switch add public igb1 > > > > > > That must make underlying calls to if config or equivalent as the bridge and tap interfaces are created automatically. > > > > > > The vm template file has these relevant parameters > > > > > > network0_type=?virtio-net? > > > network0_switch=?public? > > > > > > I?ve done nothing to the igb1 interface other than connect it to a physical switch on the appropriate VLAN. > > > > How have you configured your vtnet devices inside the guest? If your > > pass a "trunked" ethernet device to a guest the guest is going to need to run vlan decapuslation. I do this here, and it works fine. > > > > vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 > > options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE> > > ether 58:9c:fc:0e:8b:ec > > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > > media: Ethernet 10Gbase-T <full-duplex> > > status: active > > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 > > options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> > > inet 127.0.0.1 netmask 0xff000000 > > inet6 ::1 prefixlen 128 > > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 > > nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> > > groups: lo > > vlan48: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > > ether 58:9c:fc:0e:8b:ec > > inet 192.168.48.38 netmask 0xffffff00 broadcast 192.168.48.255 > > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > > media: Ethernet 10Gbase-T <full-duplex> > > status: active > > vlan: 48 vlanpcp: 0 parent interface: vtnet0 > > groups: vlan > > > > ... > > > > > > Bez?glich Paul Esson's Nachricht vom 25.04.2018 20:44 (localtime): > > > > Hi Folks, > > > > > > > > I'm struggling with networking when using vm-bhyve on FreeBSD > > > > 11.1-RELEASE. I have two NICs and have configured the first (igb0) > > > > on a management network and want to use the second (igb1) for VMs. > > > > However, I can't get any VM to communicate through the virtual > > > > switch if I have igb1 added to it. If I take the NIC out of the > > > > switch and configure an ipv4 address on it I can reach other hosts > > > > on the relevant subnet so I believe the poet set-up is valid. If > > > > I replace igb1 in the switch with igb0, I can then configure VMs > > > > on my management network and they have network connectivity. Can > > > > I only use > > > > > > Hello, > > > > > > a example of the command you used was nice. > > > I guess you're using if_bridge(4) ? the example would clarify. > > > But there's ng_bridge(4) and vale(4) also, and others are using > > > OpenVSwitch... > > > > > > > an interface that has an IP address configured at the host level before adding it to the switch? I've used other > > > > > > No. > > > But the interface has to be in promisc mode. And some offloading > > > functions must be disabled, but in case of if_bridge(4), it's done > > > automagically (and reverted if you remove the interface again). > > > > > > Hth, > > > > > > -harry > > > _______________________________________________ > > > freebsd-virtualization@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > > > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" > > > > > > > > > > -- > > Rod Grimes rgrimes@freebsd.org > > _______________________________________________ > > freebsd-virtualization@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" > > > > -- > Rod Grimes rgrimes@freebsd.org > > Paul Esson??|??Redstor Limited > t??+44 (0)118 951 5235??|???m??+44 (0)776 690 6514 e??paul.esson@redstor.com www.redstor.com > > > > > > -----Original Message----- > From: Harry Schmalzbauer <freebsd@omnilan.de> > Sent: 26 April 2018 08:39 > To: Paul Esson <paul.esson@redstor.com> > Cc: freebsd-virtualization@freebsd.org > Subject: Re: bhyve networking > > Bez?glich Paul Esson's Nachricht vom 25.04.2018 23:15 (localtime): > > Hi Rod, > > Can you share a command line for that? I also tried presenting an > > access port from my switch on a specific VLAN - not trimmed. Would I > > still have to tag the interface on the guest in that scenario? > > Hmm, I lost the overview - I'm not familar with 'vm'. > To filter a specific id (tag/untag frames) inside the guest: > 'ifconfig vlan[N] create vlandev vtnet0 vlan nnnn' > 'ifconfig vlan[N] create vlandev vtnet0 vlan nnnm' > > At boot time by rc(8): > vlans_vtnet0="vtnet_dmz vtnet_dmz2" > create_args_vtnet_dmz="vlan nnnn" > create_args_vtnet_dmz2="vlan nnnm" > > [To optionally also rename the vlan interfaces after manually creating cloned vlan interfaces, which is what the rc.conf(5) example does: > ifconfig rename vlan0 vtnet_dmz; ifconfig rename vlan0 vtnet_dmz2; ] > > Hth, > > -harry > > > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" > > -- Rod Grimes rgrimes@freebsd.org
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201804261433.w3QEX77v001811>