From owner-freebsd-virtualization@freebsd.org Thu Apr 26 14:31:26 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0F139FAEC5F for ; Thu, 26 Apr 2018 14:31:26 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 36442719FC for ; Thu, 26 Apr 2018 14:31:24 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id w3QEVM4J001779; Thu, 26 Apr 2018 07:31:22 -0700 (PDT) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id w3QEVMRa001778; Thu, 26 Apr 2018 07:31:22 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201804261431.w3QEVMRa001778@pdx.rh.CN85.dnsmgr.net> Subject: Re: bhyve networking In-Reply-To: To: Paul Esson Date: Thu, 26 Apr 2018 07:31:22 -0700 (PDT) CC: Harry Schmalzbauer , "freebsd-virtualization@freebsd.org" X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Apr 2018 14:31:26 -0000 [ Charset ISO-8859-1 unsupported, converting... ] > Hi Folks, > > Apologies for the lack of detail on my first post. To recap, I am attempting to set-up a guest using vm-bhyve. I have a Dell PER730xd server with quad-port INTEL 350 NIC. The first two ports have been configured on a) a management LAN for the host and b) an application LAN for the guests. > > FreeBSD 11.1-RELEASE-p9 > Dell PowerEdge R730xd - INTEL i350 NICs > > NIC-1 igb0 24:6E:96:B4:61:CC VLAN92 ge-6/0/11 (Host) > NIC-2 igb1 24:6E:96:B4:61:CD VLAN101 ge-6/0/18 (Guests) - not a trunk > > Both interfaces are active as viewed from the host, but I have only assigned an ipv4 address to igb0 for management of the host > > igb0: flags=8843 metric 0 mtu 1500 options=6403bb > ether 24:6e:96:b4:61:cc > hwaddr 24:6e:96:b4:61:cc > inet 172.16.92.20 netmask 0xffffff00 broadcast 172.16.92.255 > nd6 options=29 > media: Ethernet autoselect (1000baseT ) > status: active > > igb1: flags=8c02 metric 0 mtu 1500 options=6403bb ^^ MIssing UP, interface is down > ether 24:6e:96:b4:61:cd > hwaddr 24:6e:96:b4:61:cd > nd6 options=29 > media: Ethernet autoselect (1000baseT ) > status: active > > If I assign a temporary address to igb1 I can then ping other computers on the guests subnet - I've had to hide the address as the network is restricted. > > # ifconfig igb1 inet xx.xxx.xxx.xx/25 up > # ping xx.xxx.xxx.xx > PING xx.xxx.xxx.xx (xx.xxx.xxx.xx): 56 data bytes > 64 bytes from xx.xxx.xxx.xx: icmp_seq=0 ttl=64 time=0.145 ms > 64 bytes from xx.xxx.xxx.xx: icmp_seq=1 ttl=64 time=0.080 ms > 64 bytes from xx.xxx.xxx.xx: icmp_seq=2 ttl=64 time=0.078 ms > 64 bytes from xx.xxx.xxx.xx: icmp_seq=3 ttl=64 time=0.077 ms > 64 bytes from xx.xxx.xxx.xx: icmp_seq=4 ttl=64 time=0.076 ms > > I then used the "vm" command to create a virtual switch and add interface igb1 to it. This automatically created the bridge interface. > > root@dc1-olbp-hn-01:~ # vm switch create public > root@dc1-olbp-hn-01:~ # vm switch add public igb1 > root@dc1-olbp-hn-01:~ # vm switch info public > ------------------------ > Virtual Switch: public > ------------------------ > type: auto > ident: bridge0 > vlan: - > nat: - > physical-ports: igb1 > bytes-in: 0 (0.000B) > bytes-out: 0 (0.000B) > > Finally, I created a guest VM and gave its NIC the same ipv4 address details I used previously to test igb1 from the host. This automatically created the tap interface. > > igb0: flags=8843 metric 0 mtu 1500 options=6403bb > ether 24:6e:96:b4:61:cc > hwaddr 24:6e:96:b4:61:cc > inet 172.16.92.20 netmask 0xffffff00 broadcast 172.16.92.255 > nd6 options=29 > media: Ethernet autoselect (1000baseT ) > status: active > > igb1: flags=8d02 metric 0 mtu 1500 options=6403bb ^^ mising up, interface is down > ether 24:6e:96:b4:61:cd > hwaddr 24:6e:96:b4:61:cd > nd6 options=29 > media: Ethernet autoselect (1000baseT ) > status: active > > lo0: flags=8049 metric 0 mtu 16384 > options=600003 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 > inet 127.0.0.1 netmask 0xff000000 > nd6 options=21 > groups: lo > > bridge0: flags=8843 metric 0 mtu 1500 > description: vm-public > ether 02:ee:ce:b0:6a:00 > nd6 options=1 > groups: bridge > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > member: tap0 flags=143 > ifmaxaddr 0 port 7 priority 128 path cost 2000000 > member: igb1 flags=143 > ifmaxaddr 0 port 2 priority 128 path cost 20000 > > tap0: flags=8943 metric 0 mtu 1500 > description: vmnet-testvm-0-public > options=80000 > ether 00:bd:dd:51:0a:00 > hwaddr 00:bd:dd:51:0a:00 > nd6 options=29 > media: Ethernet autoselect > status: active > groups: tap > Opened by PID 1791 > > >From the guest VM I can see that the interface vtnet0 is up and has the relevant ipv4 address information. However, I cannot communicate with any other computer on the guest subnet or beyond. Guest may be up, but the host interface is in state down. > vtnet0: flags=8943 metric 0 mtu 1500 > options=80028 > ether 58:9c:fc:08:4a:20 > hwaddr 58:9c:fc:08:4a:20 > inet xx.xxx.xxx.xx netmask 0xffffff80 broadcast xx.xxx.xxx.xx > nd6 options=29 > media: Ethernet 10Gbase-T > status: active > lo0: flags=8049 metric 0 mtu 16384 > options=600003 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 > inet 127.0.0.1 netmask 0xff000000 > nd6 options=21 > groups: lo > > The guest configuration file has the following network details > network0_type="virtio-net" > network0_switch="public" > > >From the vm-bhyve.log I see the following > Apr 26 07:59:23: initialising > Apr 26 07:59:23: [loader: bhyveload] > Apr 26 07:59:23: [uefi: no] > Apr 26 07:59:23: [cpu: 1] > Apr 26 07:59:23: [memory: 256M] > Apr 26 07:59:23: [hostbridge: standard] > Apr 26 07:59:23: [com ports: com1] > Apr 26 07:59:23: [uuid: 417cfb63-491f-11e8-949b-246e96b461cc] > Apr 26 07:59:23: [utctime: no] > Apr 26 07:59:23: [debug mode: no] > Apr 26 07:59:23: [primary disk: disk0] > Apr 26 07:59:23: [primary disk dev: sparse-zvol] > Apr 26 07:59:23: generated static mac 58:9c:fc:08:4a:20 (based on 'testvm:0:1524725963:0') > Apr 26 07:59:23: initialising network device tap0 > Apr 26 07:59:23: adding tap0 -> bridge0 (public) > Apr 26 07:59:23: booting > > Should I have to supply ipv4 details anywhere other than the guest's own vtnet0 interface? If I re-configure the switch to remove the igb1 interface and add igb0 instead, then change the guest ipv4 address details to the management network (172.16.92.0/24), I can connect to other computers on that subnet and beyond. > > vtnet0: flags=8943 metric 0 mtu 1500 > options=80028 > ether 58:9c:fc:08:4a:20 > hwaddr 58:9c:fc:08:4a:20 > inet 172.16.92.21 netmask 0xffffff80 broadcast 172.16.92.127 > nd6 options=29 > media: Ethernet 10Gbase-T > status: active > lo0: flags=8049 metric 0 mtu 16384 > options=600003 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 > inet 127.0.0.1 netmask 0xff000000 > nd6 options=21 > groups: lo > $ ping 172.16.92.11 > PING 172.16.92.11 (172.16.92.11): 56 data bytes > 64 bytes from 172.16.92.11: icmp_seq=0 ttl=64 time=0.416 ms > 64 bytes from 172.16.92.11: icmp_seq=1 ttl=64 time=0.371 ms > 64 bytes from 172.16.92.11: icmp_seq=2 ttl=64 time=0.369 ms > --- 172.16.92.11 ping statistics --- > 3 packets transmitted, 3 packets received, 0.0% packet loss > round-trip min/avg/max/stddev = 0.369/0.385/0.416/0.022 ms > > $ telnet bbc.co.uk 443 > Trying 151.101.192.81... > Connected to bbc.co.uk. > Escape character is '^]'. > > > Regards, > > > Paul Esson | Redstor Limited > t +44 (0)118 951 5235 | m +44 (0)776 690 6514 > e paul.esson@redstor.com > www.redstor.com > > > > > > -----Original Message----- > From: Rodney W. Grimes > Sent: 25 April 2018 22:31 > To: Paul Esson > Cc: freebsd-virtualization@freebsd.org > Subject: Re: bhyve networking > > > Hi Rod, > > Can you share a command line for that? > Its not just a single command, but you want these in /etc/rc.conf of the GUEST: > network_interfaces="lo0" > cloned_interfaces="vlan48" > ifconfig_lo0=" inet 127.0.0.1 netmask 0xff000000" > ifconfig_vtnet0=" up" > ifconfig_vlan48=" inet 192.168.48.38 netmask 0xffffff00 vlan 48 vlandev vtnet0" > > That may be your issue... is your vtnet0 "up" in the guest. Or better yet, is your igb1 interface "up" on the host? Add ifconfig_igb1="up" to your hosts /etc/rc.conf file. And type: ifconfig igb1 up and your probelm should resolve. > It would help a whole lot to share more of the info about your system, from commands, not from "vm-bhyve" settings. > Like > ifconfig -a > on the host and the guest would be a starting point. > > > I also tried presenting an access port from my switch on a specific VLAN - not trimmed. > Trimmed? You mean you set the switch port to untagged mode, and had the switch tag/untag the packets to a specific vlan. > Be sure you also set the default incoming tag at the switch if you did this, some switches do not follow the vlan setting. > > > Would I still have to tag the interface on the guest in that scenario? > No. If I understand what I think you meant by trimmed. > > > Regards, > > > > > > Paul Esson | Redstor Limited > > t +44 (0)118 951 5235 > > m +44 (0)776 690 6514 > > e paul.esson@redstor.com > > > > ________________________________ > > From: Rodney W. Grimes > > Sent: Wednesday, April 25, 2018 9:33:57 PM > > To: Paul Esson > > Cc: Harry Schmalzbauer; freebsd-virtualization@freebsd.org > > Subject: Re: bhyve networking > > > > [ Charset windows-1252 unsupported, converting... ] > > > Hi Harry, > > > I?m simply using the ?vm? utility as in > > > > > > vm switch create public > > > vm switch add public igb1 > > > > > > That must make underlying calls to if config or equivalent as the bridge and tap interfaces are created automatically. > > > > > > The vm template file has these relevant parameters > > > > > > network0_type=?virtio-net? > > > network0_switch=?public? > > > > > > I?ve done nothing to the igb1 interface other than connect it to a physical switch on the appropriate VLAN. > > > > How have you configured your vtnet devices inside the guest? If your > > pass a "trunked" ethernet device to a guest the guest is going to need to run vlan decapuslation. I do this here, and it works fine. > > > > vtnet0: flags=8943 metric 0 mtu 1500 > > options=80028 > > ether 58:9c:fc:0e:8b:ec > > nd6 options=29 > > media: Ethernet 10Gbase-T > > status: active > > lo0: flags=8049 metric 0 mtu 16384 > > options=600003 > > inet 127.0.0.1 netmask 0xff000000 > > inet6 ::1 prefixlen 128 > > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 > > nd6 options=21 > > groups: lo > > vlan48: flags=8843 metric 0 mtu 1500 > > ether 58:9c:fc:0e:8b:ec > > inet 192.168.48.38 netmask 0xffffff00 broadcast 192.168.48.255 > > nd6 options=29 > > media: Ethernet 10Gbase-T > > status: active > > vlan: 48 vlanpcp: 0 parent interface: vtnet0 > > groups: vlan > > > > ... > > > > > > Bez?glich Paul Esson's Nachricht vom 25.04.2018 20:44 (localtime): > > > > Hi Folks, > > > > > > > > I'm struggling with networking when using vm-bhyve on FreeBSD > > > > 11.1-RELEASE. I have two NICs and have configured the first (igb0) > > > > on a management network and want to use the second (igb1) for VMs. > > > > However, I can't get any VM to communicate through the virtual > > > > switch if I have igb1 added to it. If I take the NIC out of the > > > > switch and configure an ipv4 address on it I can reach other hosts > > > > on the relevant subnet so I believe the poet set-up is valid. If > > > > I replace igb1 in the switch with igb0, I can then configure VMs > > > > on my management network and they have network connectivity. Can > > > > I only use > > > > > > Hello, > > > > > > a example of the command you used was nice. > > > I guess you're using if_bridge(4) ? the example would clarify. > > > But there's ng_bridge(4) and vale(4) also, and others are using > > > OpenVSwitch... > > > > > > > an interface that has an IP address configured at the host level before adding it to the switch? I've used other > > > > > > No. > > > But the interface has to be in promisc mode. And some offloading > > > functions must be disabled, but in case of if_bridge(4), it's done > > > automagically (and reverted if you remove the interface again). > > > > > > Hth, > > > > > > -harry > > > _______________________________________________ > > > freebsd-virtualization@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > > > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" > > > > > > > > > > -- > > Rod Grimes rgrimes@freebsd.org > > _______________________________________________ > > freebsd-virtualization@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" > > > > -- > Rod Grimes rgrimes@freebsd.org > > Paul Esson??|??Redstor Limited > t??+44 (0)118 951 5235??|???m??+44 (0)776 690 6514 > e??paul.esson@redstor.com > www.redstor.com > > > > > > -----Original Message----- > From: Harry Schmalzbauer > Sent: 26 April 2018 08:39 > To: Paul Esson > Cc: freebsd-virtualization@freebsd.org > Subject: Re: bhyve networking > > Bez?glich Paul Esson's Nachricht vom 25.04.2018 23:15 (localtime): > > Hi Rod, > > Can you share a command line for that? I also tried presenting an > > access port from my switch on a specific VLAN - not trimmed. Would I > > still have to tag the interface on the guest in that scenario? > > Hmm, I lost the overview - I'm not familar with 'vm'. > To filter a specific id (tag/untag frames) inside the guest: > 'ifconfig vlan[N] create vlandev vtnet0 vlan nnnn' > 'ifconfig vlan[N] create vlandev vtnet0 vlan nnnm' > > At boot time by rc(8): > vlans_vtnet0="vtnet_dmz vtnet_dmz2" > create_args_vtnet_dmz="vlan nnnn" > create_args_vtnet_dmz2="vlan nnnm" > > [To optionally also rename the vlan interfaces after manually creating cloned vlan interfaces, which is what the rc.conf(5) example does: > ifconfig rename vlan0 vtnet_dmz; ifconfig rename vlan0 vtnet_dmz2; ] > > Hth, > > -harry > > > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" > > -- Rod Grimes rgrimes@freebsd.org