Date: Sun, 19 Aug 2018 09:13:00 -0700 From: Pete Wright <pete@nomadlogic.org> To: freebsd-virtualization@freebsd.org Subject: bhyve NAT issue Message-ID: <bb37f29f-6604-a289-ebab-8accc72eeb94@nomadlogic.org>
next in thread | raw e-mail | index | archive | help
Hello, I have a baremetal server hosted by Vultr that I would like to host several bhyve instances on. I have been given one public ipv4 address, so my goal is to run the bhyve instances on a private subnet (172.16.0.0/24) then use pf to NAT this subnet and do port forwarding to. I am having an issues though getting the VM instances network to work. Here is how the hybervisor is configured: uname: FreeBSD cojo 11.2-STABLE FreeBSD 11.2-STABLE #1 r337947: Fri Aug 17 03:22:33 PDT 2018 pete@cojo:/usr/obj/usr/home/pete/src/freebsd-stable/sys/GENERIC amd64 rc.conf: cloned_interfaces="bridge0 tap0" ifconfig_bridge0="inet 172.16.0.1 netmask 255.255.255.0 addm tap0 up" gateway_enable="YES" ipv6_gateway_enable="YES" pf_enable="YES" pf.conf: ext_if=ix0 scrub in all nat on $ext_if inet from 172.16.0.0/24 to any -> ($ext_if) pass in all pass out all and my script to invoke the VM: /usr/sbin/bhyve -AHP -s 31:0,lpc \ -s 2:0,virtio-net,tap0 \ -s 3:0,virtio-blk,/vms/freebsd0 \ -s 29,fbuf,tcp=0.0.0.0:5900,w=1600,h=900,wait \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ -c 2 -m 1024M \ test0 I am able to bring up the FreeBSD VM and can attach to it via tightvnc, so that's great. I have configured the VM to have an IPv4 address of 172.16.0.20/24 with a gateway of 172.16.0.1. This is where I get stuck - when I attempt to ping the gateway from the VM I get "ping: sendto host down" errors. Then I run "arp -an" to see what's up from the VM and see the following: ? (172.16.0.1) at (incomplete) on vtnet0 expired [ethernet] when I run tcpdump against the bridge0 interface on the hypervisor while ping is run I see the following output: 15:56:48.995284 ARP, Request who-has 172.16.0.1 tell 172.16.0.20, length 46 15:56:48.995292 ARP, Reply 172.16.0.1 is-at 02:46:2f:56:ab:00 (oui Unknown), length 28 And I can verify that that is the correct MAC addr of the bridge0 interface: $ ifconfig bridge0|grep ether ether 02:46:2f:56:ab:00 yet, tcpdump on the vtnet0 interface from the VM does not show the responses to the ARP requests: ARP: Request who-has 172.16.0.1 tell 172.16.0.20, length 28 ARP: Request who-has 172.16.0.1 tell 172.16.0.20, length 28 ARP: Request who-has 172.16.0.1 tell 172.16.0.20, length 28 I suspect I'm missing something trivial here, so any input would be appreciated :) Oh, one last bit - here are my sysctl settings: security.bsd.see_other_uids=0 security.bsd.see_other_gids=0 security.bsd.unprivileged_read_msgbuf=0 security.bsd.unprivileged_proc_debug=0 kern.randompid=1 security.bsd.stack_guard_page=1 vfs.zfs.min_auto_ashift=12 net.link.tap.up_on_open=1 net.inet.ip.forwarding=1 net.inet6.ip6.forwarding=1 Cheers, -pete -- Pete Wright pete@nomadlogic.org @nomadlogicLA
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bb37f29f-6604-a289-ebab-8accc72eeb94>