Date: Thu, 03 Mar 2016 08:49:11 -0600 From: dweimer <dweimer@dweimer.net> To: Nikos Vassiliadis <nvass@gmx.com> Cc: freebsd-virtualization@freebsd.org Subject: Re: bhyve and CARP? Message-ID: <f008eca6e0ce6de16708a42e3418ded5@dweimer.net> In-Reply-To: <56D6022A.8030808@gmx.com> References: <cf045594cf28b5dc5d89ecc2148c2fd4@dweimer.net> <56D6022A.8030808@gmx.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2016-03-01 2:57 pm, Nikos Vassiliadis wrote:
> Hi,
>
> On 03/01/16 18:43, dweimer wrote:
>> I am considering setting up a bhyve virtual machine to run pfSense.
>> Not
>> too thrilled with the CPU heat on the PC Engines APU1D4 when under
>> heavy
>> load, but don't want to rely entirely on a VM. As I like still having
>> internet if I would have to take my server offline for disk
>> replacement
>> or other issues, having web access to search for errors is a big plus.
>> So in order to avoid spending money on a new piece of hardware I
>> thought
>> why not do a VM with CARP fail over to the physical. I am not finding
>> much searching on FreeBSD byhve and CARP, I know its somewhat of an
>> issue withing VMware on ESX making sure you enable the right options
>> on
>> the virtual switches and interfaces.
>>
>> Enable promiscuous mode on the vSwitch
>> Enable "MAC Address changes"
>> Enable "Forged transmits"
>>
>> Before I got started on the setup I was curious if anyone has done
>> something similar, or know if this isn't possible on bhyve at the
>> current version? I am running my system currently on 10.3-BETA3.
>>
>
> I am running two postgres VMs with DRBD and not CARP but UCARP which
> should be 100% compatible with CARP. Each VM has a tap interface and
> each tap is bridged to a bridge interface. There is no need for special
> configuration. Everything works as expected.
>
Well so far I have it mostly working, one issue though, that I can't
quite find the source of the problem. I have multiple port forwards
setup and use NAT reflection to make those accessible from the same host
name internally and externally. I am redirecting ports 80, 443, 7443,
and 8443 among others on of the virtual carp IP addresses. 80 and 443
are redirected to my proxy jail running Squid as a reverse proxy, jail
is on same host as bhyve. 7443 redirects to Ubiquiti UniFi Video server
for HTTPS running on another bhyve Linux virtual machine. 8443 redirects
to Ubiquiti UniFi Wireless controller for HTTPS on another jail on the
same host as the bhyve virtual machines.
Everything that is running with NAT reflection works except for the port
443 traffic from the bhyve host machine, any jails running on it, and
the other bhyve virtual machine. However it works fine from other
network clients. Of course the NAT reflection is so that the same
certificate can be used on all the HTTPS connections and show as valid.
As near as I can tell the initial request makes it through the pfSense,
to the Proxy. The Proxy's response makes it back to the pfSense. The
pfSense system sends it to the client, but the client doesn't
acknowledge that it received it. I have used tcpdump on the system to
verify that it does receive the packets.
I initially suspected something with the HTTPS was rejecting the virtual
IPs used with CARP but that doesn't explain why it works on the other
HTTPS ports. And failing over to the old physical APU1D4 it all works.
As well as it working from other clients.
I plan to add a second HTTPS port to the squid reverse proxy
configuration to see if its isolated to the port 443 or if its isolated
to the HTTPS on squid. I will also try redirecting straight to the
Apache jail that the Proxy forwards to, Squid is only used as a reverse
proxy on this setup so that I can test Squid updates here before
installing them on the Reverse proxy I maintain at work.
--
Thanks,
Dean E. Weimer
http://www.dweimer.net/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?f008eca6e0ce6de16708a42e3418ded5>