Date: Tue, 18 Sep 2018 10:18:43 +0300 From: =?UTF-8?Q?Pekka_J=C3=A4rvinen?= <pekka.jarvinen@gmail.com> To: freebsd-net@freebsd.org Subject: Bridge, VLANs and breaking packets Message-ID: <CABvnMcg6SW02XkO6JnKtUXABvdCXVLj0OQFE-L5v=8S%2BfBZXtw@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
I'm trying out FreeBSD with bhyve hypervisor and trying to replicate
VMware ESXi vSwitch config in FreeBSD so that pfSense VM gets network
traffic properly.
VLANs:
* 111 =3D LAN, 192.168.101.0/24
* 333 =3D Internet (WAN), DHCP
pfSense virtual machine setup (ESXi & FreeBSD):
NICS:
* vnic0
* vnic0.111 192.168.101.1/24
* vnic0.333 DHCP from ISP
* NAT: VLAN 333 <-> VLAN 111 & DHCP server
* HW offloads off
Old ESXi setup:
* 192.168.101.6/24 on VLAN 111 GW 192.168.101.1
* Only vSwitch, no dvSwitches
vSwitch0:
----------------------- -----------------------
| allvlans | | Physical adapters |
| VLAN ID: 4095 (all) |----| * vmnic0, 1000 Mbps |
| * pfSense24 | | -----------------------
----------------------- |
|
----------------------- |
| Management Network |--/
| VLAN ID: 111 |
| VMkernel ports (1): |
| * vmk0: 192.168.101.6 |
-----------------------
Security policy:
* Allow promiscuous mode: **no**
* Allow forged transmits: **no**
* Allow MAC changes: **no**
Port group allvlans:
* VLAN ID: 4095 (allow all tagged VLAN traffic)
* Allow promiscuous mode: **Inherit from vSwitch** (no)
* Allow forged transmits: **Inherit from vSwitch** (no)
* Allow MAC changes: **Inherit from vSwitch** (no)
FreeBSD ESXi replacement setup attempt:
sysrc -f /boot/loader.conf vmm_load=3D"YES"
sysrc -f /boot/loader.conf nmdm_load=3D"YES"
sysrc -f /boot/loader.conf if_bridge_load=3D"YES"
sysrc -f /boot/loader.conf if_tap_load=3D"YES"
sysrc if_vlan_load=3D"YES"
sysrc cloned_interfaces=3D"bridge0 tap0"
sysrc ifconfig_bridge0=3D"addm em0 addm tap0"
echo "net.link.tap.up_on_open=3D1" > /etc/sysctl.d/vm_network.conf
sysrc defaultrouter=3D"192.168.101.1"
sysrc ifconfig_em0=3D"up"
sysrc gateway_enable=3D"YES"
ESXi's *Management Network* equivalent(?) for SSH access:
sysrc vlans_em0=3D"111"
sysrc ifconfig_em0_111=3D"inet 192.168.101.6/24"
Interfaces:
em0: flags=3D8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
metric 0 mtu 1500
options=3D852099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL=
_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
ether 00:25:90:14:95:8c
nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
bridge0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric
0 mtu 1500
ether 02:eb:00:40:63:00
nd6 options=3D9<PERFORMNUD,IFDISABLED>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap0 flags=3D143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 2000000
member: em0 flags=3D143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 2000000
tap0: flags=3D8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
metric 0 mtu 1500
options=3D80000<LINKSTATE>
ether 00:bd:f0:02:f7:00
nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
groups: tap
Opened by PID 45408
em0.111: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric
0 mtu 1500
options=3D103<RXCSUM,TXCSUM,TSO4>
ether 00:25:90:14:95:8c
inet 192.168.101.6 netmask 0xffffff00 broadcast 192.168.101.255
nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 111 vlanpcp: 0 parent interface: em0
groups: vlan
`pciconf -lv`:
em0@pci0:1:0:0: class=3D0x020000 card=3D0x10d315d9 chip=3D0x10d38086
rev=3D0x00 hdr=3D0x00
vendor =3D 'Intel Corporation'
device =3D '82574L Gigabit Network Connection'
class =3D network
subclass =3D ethernet
Running pfSense VM with:
sh /usr/share/examples/bhyve/vmrun.sh -m 2048M -d
/dev/zvol/tank/pfsense0 pfsensevm
It uses `tap0`.
Currently I can access pfSense's web admin but `vmnet0.333` doesn't
get IP from my ISP. I'd like to set VLAN bridging as securely as
possible (`private` for `bridge0`?). If it's possible to send all
traffic to the physical switch first, that's what I would prefer.
openvSwitch is also ok, but I'm not familiar with it.
DHCP request packet is correctly seen in tap0, bridge0, and em0 with
`tcpdump -lnexv -i <iface> "vlan 333"`and not in em0.111, which is
correct when requesting new IP in pfSense. There's no DHCP response
from ISP. em0 and tap0 are in promisc mode which is disabled in
vSwitch version.
More debugging:
Connected Raspberry Pi to my VDSL modem and the DHCP packet is seen. I
also tried spoofing the pfSense's MAC address with RPi and it works.
So possible culprits currently are packet checksum and packet
truncating. At least Linux bridge implementation is very well known to
break packets with bridging and VLANs so FreeBSD might be doing it as
well? Adding VLAN 333 to em0 DHCP works. So what is bridge0 and/or
tap0 doing?
What I am missing?
You can see this same message properly formatted @
https://unix.stackexchange.com/questions/469514/freebsd-ovs-equivalent-of-e=
sxi-vswitch-vlan-config
--=20
Pekka J=C3=A4rvinen
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CABvnMcg6SW02XkO6JnKtUXABvdCXVLj0OQFE-L5v=8S%2BfBZXtw>