Date: Sun, 4 Dec 2016 23:35:27 +0100 From: =?UTF-8?B?Sm9zw6kgR2FyY8OtYSBKdWFuaW5v?= <jjuanino@gmail.com> To: freebsd-virtualization@freebsd.org Subject: bhyve: cannot send jumbo frames from linux or freebsd guest Message-ID: <CAAVO5%2BJnk1MepU7mBdQiWsRJf4FF0EuheaERGCHJJw3_pgG_dA@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hi everybody,
On my FreeBSD 11.0-RELEASE I am running three bhyve virtual machines:
1- Windows Server 2012
2- Centos 6
3- FreeBSD 11.0-RELEASE
An iSCSI target is configured in the host, and the three above guests runs a
iSCSI initiator. I use iSCSI to present zvols to guests, and so be
able to increase
the guests storage without rebooting. This setup works as expected: initiators
connect to target, and everything run smoothly.
In order to boost performance, I want to enable jumbo frames in the private,
non routing, iSCSI subnet. The subnet is 192.168.253/24, and is configured
as follows:
# ifconfig bridge6
bridge6: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: vm-iscsi0
ether 02:10:ef:9e:c0:06
inet 192.168.253.1 netmask 0xffffff00 broadcast 192.168.253.255
nd6 options=1<PERFORMNUD>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap9 flags=943<LEARNING,DISCOVER,PRIVATE,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 22 priority 128 path cost 2000000
member: tap7 flags=943<LEARNING,DISCOVER,PRIVATE,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 20 priority 128 path cost 2000000
member: tap6 flags=943<LEARNING,DISCOVER,PRIVATE,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 19 priority 128 path cost 2000000
192.168.253.1 is the ip address of the FreeBSD bhyve host (the iSCSI target) in
this private subnet. Notice that I have set mtu=9000 in the bridge interface.
Automatically, the taps interfaces inherit that setting when the guests starts
up.
Inside each guest, I have configured the mtu setting to 9000. In the FreeBSD and
Linux, I get, respectively:
# ifconfig vtnet1
vtnet1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
metric 0 mtu 9000
options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether 58:9c:fc:05:55:66
inet 192.168.253.4 netmask 0xffffff00 broadcast 192.168.253.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T <full-duplex>
status: active
# ifconfig eth2
eth2 Link encap:Ethernet HWaddr 58:9C:FC:0E:9D:01
inet addr:192.168.253.3 Bcast:192.168.253.255 Mask:255.255.255.0
inet6 addr: fe80::5a9c:fcff:fe0e:9d01/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:19 errors:0 dropped:0 overruns:0 frame:0
TX packets:26 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:65470 (63.9 KiB) TX bytes:108972 (106.4 KiB)
In order to test if jumbo frames are set properly, I run the following from the
Windows guest:
C:\Users\Administrator>ping -n 1 -f -l 8972 192.168.253.1
Pinging 192.168.253.1 with 8972 bytes of data:
Reply from 192.168.253.1: bytes=8972 time<1ms TTL=64
Ping statistics for 192.168.253.1:
Packets: Sent = 1, Received = 1, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
As you can see, jumbo frames seems to work properly from Windows guest, as
I can trasmit a 8972 bytes frame with no fragmentation. A tcpdump in the host
side shows the following:
23:22:42.111043 IP (tos 0x0, ttl 128, id 20704, offset 0, flags [DF],
proto ICMP (1), length 9000)
192.168.253.2 > 192.168.253.1: ICMP echo request, id 1, seq 301, length 8980
23:22:42.111106 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto
ICMP (1), length 9000)
192.168.253.1 > 192.168.253.2: ICMP echo reply, id 1, seq 301, length 8980
But, If I repeat the analogous test from FreeBSD or Linux guest, I get wrong
results, as I will explain.
>From Linux guest I run:
# ping -c 4 -s 8972 -M do 192.168.253.1
PING 192.168.253.1 (192.168.253.1) 8972(9000) bytes of data.
--- 192.168.253.1 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 13000ms
If I decrease the packet size to 4042, I get a succesful answer from the iSCSI
target:
# ping -c 4 -s 4042 -M do 192.168.253.1
PING 192.168.253.1 (192.168.253.1) 4042(4070) bytes of data.
4050 bytes from 192.168.253.1: icmp_seq=1 ttl=64 time=0.191 ms
4050 bytes from 192.168.253.1: icmp_seq=2 ttl=64 time=0.203 ms
4050 bytes from 192.168.253.1: icmp_seq=3 ttl=64 time=0.159 ms
4050 bytes from 192.168.253.1: icmp_seq=4 ttl=64 time=0.191 ms
--- 192.168.253.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3000ms
rtt min/avg/max/mdev = 0.159/0.186/0.203/0.016 ms
With a 4043 packet size, the ping fails again.
>From FreeBSD guest, the test would be:
# ping -c 4 -D -s 8972 192.168.253.1
PING 192.168.253.1 (192.168.253.1): 8972 data bytes
--- 192.168.253.1 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
After some trial and error, I get 1994 as the upper bound:
# ping -c 4 -D -s 1994 192.168.253.1
PING 192.168.253.1 (192.168.253.1): 1994 data bytes
2002 bytes from 192.168.253.1: icmp_seq=0 ttl=64 time=0.203 ms
2002 bytes from 192.168.253.1: icmp_seq=1 ttl=64 time=0.239 ms
2002 bytes from 192.168.253.1: icmp_seq=2 ttl=64 time=0.224 ms
2002 bytes from 192.168.253.1: icmp_seq=3 ttl=64 time=0.214 ms
--- 192.168.253.1 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.203/0.220/0.239/0.013 ms
I have run tcpdump on both FreeBSD guest and host to debug the issue and I get
the following:
* From FreeBSD guest:
# tcpdump -vvn -i vtnet1 &
tcpdump: listening on vtnet1, link-type EN10MB (Ethernet), capture
size 262144 bytes
# ping -c 1 -D -s 1995 192.168.253.1
PING 192.168.253.1 (192.168.253.1): 1995 data bytes
23:08:46.808187 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto
ICMP (1), length 2023)
192.168.253.4 > 192.168.253.1: ICMP echo request, id 58116, seq 0,
length 2003
23:08:46.808430 IP truncated-ip - 1 bytes missing! (tos 0x0, ttl 64,
id 0, offset 0, flags [DF], proto ICMP (1), length 2023)
192.168.253.1 > 192.168.253.4: ICMP echo reply, id 58116, seq 0, length 2003
--- 192.168.253.1 ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss
* From FreeBSD host:
# tcpdump -vvn -i bridge6
tcpdump: listening on bridge6, link-type EN10MB (Ethernet), capture
size 262144 bytes
23:09:58.315600 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto
ICMP (1), length 2023)
192.168.253.4 > 192.168.253.1: ICMP echo request, id 59140, seq 0,
length 2003
23:09:58.315663 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto
ICMP (1), length 2023)
192.168.253.1 > 192.168.253.4: ICMP echo reply, id 59140, seq 0, length 2003
Notice the "truncated-ip - 1 bytes missing!" from the guest. Indeed, is clear
that the issue is related to the Linux or FreeBSD guest, as from Windows guest
I cant transmit jumbo frames.
Could anyone explain why I cannot send jumbo frames from Linux and FreeBSD
guests, but I *can* send them from Windows guest? Am I missing o
missunderstanding something? Thanks in advanced, any comment o suggestion will
be wellcome. Best regards.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAAVO5%2BJnk1MepU7mBdQiWsRJf4FF0EuheaERGCHJJw3_pgG_dA>