From owner-freebsd-virtualization@freebsd.org Sun Dec 4 22:35:29 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 38252C67298 for ; Sun, 4 Dec 2016 22:35:29 +0000 (UTC) (envelope-from jjuanino@gmail.com) Received: from mail-oi0-x231.google.com (mail-oi0-x231.google.com [IPv6:2607:f8b0:4003:c06::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 061241BD3 for ; Sun, 4 Dec 2016 22:35:29 +0000 (UTC) (envelope-from jjuanino@gmail.com) Received: by mail-oi0-x231.google.com with SMTP id v84so320881694oie.3 for ; Sun, 04 Dec 2016 14:35:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=X2oYxmDRFSzkKwrH73ehf3YYO8Q9MNku4Qlz3c6KI3k=; b=oZf5lu+wG6apgUIEePwKx9kP/KRdslwXnXSo4XhoCFEpBtde6OlwxIuGDoKNr5Oxt6 PHFfxaxE2iDlvH5LjtHZcHuXAkGx0r2PdLQ0t42exP2DtBJv/U/oWFvaBh7ewW48lTWc EzNXB9My3rGDB1d+GzHWnm1Gy+ZceGy23PtpmqJJk+nc6x5w9sS3abnMegO0y2RpQtLl vnr53gfTDfC2dXrpdqQgG8iIqbdSFFmfDGiTHszgNBDZWY6IB51q2djWSzmgIGtGsQ40 pWyaqxzB1F9vQQdDCgW+FMUZ4NWHnp8Gg9GB2sEvpYTlyeomV8grVf3f1wqtt8hZKVRJ 5xQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=X2oYxmDRFSzkKwrH73ehf3YYO8Q9MNku4Qlz3c6KI3k=; b=RkX/hpwQcFwbKlcbVUG6inhbCoRG21a29u3n0PDmfCTWksqiqO3jmmUQcrvQYOizrq i0sq7tp4Xcd1yFmsaZPfxLVxZVYgh0Vdn3jdAUez72lKymkorCmSCRbEeZ/1QhuiXVdN riFarvGDWHe6Gho+5tKKQ9eTRlgOrmYyiCY590aQC//+u+ejU6MXS/Y9/qj4mmJFjS0c LXasufnFdFL82IRsWRnKcYdJ0iJ6X2Jk3nBdFNKOivuRRJLU7hR4QB/NsnUyoRszA+RH 5tdNG+zSfye1ObHxnpZ9Odvs1lMXozsNx7xRlBHIAvqdR7oLDqSnnOkaxXsH8itvXJHl rMcw== X-Gm-Message-State: AKaTC02psS6NbN8DPV4q5FX2Pd1h8lSTYshauySilhrMjFavgmEiMZMOuRdqxKZaUtdVs38DjdNP6A14GNsSlw== X-Received: by 10.202.225.85 with SMTP id y82mr28616463oig.209.1480890927842; Sun, 04 Dec 2016 14:35:27 -0800 (PST) MIME-Version: 1.0 Received: by 10.202.49.132 with HTTP; Sun, 4 Dec 2016 14:35:27 -0800 (PST) From: =?UTF-8?B?Sm9zw6kgR2FyY8OtYSBKdWFuaW5v?= Date: Sun, 4 Dec 2016 23:35:27 +0100 Message-ID: Subject: bhyve: cannot send jumbo frames from linux or freebsd guest To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Dec 2016 22:35:29 -0000 Hi everybody, On my FreeBSD 11.0-RELEASE I am running three bhyve virtual machines: 1- Windows Server 2012 2- Centos 6 3- FreeBSD 11.0-RELEASE An iSCSI target is configured in the host, and the three above guests runs a iSCSI initiator. I use iSCSI to present zvols to guests, and so be able to increase the guests storage without rebooting. This setup works as expected: initiators connect to target, and everything run smoothly. In order to boost performance, I want to enable jumbo frames in the private, non routing, iSCSI subnet. The subnet is 192.168.253/24, and is configured as follows: # ifconfig bridge6 bridge6: flags=8843 metric 0 mtu 9000 description: vm-iscsi0 ether 02:10:ef:9e:c0:06 inet 192.168.253.1 netmask 0xffffff00 broadcast 192.168.253.255 nd6 options=1 groups: bridge id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: tap9 flags=943 ifmaxaddr 0 port 22 priority 128 path cost 2000000 member: tap7 flags=943 ifmaxaddr 0 port 20 priority 128 path cost 2000000 member: tap6 flags=943 ifmaxaddr 0 port 19 priority 128 path cost 2000000 192.168.253.1 is the ip address of the FreeBSD bhyve host (the iSCSI target) in this private subnet. Notice that I have set mtu=9000 in the bridge interface. Automatically, the taps interfaces inherit that setting when the guests starts up. Inside each guest, I have configured the mtu setting to 9000. In the FreeBSD and Linux, I get, respectively: # ifconfig vtnet1 vtnet1: flags=8943 metric 0 mtu 9000 options=80028 ether 58:9c:fc:05:55:66 inet 192.168.253.4 netmask 0xffffff00 broadcast 192.168.253.255 nd6 options=29 media: Ethernet 10Gbase-T status: active # ifconfig eth2 eth2 Link encap:Ethernet HWaddr 58:9C:FC:0E:9D:01 inet addr:192.168.253.3 Bcast:192.168.253.255 Mask:255.255.255.0 inet6 addr: fe80::5a9c:fcff:fe0e:9d01/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1 RX packets:19 errors:0 dropped:0 overruns:0 frame:0 TX packets:26 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:65470 (63.9 KiB) TX bytes:108972 (106.4 KiB) In order to test if jumbo frames are set properly, I run the following from the Windows guest: C:\Users\Administrator>ping -n 1 -f -l 8972 192.168.253.1 Pinging 192.168.253.1 with 8972 bytes of data: Reply from 192.168.253.1: bytes=8972 time<1ms TTL=64 Ping statistics for 192.168.253.1: Packets: Sent = 1, Received = 1, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms As you can see, jumbo frames seems to work properly from Windows guest, as I can trasmit a 8972 bytes frame with no fragmentation. A tcpdump in the host side shows the following: 23:22:42.111043 IP (tos 0x0, ttl 128, id 20704, offset 0, flags [DF], proto ICMP (1), length 9000) 192.168.253.2 > 192.168.253.1: ICMP echo request, id 1, seq 301, length 8980 23:22:42.111106 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 9000) 192.168.253.1 > 192.168.253.2: ICMP echo reply, id 1, seq 301, length 8980 But, If I repeat the analogous test from FreeBSD or Linux guest, I get wrong results, as I will explain. >From Linux guest I run: # ping -c 4 -s 8972 -M do 192.168.253.1 PING 192.168.253.1 (192.168.253.1) 8972(9000) bytes of data. --- 192.168.253.1 ping statistics --- 4 packets transmitted, 0 received, 100% packet loss, time 13000ms If I decrease the packet size to 4042, I get a succesful answer from the iSCSI target: # ping -c 4 -s 4042 -M do 192.168.253.1 PING 192.168.253.1 (192.168.253.1) 4042(4070) bytes of data. 4050 bytes from 192.168.253.1: icmp_seq=1 ttl=64 time=0.191 ms 4050 bytes from 192.168.253.1: icmp_seq=2 ttl=64 time=0.203 ms 4050 bytes from 192.168.253.1: icmp_seq=3 ttl=64 time=0.159 ms 4050 bytes from 192.168.253.1: icmp_seq=4 ttl=64 time=0.191 ms --- 192.168.253.1 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3000ms rtt min/avg/max/mdev = 0.159/0.186/0.203/0.016 ms With a 4043 packet size, the ping fails again. >From FreeBSD guest, the test would be: # ping -c 4 -D -s 8972 192.168.253.1 PING 192.168.253.1 (192.168.253.1): 8972 data bytes --- 192.168.253.1 ping statistics --- 4 packets transmitted, 0 packets received, 100.0% packet loss After some trial and error, I get 1994 as the upper bound: # ping -c 4 -D -s 1994 192.168.253.1 PING 192.168.253.1 (192.168.253.1): 1994 data bytes 2002 bytes from 192.168.253.1: icmp_seq=0 ttl=64 time=0.203 ms 2002 bytes from 192.168.253.1: icmp_seq=1 ttl=64 time=0.239 ms 2002 bytes from 192.168.253.1: icmp_seq=2 ttl=64 time=0.224 ms 2002 bytes from 192.168.253.1: icmp_seq=3 ttl=64 time=0.214 ms --- 192.168.253.1 ping statistics --- 4 packets transmitted, 4 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.203/0.220/0.239/0.013 ms I have run tcpdump on both FreeBSD guest and host to debug the issue and I get the following: * From FreeBSD guest: # tcpdump -vvn -i vtnet1 & tcpdump: listening on vtnet1, link-type EN10MB (Ethernet), capture size 262144 bytes # ping -c 1 -D -s 1995 192.168.253.1 PING 192.168.253.1 (192.168.253.1): 1995 data bytes 23:08:46.808187 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 2023) 192.168.253.4 > 192.168.253.1: ICMP echo request, id 58116, seq 0, length 2003 23:08:46.808430 IP truncated-ip - 1 bytes missing! (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 2023) 192.168.253.1 > 192.168.253.4: ICMP echo reply, id 58116, seq 0, length 2003 --- 192.168.253.1 ping statistics --- 1 packets transmitted, 0 packets received, 100.0% packet loss * From FreeBSD host: # tcpdump -vvn -i bridge6 tcpdump: listening on bridge6, link-type EN10MB (Ethernet), capture size 262144 bytes 23:09:58.315600 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 2023) 192.168.253.4 > 192.168.253.1: ICMP echo request, id 59140, seq 0, length 2003 23:09:58.315663 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 2023) 192.168.253.1 > 192.168.253.4: ICMP echo reply, id 59140, seq 0, length 2003 Notice the "truncated-ip - 1 bytes missing!" from the guest. Indeed, is clear that the issue is related to the Linux or FreeBSD guest, as from Windows guest I cant transmit jumbo frames. Could anyone explain why I cannot send jumbo frames from Linux and FreeBSD guests, but I *can* send them from Windows guest? Am I missing o missunderstanding something? Thanks in advanced, any comment o suggestion will be wellcome. Best regards.