From owner-freebsd-xen@freebsd.org Thu Jul 4 07:25:17 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1CCCF15CAF35; Thu, 4 Jul 2019 07:25:17 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received: from esa4.hc3370-68.iphmx.com (esa4.hc3370-68.iphmx.com [216.71.155.144]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "esa4.hc3370-68.iphmx.com", Issuer "HydrantID SSL ICA G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1E65084982; Thu, 4 Jul 2019 07:25:14 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received-SPF: None (esa4.hc3370-68.iphmx.com: no sender authenticity information available from domain of roger.pau@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa4.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa4.hc3370-68.iphmx.com: domain of roger.pau@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa4.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ~all" Received-SPF: None (esa4.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa4.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: GndjarNKt2286LNYUFsbmBCGHDQ60bE6p39594iQQ8ZCUtdt+F0doHOYdZz7Q1YdG6PRtuX3Mf cBBGQ8eyCOluBPnX7YYLVyUqFrNkElap9B2wCa74GwLepVJgIgcDG50juMPhH4WS6eeIu/zrKT w6GPNKYv1LxVtiiOOwpF3pev+9u4Zww68utunGtr3wLbLmeS/XTbQ0+lF+Z2HIF5MlzNZU9wlY /HXy2OBaMhBTUMKDFdYYLqcIFMckWEpOQ5NfbZxZuYRPnnagNfyFt6o3gTnoa51l+MqDE6/HHs oIc= X-SBRS: 2.7 X-MesageID: 2637404 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.63,449,1557201600"; d="scan'208";a="2637404" Date: Thu, 4 Jul 2019 09:23:54 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Christian M CC: Eugene Grosbein , , Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer Message-ID: <20190704072354.6ycduisiidz5iy2x@Air-de-Roger> References: <9c377fe7-5546-9e0b-fac9-076dd094cbac@grosbein.net> <47bb1bfe-ac39-78d3-c9b9-17ff1896e0ec@grosbein.net> <20190627101901.3szaaxhp4xzjiay5@MacBook-Air-de-Roger.local> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: NeoMutt/20180716 X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-Rspamd-Queue-Id: 1E65084982 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of roger.pau@citrix.com designates 216.71.155.144 as permitted sender) smtp.mailfrom=roger.pau@citrix.com X-Spamd-Result: default: False [-3.67 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip4:216.71.155.144]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[citrix.com]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com]; NEURAL_HAM_SHORT(-0.91)[-0.907,0]; IP_SCORE(-0.85)[ip: (-1.24), ipnet: 216.71.154.0/23(-1.81), asn: 16417(-1.15), country: US(-0.06)]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_COUNT_ZERO(0.00)[0]; RCVD_IN_DNSWL_LOW(-0.10)[144.155.71.216.list.dnswl.org : 127.0.3.1]; R_DKIM_NA(0.00)[]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Jul 2019 07:25:17 -0000 On Thu, Jun 27, 2019 at 12:31:35PM +0200, Christian M wrote: > Den tors 27 juni 2019 kl 12:19 skrev Roger Pau Monné : > > > On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote: > > > I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and .128. > > VIF > > > cheksum offloading is turned off, and -txcsum for xn0 for both VM's. > > > > > > I feel the throughput is more consistent now, not all over the place as > > > before, even between runs. But the Retr column (tcp retries) in iperf3 > > has > > > jumped up considerably from hundreds/s to thousands/s. > > > > > > Just a reminder, I have tested this with 11.0-RELEASE also, where the > > issue > > > appeared first for me. 10.4-RELEASE is as fast as I could expect it to > > be, > > > and 0 retries. > > > > > > 12.0-STABLE: > > > > > > Connecting to host 172.31.16.128, port 5201 > > > [ 5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port 5201 > > > [ ID] Interval Transfer Bitrate Retr Cwnd > > > [ 5] 0.00-1.00 sec 96.3 MBytes 808 Mbits/sec 2401 2.85 KBytes > > > > > > [ 5] 1.00-2.00 sec 118 MBytes 991 Mbits/sec 3120 17.0 KBytes > > > > > > [ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 3203 69.8 KBytes > > > > > > [ 5] 3.00-4.00 sec 102 MBytes 853 Mbits/sec 3126 15.6 KBytes > > > > > > [ 5] 4.00-5.00 sec 110 MBytes 921 Mbits/sec 2890 15.6 KBytes > > > > > > [ 5] 5.00-6.00 sec 108 MBytes 908 Mbits/sec 3308 17.0 KBytes > > > > > > [ 5] 6.00-7.00 sec 104 MBytes 869 Mbits/sec 3046 48.2 KBytes > > > > > > [ 5] 7.00-8.00 sec 98.9 MBytes 830 Mbits/sec 2845 2.85 KBytes > > > > > > [ 5] 8.00-9.00 sec 104 MBytes 874 Mbits/sec 2711 86.8 KBytes > > > > > > [ 5] 9.00-10.00 sec 108 MBytes 904 Mbits/sec 2696 14.2 KBytes > > > > > > [ 5] 10.00-11.00 sec 103 MBytes 864 Mbits/sec 2660 31.3 KBytes > > > > > > [ 5] 11.00-12.00 sec 98.8 MBytes 828 Mbits/sec 2476 19.9 KBytes > > > > > > [ 5] 12.00-13.00 sec 99.9 MBytes 838 Mbits/sec 2857 11.3 KBytes > > > > > > [ 5] 13.00-14.00 sec 107 MBytes 894 Mbits/sec 2685 24.1 KBytes > > > > > > [ 5] 14.00-15.00 sec 114 MBytes 953 Mbits/sec 2321 25.5 KBytes > > > > > > [ 5] 15.00-16.00 sec 93.1 MBytes 781 Mbits/sec 2427 48.3 KBytes > > > > > > [ 5] 16.00-17.00 sec 107 MBytes 895 Mbits/sec 2219 29.8 KBytes > > > > > > [ 5] 17.00-18.00 sec 92.5 MBytes 776 Mbits/sec 2441 12.8 KBytes > > > > > > [ 5] 18.00-19.00 sec 116 MBytes 976 Mbits/sec 2840 38.2 KBytes > > > > > > [ 5] 19.00-20.00 sec 102 MBytes 853 Mbits/sec 2573 43.9 KBytes > > > > > > - - - - - - - - - - - - - - - - - - - - - - - - - > > > [ ID] Interval Transfer Bitrate Retr > > > [ 5] 0.00-20.00 sec 2.05 GBytes 882 Mbits/sec 54845 > > > > Can you paste the output of ifconfig for both the interfaces used in > > the test? > > > > Are you sure all hardware offloading capabilities are turned off on > > both interfaces? > > > > Can you check what's causing those retries? > > > > Either using tcpdump, whireshark or some other tool to analyze the > > network traffic and detect the errors that cause such retries? > > > > Thanks, Roger. > > > > 172.31.16.127 (12.0-STABLE): > > lo0: flags=8049 metric 0 mtu 16384 > options=680003 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 > inet 127.0.0.1 netmask 0xff000000 > groups: lo > nd6 options=21 > xn0: flags=8843 metric 0 mtu 1500 > options=501 I would try to disable rxcsum, tso4 and lro also. > ether 6e:83:99:ed:ce:f7 > inet 172.31.16.127 netmask 0xffffff00 broadcast 172.31.16.255 > media: Ethernet manual > status: active > nd6 options=29 > > ethtool -k vif68.0 > Features for vif68.0: > rx-checksumming: on [fixed] > tx-checksumming: off > tx-checksum-ipv4: off > tx-checksum-ip-generic: off [fixed] > tx-checksum-ipv6: off > tx-checksum-fcoe-crc: off [fixed] > tx-checksum-sctp: off [fixed] > scatter-gather: off > tx-scatter-gather: off > tx-scatter-gather-fraglist: off > tcp-segmentation-offload: off > tx-tcp-segmentation: off > tx-tcp-ecn-segmentation: off [fixed] > tx-tcp6-segmentation: off > udp-fragmentation-offload: off [fixed] > generic-segmentation-offload: off > generic-receive-offload: on > large-receive-offload: off [fixed] > rx-vlan-offload: off [fixed] > tx-vlan-offload: off [fixed] > ntuple-filters: off [fixed] > receive-hashing: off [fixed] > highdma: off [fixed] > rx-vlan-filter: off [fixed] > vlan-challenged: off [fixed] > tx-lockless: off [fixed] > netns-local: off [fixed] > tx-gso-robust: off [fixed] > tx-fcoe-segmentation: off [fixed] > tx-gre-segmentation: off [fixed] > tx-ipip-segmentation: off [fixed] > tx-sit-segmentation: off [fixed] > tx-udp_tnl-segmentation: off [fixed] > fcoe-mtu: off [fixed] > tx-nocache-copy: off > loopback: off [fixed] > rx-fcs: off [fixed] > rx-all: off [fixed] > tx-vlan-stag-hw-insert: off [fixed] > rx-vlan-stag-hw-parse: off [fixed] > rx-vlan-stag-filter: off [fixed] > l2-fwd-offload: off [fixed] > busy-poll: off [fixed] > > 172.31.16.128 (12.0-STABLE): > > lo0: flags=8049 metric 0 mtu 16384 > options=680003 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 > inet 127.0.0.1 netmask 0xff000000 > groups: lo > nd6 options=21 > xn0: flags=8843 metric 0 mtu 1500 > options=501 > ether 32:a7:9f:cc:94:8a > inet 172.31.16.128 netmask 0xffffff00 broadcast 172.31.16.255 > media: Ethernet manual > status: active > nd6 options=29 > > ethtool -k vif67.0 > Features for vif67.0: > rx-checksumming: on [fixed] > tx-checksumming: off > tx-checksum-ipv4: off > tx-checksum-ip-generic: off [fixed] > tx-checksum-ipv6: off > tx-checksum-fcoe-crc: off [fixed] > tx-checksum-sctp: off [fixed] > scatter-gather: off > tx-scatter-gather: off > tx-scatter-gather-fraglist: off > tcp-segmentation-offload: off > tx-tcp-segmentation: off > tx-tcp-ecn-segmentation: off [fixed] > tx-tcp6-segmentation: off > udp-fragmentation-offload: off [fixed] > generic-segmentation-offload: off > generic-receive-offload: on > large-receive-offload: off [fixed] > rx-vlan-offload: off [fixed] > tx-vlan-offload: off [fixed] > ntuple-filters: off [fixed] > receive-hashing: off [fixed] > highdma: off [fixed] > rx-vlan-filter: off [fixed] > vlan-challenged: off [fixed] > tx-lockless: off [fixed] > netns-local: off [fixed] > tx-gso-robust: off [fixed] > tx-fcoe-segmentation: off [fixed] > tx-gre-segmentation: off [fixed] > tx-ipip-segmentation: off [fixed] > tx-sit-segmentation: off [fixed] > tx-udp_tnl-segmentation: off [fixed] > fcoe-mtu: off [fixed] > tx-nocache-copy: off > loopback: off [fixed] > rx-fcs: off [fixed] > rx-all: off [fixed] > tx-vlan-stag-hw-insert: off [fixed] > rx-vlan-stag-hw-parse: off [fixed] > rx-vlan-stag-filter: off [fixed] > l2-fwd-offload: off [fixed] > busy-poll: off [fixed] > > Should I run tcpdump on xen host, or on FreeBSD while running iperf3? On the Xen host would be better, since that tcpdump will read the packets as FreeBSD puts them on the wire. > As I side note, I installed XCP-ng 7.6.0 on a older server I had laying > around just to see how 12.0-RELEASE behaved there, and It was the same > thing, super slow between VM's. Yes, I don't think this is related to hardware at all. I will try to find some time to look into this, but it's not going to be soon since I'm currently preparing a talk for a conference I'm attending next week, so I won't likely be able to get to it until the end of July, sorry. Also, I'm no networking expert, so I cannot assure my success in fixing this. As a workaround you can switch to the emulated network card by setting 'hw.xen.disable_pv_nics=1' in /boot/loader.conf. That will give you worse performance than a fully working PV network card, but at least should be consistent. There are others that have switched to virtio-net, but I have no idea how to do that with XCP. Roger. From owner-freebsd-xen@freebsd.org Thu Jul 4 07:27:52 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F41D715CB112 for ; Thu, 4 Jul 2019 07:27:51 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received: from esa2.hc3370-68.iphmx.com (esa2.hc3370-68.iphmx.com [216.71.145.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "esa2.hc3370-68.iphmx.com", Issuer "HydrantID SSL ICA G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D29AF84B49 for ; Thu, 4 Jul 2019 07:27:50 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received-SPF: None (esa2.hc3370-68.iphmx.com: no sender authenticity information available from domain of roger.pau@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa2.hc3370-68.iphmx.com: domain of roger.pau@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ~all" Received-SPF: None (esa2.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: nKUpxTXNtJl6yViG0fB8RPtuNv+TcgWcSucnUlEbaMqcy6UoOFhh8TEFV271BTV/xlK6FKuc9s 2R4KhQ2QPqkpHpuDFz3Vl/ORL1mMYiEkA3SU5jjcdDpoDLzjAoN6lHf+vJc4vUHxqmps0EWzlU rtO8iOw/Vk9sFzTd9RpMnBhmYU2OWBvc60ygKii6AArhvZLhRke0dV9EzYi+ZMTk9be9UuxBCu fan93x9tkYZIo59pwuHuXNeVsWDGQeboZMz7JB0adRFjzLnaQykyW2OpXt7TA8GnlVNyiuWkyU 2wo= X-SBRS: 2.7 X-MesageID: 2576375 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.63,449,1557201600"; d="scan'208";a="2576375" Date: Thu, 4 Jul 2019 09:26:29 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: R0me0 *** CC: Subject: Re: Xen 4.12 - systemrescuecd-6.0.3.iso - Baremetal reboots Message-ID: <20190704072629.v4qegclh7ljfdt64@Air-de-Roger> References: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: User-Agent: NeoMutt/20180716 X-ClientProxiedBy: AMSPEX02CAS01.citrite.net (10.69.22.112) To AMSPEX02CL02.citrite.net (10.69.22.126) X-Rspamd-Queue-Id: D29AF84B49 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of roger.pau@citrix.com designates 216.71.145.153 as permitted sender) smtp.mailfrom=roger.pau@citrix.com X-Spamd-Result: default: False [-3.17 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.995,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:216.71.145.153]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[citrix.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: mx1.hc3370-68.iphmx.com]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.73)[-0.725,0]; IP_SCORE(-0.54)[ip: (-1.24), ipnet: 216.71.145.0/24(-0.27), asn: 16417(-1.15), country: US(-0.06)]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_COUNT_ZERO(0.00)[0]; RCVD_IN_DNSWL_LOW(-0.10)[153.145.71.216.list.dnswl.org : 127.0.3.1]; R_DKIM_NA(0.00)[]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Jul 2019 07:27:52 -0000 On Fri, Jun 28, 2019 at 12:02:37PM +0000, R0me0 *** wrote: > Guys, > > I'am using FreeBSD 12.0-RELEASE-p6 FreeBSD 12.0-RELEASE-p6 GENERIC amd64 I guess this is on the host? > When booting systemrescuecd-6.0.3.iso ( http://www.system-rescue-cd.org ) > , my system just hangs and reboot. So you are trying to boot systemrescuecd as a guest on the Xen host? Can you paste your guest config file? > Just tried an older version and it's ok. > > I don't have any log to help. If you guys want and some help, I can provide > more information. Could you get a serial hooked up to that box so we can get the trace? Without some kind of trace it's going to be hard to debug this. Thanks, Roger. From owner-freebsd-xen@freebsd.org Thu Jul 4 07:58:23 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9858215CBE01; Thu, 4 Jul 2019 07:58:23 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: from mail-io1-xd2c.google.com (mail-io1-xd2c.google.com [IPv6:2607:f8b0:4864:20::d2c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B0D2085AA1; Thu, 4 Jul 2019 07:58:22 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: by mail-io1-xd2c.google.com with SMTP id w25so10985483ioc.8; Thu, 04 Jul 2019 00:58:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ZlRMaCe0KI/YXXRnuRRHklytOAp0JrqXFp/kEl/fZNY=; b=pAjOlkxgQggihzW+VBa6OiADZPLHbAGldc22JWwEc1VFuQb2N/PStjfctKHeBjuBHL x+a3JetSbhIEc6/0PLvm7PcHNpdlrS3Oc47KIe2e1FjpfiwT3KofUliqWB8zhMvFz8lN th6eRBdLMYr7VMNOQsGzn8k4UwirOPKywsbHjSHWwNQ7d+Fo2+FG5uOAP9brLHw9WCRm QmumBhA47BCDowkWEqoskkI4nhsX+xdt1l6IK7vPBLWlQMRY82PUU70fybWZwKFdSRaV 8pO2eCBTbEPk+i7TZGWTPIcYEfGpzwTi3VTM8riRj6r43c2n7q5EpVtUuWseVs66b7Ql jEVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ZlRMaCe0KI/YXXRnuRRHklytOAp0JrqXFp/kEl/fZNY=; b=jTcZ+/kodAWZCytdGvTXYTl+T/YuwfjlyK6Q95KV2UJdvF08zqi5ITZRLJzFOXI8H3 lOhNDJ6SbaTf9FxslL7P5gYNXVuk/yNwF24TkMsyMmZW5ewnYrBMU6gjw3fcG8ziaBVu W0JhtCSAsotCqe8QZvusmCBVrAlBwshCh6nKoL7QEMefuaHL8Ljna5AU7tTdWW+XLARP DFCSodkl5NcfijuVfIIpaHD8g64gXxyN4XlQE1ZDSVbelObzDsOUZHE9Q8ayGybsVbml u/6PTZQr3x6nKIUrRiPmikog7JZTBqgeZDpESB9pEQrlG9YGhMp2Ix4g0VHdXJhElsgd ovWw== X-Gm-Message-State: APjAAAUGJadUbd6bkYAILZNNt9qU+pp2oW67pZbi1ro2Mzvej1OUTpt+ M2Npah5gSic2nNJu5dT/dyyhwhlOxALtyHpVPEk= X-Google-Smtp-Source: APXvYqxbAGtrkwxE0kq+GS4RZD6Czdqu9KCfY99f1F5y7V5wctI9pOpovJSv+17AFe7uKIbQk4Q1pk/c69dWEaX5Hxg= X-Received: by 2002:a02:710f:: with SMTP id n15mr46377043jac.119.1562227101773; Thu, 04 Jul 2019 00:58:21 -0700 (PDT) MIME-Version: 1.0 References: <9c377fe7-5546-9e0b-fac9-076dd094cbac@grosbein.net> <47bb1bfe-ac39-78d3-c9b9-17ff1896e0ec@grosbein.net> <20190627101901.3szaaxhp4xzjiay5@MacBook-Air-de-Roger.local> <20190704072354.6ycduisiidz5iy2x@Air-de-Roger> In-Reply-To: <20190704072354.6ycduisiidz5iy2x@Air-de-Roger> From: Christian M Date: Thu, 4 Jul 2019 09:58:10 +0200 Message-ID: Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Cc: Eugene Grosbein , freebsd-xen@freebsd.org, freebsd-net@freebsd.org X-Rspamd-Queue-Id: B0D2085AA1 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=pAjOlkxg; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of christianmarcos@gmail.com designates 2607:f8b0:4864:20::d2c as permitted sender) smtp.mailfrom=christianmarcos@gmail.com X-Spamd-Result: default: False [-6.90 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[c.2.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.92)[ip: (-8.99), ipnet: 2607:f8b0::/32(-3.16), asn: 15169(-2.39), country: US(-0.06)]; NEURAL_HAM_SHORT(-0.97)[-0.969,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Jul 2019 07:58:23 -0000 Thanks Roger. I disabled pv nic entirely on my two 12.0-RELEASE test VM's. I got 1000baseT full duplex auto selected, so I expected throughput close to that, but to my surprise this are the results: $ ifconfig em0: flags=3D8843 metric 0 mtu 1500 options=3D812099 ether 56:65:6f:f3:02:fb inet 172.31.16.125 netmask 0xffffff00 broadcast 172.31.16.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=3D29 lo0: flags=3D8049 metric 0 mtu 16384 options=3D680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=3D21 $ iperf3 -c 172.31.16.126 Connecting to host 172.31.16.126, port 5201 [ 5] local 172.31.16.125 port 11247 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 6.50 MBytes 54.5 Mbits/sec 0 368 KBytes [ 5] 1.00-2.00 sec 7.22 MBytes 60.6 Mbits/sec 0 368 KBytes [ 5] 2.00-3.00 sec 7.26 MBytes 60.9 Mbits/sec 0 368 KBytes [ 5] 3.00-4.00 sec 7.09 MBytes 59.4 Mbits/sec 0 368 KBytes [ 5] 4.00-5.00 sec 7.32 MBytes 61.4 Mbits/sec 0 368 KBytes [ 5] 5.00-6.00 sec 7.22 MBytes 60.6 Mbits/sec 0 385 KBytes [ 5] 6.00-7.00 sec 7.24 MBytes 60.7 Mbits/sec 0 385 KBytes [ 5] 7.00-8.00 sec 7.28 MBytes 61.1 Mbits/sec 0 385 KBytes [ 5] 8.00-9.00 sec 7.55 MBytes 63.3 Mbits/sec 0 385 KBytes [ 5] 9.00-10.00 sec 7.12 MBytes 59.7 Mbits/sec 0 385 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 71.8 MBytes 60.2 Mbits/sec 0 sende= r [ 5] 0.00-10.06 sec 71.8 MBytes 59.9 Mbits/sec receiver iperf Done. $ iperf3 -c 172.31.16.126 -R Connecting to host 172.31.16.126, port 5201 Reverse mode, remote host 172.31.16.126 is sending [ 5] local 172.31.16.125 port 22443 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 5.06 MBytes 42.5 Mbits/sec [ 5] 1.00-2.00 sec 5.60 MBytes 47.0 Mbits/sec [ 5] 2.00-3.00 sec 6.22 MBytes 52.2 Mbits/sec [ 5] 3.00-4.00 sec 6.92 MBytes 58.0 Mbits/sec [ 5] 4.00-5.00 sec 7.39 MBytes 62.0 Mbits/sec [ 5] 5.00-6.00 sec 7.17 MBytes 60.1 Mbits/sec [ 5] 6.00-7.00 sec 7.18 MBytes 60.3 Mbits/sec [ 5] 7.00-8.00 sec 7.32 MBytes 61.4 Mbits/sec [ 5] 8.00-9.00 sec 6.94 MBytes 58.2 Mbits/sec [ 5] 9.00-10.00 sec 7.04 MBytes 59.0 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 67.3 MBytes 56.5 Mbits/sec 0 sende= r [ 5] 0.00-10.00 sec 66.8 MBytes 56.1 Mbits/sec receiver iperf Done. Good luck with your talk. Regards, Christian Den tors 4 juli 2019 kl 09:24 skrev Roger Pau Monn=C3=A9 : > On Thu, Jun 27, 2019 at 12:31:35PM +0200, Christian M wrote: > > Den tors 27 juni 2019 kl 12:19 skrev Roger Pau Monn=C3=A9 < > roger.pau@citrix.com>: > > > > > On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote: > > > > I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and > .128. > > > VIF > > > > cheksum offloading is turned off, and -txcsum for xn0 for both VM's= . > > > > > > > > I feel the throughput is more consistent now, not all over the plac= e > as > > > > before, even between runs. But the Retr column (tcp retries) in > iperf3 > > > has > > > > jumped up considerably from hundreds/s to thousands/s. > > > > > > > > Just a reminder, I have tested this with 11.0-RELEASE also, where t= he > > > issue > > > > appeared first for me. 10.4-RELEASE is as fast as I could expect it > to > > > be, > > > > and 0 retries. > > > > > > > > 12.0-STABLE: > > > > > > > > Connecting to host 172.31.16.128, port 5201 > > > > [ 5] local 172.31.16.127 port 16833 connected to 172.31.16.128 por= t > 5201 > > > > [ ID] Interval Transfer Bitrate Retr Cwnd > > > > [ 5] 0.00-1.00 sec 96.3 MBytes 808 Mbits/sec 2401 2.85 > KBytes > > > > > > > > [ 5] 1.00-2.00 sec 118 MBytes 991 Mbits/sec 3120 17.0 > KBytes > > > > > > > > [ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 3203 69.8 > KBytes > > > > > > > > [ 5] 3.00-4.00 sec 102 MBytes 853 Mbits/sec 3126 15.6 > KBytes > > > > > > > > [ 5] 4.00-5.00 sec 110 MBytes 921 Mbits/sec 2890 15.6 > KBytes > > > > > > > > [ 5] 5.00-6.00 sec 108 MBytes 908 Mbits/sec 3308 17.0 > KBytes > > > > > > > > [ 5] 6.00-7.00 sec 104 MBytes 869 Mbits/sec 3046 48.2 > KBytes > > > > > > > > [ 5] 7.00-8.00 sec 98.9 MBytes 830 Mbits/sec 2845 2.85 > KBytes > > > > > > > > [ 5] 8.00-9.00 sec 104 MBytes 874 Mbits/sec 2711 86.8 > KBytes > > > > > > > > [ 5] 9.00-10.00 sec 108 MBytes 904 Mbits/sec 2696 14.2 > KBytes > > > > > > > > [ 5] 10.00-11.00 sec 103 MBytes 864 Mbits/sec 2660 31.3 > KBytes > > > > > > > > [ 5] 11.00-12.00 sec 98.8 MBytes 828 Mbits/sec 2476 19.9 > KBytes > > > > > > > > [ 5] 12.00-13.00 sec 99.9 MBytes 838 Mbits/sec 2857 11.3 > KBytes > > > > > > > > [ 5] 13.00-14.00 sec 107 MBytes 894 Mbits/sec 2685 24.1 > KBytes > > > > > > > > [ 5] 14.00-15.00 sec 114 MBytes 953 Mbits/sec 2321 25.5 > KBytes > > > > > > > > [ 5] 15.00-16.00 sec 93.1 MBytes 781 Mbits/sec 2427 48.3 > KBytes > > > > > > > > [ 5] 16.00-17.00 sec 107 MBytes 895 Mbits/sec 2219 29.8 > KBytes > > > > > > > > [ 5] 17.00-18.00 sec 92.5 MBytes 776 Mbits/sec 2441 12.8 > KBytes > > > > > > > > [ 5] 18.00-19.00 sec 116 MBytes 976 Mbits/sec 2840 38.2 > KBytes > > > > > > > > [ 5] 19.00-20.00 sec 102 MBytes 853 Mbits/sec 2573 43.9 > KBytes > > > > > > > > - - - - - - - - - - - - - - - - - - - - - - - - - > > > > [ ID] Interval Transfer Bitrate Retr > > > > [ 5] 0.00-20.00 sec 2.05 GBytes 882 Mbits/sec 54845 > > > > > > Can you paste the output of ifconfig for both the interfaces used in > > > the test? > > > > > > Are you sure all hardware offloading capabilities are turned off on > > > both interfaces? > > > > > > Can you check what's causing those retries? > > > > > > Either using tcpdump, whireshark or some other tool to analyze the > > > network traffic and detect the errors that cause such retries? > > > > > > Thanks, Roger. > > > > > > > 172.31.16.127 (12.0-STABLE): > > > > lo0: flags=3D8049 metric 0 mtu 16384 > > options=3D680003 > > inet6 ::1 prefixlen 128 > > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 > > inet 127.0.0.1 netmask 0xff000000 > > groups: lo > > nd6 options=3D21 > > xn0: flags=3D8843 metric 0 mtu = 1500 > > options=3D501 > > I would try to disable rxcsum, tso4 and lro also. > > > ether 6e:83:99:ed:ce:f7 > > inet 172.31.16.127 netmask 0xffffff00 broadcast 172.31.16.255 > > media: Ethernet manual > > status: active > > nd6 options=3D29 > > > > ethtool -k vif68.0 > > Features for vif68.0: > > rx-checksumming: on [fixed] > > tx-checksumming: off > > tx-checksum-ipv4: off > > tx-checksum-ip-generic: off [fixed] > > tx-checksum-ipv6: off > > tx-checksum-fcoe-crc: off [fixed] > > tx-checksum-sctp: off [fixed] > > scatter-gather: off > > tx-scatter-gather: off > > tx-scatter-gather-fraglist: off > > tcp-segmentation-offload: off > > tx-tcp-segmentation: off > > tx-tcp-ecn-segmentation: off [fixed] > > tx-tcp6-segmentation: off > > udp-fragmentation-offload: off [fixed] > > generic-segmentation-offload: off > > generic-receive-offload: on > > large-receive-offload: off [fixed] > > rx-vlan-offload: off [fixed] > > tx-vlan-offload: off [fixed] > > ntuple-filters: off [fixed] > > receive-hashing: off [fixed] > > highdma: off [fixed] > > rx-vlan-filter: off [fixed] > > vlan-challenged: off [fixed] > > tx-lockless: off [fixed] > > netns-local: off [fixed] > > tx-gso-robust: off [fixed] > > tx-fcoe-segmentation: off [fixed] > > tx-gre-segmentation: off [fixed] > > tx-ipip-segmentation: off [fixed] > > tx-sit-segmentation: off [fixed] > > tx-udp_tnl-segmentation: off [fixed] > > fcoe-mtu: off [fixed] > > tx-nocache-copy: off > > loopback: off [fixed] > > rx-fcs: off [fixed] > > rx-all: off [fixed] > > tx-vlan-stag-hw-insert: off [fixed] > > rx-vlan-stag-hw-parse: off [fixed] > > rx-vlan-stag-filter: off [fixed] > > l2-fwd-offload: off [fixed] > > busy-poll: off [fixed] > > > > 172.31.16.128 (12.0-STABLE): > > > > lo0: flags=3D8049 metric 0 mtu 16384 > > options=3D680003 > > inet6 ::1 prefixlen 128 > > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 > > inet 127.0.0.1 netmask 0xff000000 > > groups: lo > > nd6 options=3D21 > > xn0: flags=3D8843 metric 0 mtu = 1500 > > options=3D501 > > ether 32:a7:9f:cc:94:8a > > inet 172.31.16.128 netmask 0xffffff00 broadcast 172.31.16.255 > > media: Ethernet manual > > status: active > > nd6 options=3D29 > > > > ethtool -k vif67.0 > > Features for vif67.0: > > rx-checksumming: on [fixed] > > tx-checksumming: off > > tx-checksum-ipv4: off > > tx-checksum-ip-generic: off [fixed] > > tx-checksum-ipv6: off > > tx-checksum-fcoe-crc: off [fixed] > > tx-checksum-sctp: off [fixed] > > scatter-gather: off > > tx-scatter-gather: off > > tx-scatter-gather-fraglist: off > > tcp-segmentation-offload: off > > tx-tcp-segmentation: off > > tx-tcp-ecn-segmentation: off [fixed] > > tx-tcp6-segmentation: off > > udp-fragmentation-offload: off [fixed] > > generic-segmentation-offload: off > > generic-receive-offload: on > > large-receive-offload: off [fixed] > > rx-vlan-offload: off [fixed] > > tx-vlan-offload: off [fixed] > > ntuple-filters: off [fixed] > > receive-hashing: off [fixed] > > highdma: off [fixed] > > rx-vlan-filter: off [fixed] > > vlan-challenged: off [fixed] > > tx-lockless: off [fixed] > > netns-local: off [fixed] > > tx-gso-robust: off [fixed] > > tx-fcoe-segmentation: off [fixed] > > tx-gre-segmentation: off [fixed] > > tx-ipip-segmentation: off [fixed] > > tx-sit-segmentation: off [fixed] > > tx-udp_tnl-segmentation: off [fixed] > > fcoe-mtu: off [fixed] > > tx-nocache-copy: off > > loopback: off [fixed] > > rx-fcs: off [fixed] > > rx-all: off [fixed] > > tx-vlan-stag-hw-insert: off [fixed] > > rx-vlan-stag-hw-parse: off [fixed] > > rx-vlan-stag-filter: off [fixed] > > l2-fwd-offload: off [fixed] > > busy-poll: off [fixed] > > > > Should I run tcpdump on xen host, or on FreeBSD while running iperf3? > > On the Xen host would be better, since that tcpdump will read the > packets as FreeBSD puts them on the wire. > > > As I side note, I installed XCP-ng 7.6.0 on a older server I had laying > > around just to see how 12.0-RELEASE behaved there, and It was the same > > thing, super slow between VM's. > > Yes, I don't think this is related to hardware at all. > > I will try to find some time to look into this, but it's not going to > be soon since I'm currently preparing a talk for a conference I'm > attending next week, so I won't likely be able to get to it until the > end of July, sorry. Also, I'm no networking expert, so I cannot assure > my success in fixing this. > > As a workaround you can switch to the emulated network card by > setting 'hw.xen.disable_pv_nics=3D1' in /boot/loader.conf. That will > give you worse performance than a fully working PV network card, but > at least should be consistent. There are others that have switched to > virtio-net, but I have no idea how to do that with XCP. > > Roger. > From owner-freebsd-xen@freebsd.org Thu Jul 4 08:39:46 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 111D915CCE84; Thu, 4 Jul 2019 08:39:46 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from smtp.krpservers.com (smtp.krpservers.com [62.13.128.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.krpservers.com", Issuer "RapidSSL RSA CA 2018" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AA14086B88; Thu, 4 Jul 2019 08:39:44 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from [10.12.30.106] (host165-120-77-244.range165-120.btcentralplus.com [165.120.77.244]) (authenticated bits=0) by smtp.krpservers.com (8.15.2/8.15.2) with ESMTPSA id x648db7Y057371 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 4 Jul 2019 09:39:38 +0100 (BST) (envelope-from kpielorz_lst@tdx.co.uk) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tdx.co.uk; s=krpdkim; t=1562229579; bh=vBUCabX76MGhYGJhCK52Q6mlTpZreRDLYUzOOcKeL1I=; h=Date:From:To:Subject; b=DCjNN+8lS9/X6IaVqEgbKenS8b4TAkrgu3ay7pe4pdKkHJHj/fBfHhPYTjTgTijQ2 FYlCVk+KUa7EhlhUxaUIzVK3qh4dGOe5fdP5afLJS6h4Pd5NZi69swrcocFit5aH/x I69sXtfPjzkWJXO/kluuvMC3hZC/Gkd8aDo0pUGmUwyEfzLdKwNNh0l56ikFBqWqgH xwCieAj44cF0/cccIt7EJcVPzI3F1I8fl3pQM/vO4gkBBIAZxbXWWL7FEGqI/HjshM PVAWAa/J/siTPZajvqXfE/GrXu1TbvLsLkB5FXXnyWRXMVIxmI8SSZ6v16NajtRXhW Uj5x4oOIOPTHw== Date: Thu, 04 Jul 2019 09:39:27 +0100 From: Karl Pielorz To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= , Christian M cc: freebsd-xen@freebsd.org, Eugene Grosbein , freebsd-net@freebsd.org Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer Message-ID: In-Reply-To: <20190704072354.6ycduisiidz5iy2x@Air-de-Roger> References: <9c377fe7-5546-9e0b-fac9-076dd094cbac@grosbein.net> <47bb1bfe-ac39-78d3-c9b9-17ff1896e0ec@grosbein.net> <20190627101901.3szaaxhp4xzjiay5@MacBook-Air-de-Roger.local> <20190704072354.6ycduisiidz5iy2x@Air-de-Roger> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Rspamd-Queue-Id: AA14086B88 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tdx.co.uk header.s=krpdkim header.b=DCjNN+8l; dmarc=pass (policy=none) header.from=tdx.co.uk; spf=pass (mx1.freebsd.org: domain of kpielorz_lst@tdx.co.uk designates 62.13.128.145 as permitted sender) smtp.mailfrom=kpielorz_lst@tdx.co.uk X-Spamd-Result: default: False [-2.28 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[tdx.co.uk:s=krpdkim]; NEURAL_HAM_MEDIUM(-0.97)[-0.975,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:smtp.krpservers.com]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_IP_LITERAL(0.50)[]; DKIM_TRACE(0.00)[tdx.co.uk:+]; MX_GOOD(-0.01)[mx0.krpservers.com]; DMARC_POLICY_ALLOW(-0.50)[tdx.co.uk,none]; NEURAL_SPAM_SHORT(0.22)[0.220,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-0.02)[country: GB(-0.09)]; ASN(0.00)[asn:60969, ipnet:62.13.128.0/24, country:GB]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Jul 2019 08:39:46 -0000 --On 04 July 2019 09:23 +0200 Roger Pau Monn=C3=A9 = wrote: > As a workaround you can switch to the emulated network card by > setting 'hw.xen.disable_pv_nics=3D1' in /boot/loader.conf. That will > give you worse performance than a fully working PV network card, but > at least should be consistent. There are others that have switched to > virtio-net, but I have no idea how to do that with XCP. Just to add my $0.02's to the conversation... As Roger knows we've had numerous issues with Xen xn based networking and=20 FreeBSD guests over the years. We currently run VirtIO (vtnet) on XenServer = 7.1 and it solves all these issues (it's probably not quite as 'performant' = as xn - but it's close for what we're using it with - it is miles better=20 than e1000 performance). Unfortunately in XenServer 7.6 virtio is no longer compiled into Qemu from=20 what I can see. XCP-ng has the same issue (as it's based on XenServer) - so = virtio is currently not an option with XCP-ng either. e1000/rtl8139 will solve 'weirdness' (think of VM's routing traffic, doing=20 DHCP or VPN duties) - but virtio solves the same issues, with much better=20 performance. I've posted to the XCP-ng forums asking if virtio can be enabled in XCP-ng=20 builds (as this is probably more likely that getting XenServer to release=20 with it enabled). Aside from getting Xen xn 'fixed' for the cases it currently fails (no=20 small undertaking from what I understand) - it leaves FreeBSD pretty much=20 stuck, at least for some usage cases. Regards, -Karl From owner-freebsd-xen@freebsd.org Thu Jul 4 09:00:10 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DD4A415CDA8F; Thu, 4 Jul 2019 09:00:09 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: from mail-io1-xd2b.google.com (mail-io1-xd2b.google.com [IPv6:2607:f8b0:4864:20::d2b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 02FA587871; Thu, 4 Jul 2019 09:00:09 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: by mail-io1-xd2b.google.com with SMTP id h6so11382755ioh.3; Thu, 04 Jul 2019 02:00:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=R6ZVpY4DFhvpA1IVhW/B+YgAgCRjSDPCO114z2rwVHM=; b=I8JPMJQVOGqjCTEIAvyO27tqkFz62V9WiJNfqVRiAAgJmevkFP40i3OUufgIEaMLWw iYJRB7dAnRs59VUJSzZ9i+kLePuts4BxtCP8hx0r2tCNhKZEPSNbNiaYAOsHotE2q0Mh 51Rl1hJdOS2eWpB3DFElA+vU1UnbsZKo5/rbaHZMH6NnR18GyYKTuUJIqRFQr3gWDni5 ubnomA41UhSxrlfGJBAOIVizV0+ou2IYu8SSkU89jcuA1d1uCiHIJe8DjdT4uHfc8rsX iXz2LERATJUl6+tJOpID/eTwiR/kF3jEkszw8S0FnPfG6ZwPgl7C8aSHe8GRGlcgfVEV AG9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=R6ZVpY4DFhvpA1IVhW/B+YgAgCRjSDPCO114z2rwVHM=; b=mbMK/rJEqSGKG3DCXTmNUpV2IHRezJkGN0zFvVHwcq/JipA/vhjFJ8Q6lFZ48dG2+p 7An1iEVEpCbCkZLeM3x/w9KD/mOX7dYbHNaeUfzrJ10Y+mOfkF65ZqOT8sdHpqls56zL id9cdcI+2LDc4lw4curFNZBOXnG/dwB4FrprMH8t9ZP3p/CtBoSNx12FYjlqYeE1Pdf9 uWRbwAnWPgx3p8poC//s/NZ0tSk4WDyv6UqU5fYOYLUxd4CxVKw2X1EnCaPwKRJF/tKY +/kQFhXIuz+1l1DBMUFHp7ysjfX/6VzLxQnZXWPVgz0VNjxEgn0TVfTn6h6/r2c+C5ff 9NTw== X-Gm-Message-State: APjAAAXP3J4CZmGRxb5UbBRemi8iLGhtNJv2TRDVUlGe3edOEtOpiKhp qXVcESDDcCD2LSqowW8hQbMNoXdnTcvJqhz5lR8= X-Google-Smtp-Source: APXvYqxHMFqzQ2CyBc1qWYbZOjElyGAe4B2zNTaIdIGKM+jG4i7F2TApbMCCIgEVOExUoXYjNsLcNrc5q+0P/FNvrYM= X-Received: by 2002:a02:710f:: with SMTP id n15mr46618313jac.119.1562230807926; Thu, 04 Jul 2019 02:00:07 -0700 (PDT) MIME-Version: 1.0 References: <9c377fe7-5546-9e0b-fac9-076dd094cbac@grosbein.net> <47bb1bfe-ac39-78d3-c9b9-17ff1896e0ec@grosbein.net> <20190627101901.3szaaxhp4xzjiay5@MacBook-Air-de-Roger.local> <20190704072354.6ycduisiidz5iy2x@Air-de-Roger> In-Reply-To: From: Christian M Date: Thu, 4 Jul 2019 10:59:56 +0200 Message-ID: Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: Karl Pielorz Cc: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= , freebsd-xen@freebsd.org, Eugene Grosbein , freebsd-net@freebsd.org X-Rspamd-Queue-Id: 02FA587871 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=I8JPMJQV; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of christianmarcos@gmail.com designates 2607:f8b0:4864:20::d2b as permitted sender) smtp.mailfrom=christianmarcos@gmail.com X-Spamd-Result: default: False [-6.74 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCPT_COUNT_FIVE(0.00)[5]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[b.2.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; IP_SCORE(-2.80)[ip: (-8.39), ipnet: 2607:f8b0::/32(-3.16), asn: 15169(-2.39), country: US(-0.06)]; NEURAL_HAM_SHORT(-0.93)[-0.933,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Jul 2019 09:00:10 -0000 >From my tests I found that 10.4-RELEASE was as fast as you could expect (>10Gbit/s), then something changed in 11.0-RELEASE and carried on and got even worse in 12.0-RELEASE. Would it not be a good idea to begin there (10.4 -> 11.0), and try to identify what changes could impact the performance drop so significantly? This is perhaps a much harder task than it sounds for someone that knows nothing about what changes was made, and how difficult it would be to identify what changes actually could be relevant. Just a thought. Regards, Christian Den tors 4 juli 2019 kl 10:39 skrev Karl Pielorz : > > > --On 04 July 2019 09:23 +0200 Roger Pau Monn=C3=A9 > wrote: > > > As a workaround you can switch to the emulated network card by > > setting 'hw.xen.disable_pv_nics=3D1' in /boot/loader.conf. That will > > give you worse performance than a fully working PV network card, but > > at least should be consistent. There are others that have switched to > > virtio-net, but I have no idea how to do that with XCP. > > Just to add my $0.02's to the conversation... > > As Roger knows we've had numerous issues with Xen xn based networking and > FreeBSD guests over the years. We currently run VirtIO (vtnet) on > XenServer > 7.1 and it solves all these issues (it's probably not quite as > 'performant' > as xn - but it's close for what we're using it with - it is miles better > than e1000 performance). > > Unfortunately in XenServer 7.6 virtio is no longer compiled into Qemu fro= m > what I can see. XCP-ng has the same issue (as it's based on XenServer) - > so > virtio is currently not an option with XCP-ng either. > > e1000/rtl8139 will solve 'weirdness' (think of VM's routing traffic, doin= g > DHCP or VPN duties) - but virtio solves the same issues, with much better > performance. > > I've posted to the XCP-ng forums asking if virtio can be enabled in XCP-n= g > builds (as this is probably more likely that getting XenServer to release > with it enabled). > > Aside from getting Xen xn 'fixed' for the cases it currently fails (no > small undertaking from what I understand) - it leaves FreeBSD pretty much > stuck, at least for some usage cases. > > > Regards, > > -Karl > From owner-freebsd-xen@freebsd.org Thu Jul 4 10:01:15 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DD05F15CF07C; Thu, 4 Jul 2019 10:01:14 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from smtp.krpservers.com (smtp.krpservers.com [62.13.128.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.krpservers.com", Issuer "RapidSSL RSA CA 2018" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5C1C389786; Thu, 4 Jul 2019 10:01:13 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from [10.12.30.106] (host165-120-77-244.range165-120.btcentralplus.com [165.120.77.244]) (authenticated bits=0) by smtp.krpservers.com (8.15.2/8.15.2) with ESMTPSA id x64A17tj077015 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 4 Jul 2019 11:01:08 +0100 (BST) (envelope-from kpielorz_lst@tdx.co.uk) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tdx.co.uk; s=krpdkim; t=1562234468; bh=HxfnP60J9LwjAJkxy43aDndOxikiCS4w20sTfNeHk/o=; h=Date:From:To:Subject; b=lamkrcvRiHn4S/nr/4QKsbkczOPewQ9YrGQg0zpWO9aEVG91JeehkvoABE4JwQpzE MyW6Z4I99m2+XoaTs4ntcOtzQC1mOz9hLI5vjYNXH0RRXYwV64uklUL1rOpdKg/Png hCUQcaTVW3CtHlDYz1zf/i03Og+XQ1SgS1u+Esqes0MwS0Rx6ur6RYh8WhqYH1Mxaj TqXRufbhjwiJ/DUipkDle4s4HuB6BiAc5eOm/8Hu6ZcpAIYqGackw3fLLs599gRLZd Y6MAlUZRUnldFJMsS7yWJl8YKXV7CRiqJ8p/ikmFsoT/MUqFuZYoFqegtaWR7Iim9A w4TszpTWVZUMQ== Date: Thu, 04 Jul 2019 11:00:56 +0100 From: Karl Pielorz To: Christian M cc: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= , freebsd-xen@freebsd.org, Eugene Grosbein , freebsd-net@freebsd.org Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer Message-ID: <9BAF7DF10B3842D6ADEE168D@[10.12.30.106]> In-Reply-To: References: <9c377fe7-5546-9e0b-fac9-076dd094cbac@grosbein.net> <47bb1bfe-ac39-78d3-c9b9-17ff1896e0ec@grosbein.net> <20190627101901.3szaaxhp4xzjiay5@MacBook-Air-de-Roger.local> <20190704072354.6ycduisiidz5iy2x@Air-de-Roger> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Rspamd-Queue-Id: 5C1C389786 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tdx.co.uk header.s=krpdkim header.b=lamkrcvR; dmarc=pass (policy=none) header.from=tdx.co.uk; spf=pass (mx1.freebsd.org: domain of kpielorz_lst@tdx.co.uk designates 62.13.128.145 as permitted sender) smtp.mailfrom=kpielorz_lst@tdx.co.uk X-Spamd-Result: default: False [-3.30 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[tdx.co.uk:s=krpdkim]; NEURAL_HAM_MEDIUM(-1.00)[-0.997,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:smtp.krpservers.com]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_IP_LITERAL(0.50)[]; DKIM_TRACE(0.00)[tdx.co.uk:+]; MX_GOOD(-0.01)[cached: mx0.krpservers.com]; DMARC_POLICY_ALLOW(-0.50)[tdx.co.uk,none]; NEURAL_HAM_SHORT(-0.42)[-0.419,0]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-0.37)[asn: 60969(-1.78), country: GB(-0.09)]; ASN(0.00)[asn:60969, ipnet:62.13.128.0/24, country:GB]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Jul 2019 10:01:15 -0000 --On 04 July 2019 10:59 +0200 Christian M wrote: > From my tests I found that 10.4-RELEASE was as fast as you could expect > (>10Gbit/s), then something changed in 11.0-RELEASE and carried on and got > even worse in 12.0-RELEASE. Would it not be a good idea to begin there > (10.4 -> 11.0), and try to identify what changes could impact the > performance drop so significantly? This is perhaps a much harder task than > it sounds for someone that knows nothing about what changes was made, and > how difficult it would be to identify what changes actually could be > relevant. Just a thought. Looking at FreeBSD source - there doesn't appear to be a lot that has changed (at least in sys/xen) even from 10.4 through to 12 - certainly nothing networking I can see (unless I'm using svn log 'wrong', or there's other xen stuff within FreeBSD I should be looking at). So this could be caused by other changes that have happened with the OS over that time. If I get time - I'll see if I can follow the thread and setup a similar test system here, to be fair we've not had a performance issue with Xen Networking (as nothing we do needs the performance - yet). We've been much more plagued by packet weirdness which virtio has fixed. Even if I can re-create things here - it may only be an "oh yeah, so it is" moment. -Kp