Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 27 Jun 2019 12:31:35 +0200
From:      Christian M <christian.marcos@gmail.com>
To:        =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= <roger.pau@citrix.com>
Cc:        Eugene Grosbein <eugen@grosbein.net>, freebsd-xen@freebsd.org, freebsd-net@freebsd.org
Subject:   Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
Message-ID:  <CAKwR994OYS2Hjhu=QdR2DSoZVyDWEky2Nkmu_LWDQRJiggqxmA@mail.gmail.com>
In-Reply-To: <20190627101901.3szaaxhp4xzjiay5@MacBook-Air-de-Roger.local>
References:  <8ec84f75-2100-ff8a-6337-3c955f31da62@grosbein.net> <CAKwR994qWVpY0D8s0n5SOZcY3GzOSV87y7sG0LiMWps3-_ACcg@mail.gmail.com> <d89b2b9a-5005-c9ee-5364-9affff5fd1f5@grosbein.net> <CAKwR995FNv-QjkUQKaHw7Wb2jhA3DoiWzQ7prnA-k56qE8vifw@mail.gmail.com> <9c377fe7-5546-9e0b-fac9-076dd094cbac@grosbein.net> <CAKwR996i6ZJLt4iBFTedDNzPaO2mFt5jz3P6iFM-Y1SJ0pu_wA@mail.gmail.com> <ce1f2ebb-c0bf-73c5-f857-be6b24e71da1@grosbein.net> <CAKwR997Zjxkpr48DJ08csLobLK5u5bBY%2BKrQ8r22hdZYJUNREA@mail.gmail.com> <47bb1bfe-ac39-78d3-c9b9-17ff1896e0ec@grosbein.net> <CAKwR9968gf1RkgP7hD8XOh4Z-Ti917P6W5Lt3-Lakx6r9pei9g@mail.gmail.com> <20190627101901.3szaaxhp4xzjiay5@MacBook-Air-de-Roger.local>

next in thread | previous in thread | raw e-mail | index | archive | help
Den tors 27 juni 2019 kl 12:19 skrev Roger Pau Monn=C3=A9 <roger.pau@citrix=
.com>:

> On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote:
> > I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and .128.
> VIF
> > cheksum offloading is turned off, and -txcsum for xn0 for both VM's.
> >
> > I feel the throughput is more consistent now, not all over the place as
> > before, even between runs. But the Retr column (tcp retries) in iperf3
> has
> > jumped up considerably from hundreds/s to thousands/s.
> >
> > Just a reminder, I have tested this with 11.0-RELEASE also, where the
> issue
> > appeared first for me. 10.4-RELEASE is as fast as I could expect it to
> be,
> > and 0 retries.
> >
> > 12.0-STABLE:
> >
> > Connecting to host 172.31.16.128, port 5201
> > [  5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port 52=
01
> > [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
> > [  5]   0.00-1.00   sec  96.3 MBytes   808 Mbits/sec  2401   2.85 KByte=
s
> >
> > [  5]   1.00-2.00   sec   118 MBytes   991 Mbits/sec  3120   17.0 KByte=
s
> >
> > [  5]   2.00-3.00   sec   121 MBytes  1.02 Gbits/sec  3203   69.8 KByte=
s
> >
> > [  5]   3.00-4.00   sec   102 MBytes   853 Mbits/sec  3126   15.6 KByte=
s
> >
> > [  5]   4.00-5.00   sec   110 MBytes   921 Mbits/sec  2890   15.6 KByte=
s
> >
> > [  5]   5.00-6.00   sec   108 MBytes   908 Mbits/sec  3308   17.0 KByte=
s
> >
> > [  5]   6.00-7.00   sec   104 MBytes   869 Mbits/sec  3046   48.2 KByte=
s
> >
> > [  5]   7.00-8.00   sec  98.9 MBytes   830 Mbits/sec  2845   2.85 KByte=
s
> >
> > [  5]   8.00-9.00   sec   104 MBytes   874 Mbits/sec  2711   86.8 KByte=
s
> >
> > [  5]   9.00-10.00  sec   108 MBytes   904 Mbits/sec  2696   14.2 KByte=
s
> >
> > [  5]  10.00-11.00  sec   103 MBytes   864 Mbits/sec  2660   31.3 KByte=
s
> >
> > [  5]  11.00-12.00  sec  98.8 MBytes   828 Mbits/sec  2476   19.9 KByte=
s
> >
> > [  5]  12.00-13.00  sec  99.9 MBytes   838 Mbits/sec  2857   11.3 KByte=
s
> >
> > [  5]  13.00-14.00  sec   107 MBytes   894 Mbits/sec  2685   24.1 KByte=
s
> >
> > [  5]  14.00-15.00  sec   114 MBytes   953 Mbits/sec  2321   25.5 KByte=
s
> >
> > [  5]  15.00-16.00  sec  93.1 MBytes   781 Mbits/sec  2427   48.3 KByte=
s
> >
> > [  5]  16.00-17.00  sec   107 MBytes   895 Mbits/sec  2219   29.8 KByte=
s
> >
> > [  5]  17.00-18.00  sec  92.5 MBytes   776 Mbits/sec  2441   12.8 KByte=
s
> >
> > [  5]  18.00-19.00  sec   116 MBytes   976 Mbits/sec  2840   38.2 KByte=
s
> >
> > [  5]  19.00-20.00  sec   102 MBytes   853 Mbits/sec  2573   43.9 KByte=
s
> >
> > - - - - - - - - - - - - - - - - - - - - - - - - -
> > [ ID] Interval           Transfer     Bitrate         Retr
> > [  5]   0.00-20.00  sec  2.05 GBytes   882 Mbits/sec  54845
>
> Can you paste the output of ifconfig for both the interfaces used in
> the test?
>
> Are you sure all hardware offloading capabilities are turned off on
> both interfaces?
>
> Can you check what's causing those retries?
>
> Either using tcpdump, whireshark or some other tool to analyze the
> network traffic and detect the errors that cause such retries?
>
> Thanks, Roger.
>

172.31.16.127 (12.0-STABLE):

lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3D680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL>
xn0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=3D501<RXCSUM,TSO4,LRO>
ether 6e:83:99:ed:ce:f7
inet 172.31.16.127 netmask 0xffffff00 broadcast 172.31.16.255
media: Ethernet manual
status: active
nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

ethtool -k vif68.0
Features for vif68.0:
rx-checksumming: on [fixed]
tx-checksumming: off
tx-checksum-ipv4: off
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: off
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: off
tx-scatter-gather: off
tx-scatter-gather-fraglist: off
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: off
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]

172.31.16.128 (12.0-STABLE):

lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3D680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL>
xn0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=3D501<RXCSUM,TSO4,LRO>
ether 32:a7:9f:cc:94:8a
inet 172.31.16.128 netmask 0xffffff00 broadcast 172.31.16.255
media: Ethernet manual
status: active
nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

ethtool -k vif67.0
Features for vif67.0:
rx-checksumming: on [fixed]
tx-checksumming: off
tx-checksum-ipv4: off
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: off
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: off
tx-scatter-gather: off
tx-scatter-gather-fraglist: off
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: off
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]

Should I run tcpdump on xen host, or on FreeBSD while running iperf3?

As I side note, I installed XCP-ng 7.6.0 on a older server I had laying
around just to see how 12.0-RELEASE behaved there, and It was the same
thing, super slow between VM's.



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAKwR994OYS2Hjhu=QdR2DSoZVyDWEky2Nkmu_LWDQRJiggqxmA>