Date: Wed, 26 Jun 2019 10:11:47 +0200 From: Christian M <christian.marcos@gmail.com> To: Eugene Grosbein <eugen@grosbein.net> Cc: freebsd-xen@freebsd.org, freebsd-net@freebsd.org Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer Message-ID: <CAKwR994qWVpY0D8s0n5SOZcY3GzOSV87y7sG0LiMWps3-_ACcg@mail.gmail.com> In-Reply-To: <8ec84f75-2100-ff8a-6337-3c955f31da62@grosbein.net> References: <mailman.7.1560945600.15387.freebsd-xen@freebsd.org> <12994df1-f847-ec92-aae8-43a32e59385f@darco.dk> <CAKwR994kQTRfB0R0_WiTYx4pgOHtR-ge7h0vP8NiM_%2B-MBPGGw@mail.gmail.com> <20190625082103.qiiz2cikrauaqlas@MacBook-Air-de-Roger.local> <CAKwR996s9Y_Bea95GOhPkdEJabmJM0r6Qi43gW7_8SXE2ryddA@mail.gmail.com> <b0d72766-e7a9-08f7-ed8f-f407fe1285e6@grosbein.net> <CAKwR996sPR=qCwk0TO=zhy5J8vOPZtowxFQuK0QooF9bRObOrw@mail.gmail.com> <8ec84f75-2100-ff8a-6337-3c955f31da62@grosbein.net>
next in thread | previous in thread | raw e-mail | index | archive | help
checksum offloading doe not seem to have any affect when done on the VM VIF's. Both 12.0-RELEASE VIF's now have: other-config (MRW): ethtool-rx: off; ethtool-tx: off; ethtool-sg: off; ethtool-tso: off; ethtool-ufo: off; ethtool-gso: off ethtool -k vif54.0 Features for vif54.0: rx-checksumming: on [fixed] tx-checksumming: off tx-checksum-ipv4: off tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: off tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: off tx-scatter-gather: off tx-scatter-gather-fraglist: off tcp-segmentation-offload: off tx-tcp-segmentation: off tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: off udp-fragmentation-offload: off [fixed] generic-segmentation-offload: off generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: off [fixed] ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: off [fixed] rx-vlan-filter: off [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] busy-poll: off [fixed] Both vif's look the same as above. Running tcpdump on the host while running iperf3 between the 12.0 VM's results in a lot of incorrect cksum like this. tcpdump -i vif54.0 -v -nn| grep -i incorrect 172.31.16.125.63013 > 172.31.16.126.5201: Flags [.], cksum 0x7f08 (incorrect -> 0x030f), seq 223153882:223155330, ack 1, win 1026, options [nop,nop,TS val 4104002274 ecr 3926764642], length 1448 ... Also noticed that iperf3 does not work to run every time, more than 50% of the tries with iperf3 -c <ip> I get: Connecting to host 172.31.16.126, port 5201 iperf3: error - unable to initialize stream: Socket is not connected Or nothing happens and I need to abort, then restart iperf3 -s on the other VM, and retry. The results from iperf3 are really inconsistent still, and I cant get a good feel of any pattern. Currently as I type this, a result looks like this: Host: tcpdump running VIF's: no promisc xn0 on both VM's: no promisc [ 5] local 172.31.16.125 port 47738 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 37.8 MBytes 316 Mbits/sec 176 1.43 KBytes [ 5] 1.00-2.00 sec 133 MBytes 1.11 Gbits/sec 1022 49.8 KBytes [ 5] 2.00-3.00 sec 179 MBytes 1.50 Gbits/sec 784 191 KBytes [ 5] 3.00-4.00 sec 132 MBytes 1.11 Gbits/sec 201 345 KBytes [ 5] 4.00-5.00 sec 95.3 MBytes 799 Mbits/sec 438 180 KBytes [ 5] 5.00-6.00 sec 126 MBytes 1.06 Gbits/sec 1256 353 KBytes [ 5] 6.00-7.00 sec 144 MBytes 1.20 Gbits/sec 1242 275 KBytes [ 5] 7.00-8.00 sec 94.5 MBytes 792 Mbits/sec 593 167 KBytes [ 5] 8.00-9.00 sec 125 MBytes 1.05 Gbits/sec 326 169 KBytes [ 5] 9.00-10.00 sec 57.0 MBytes 478 Mbits/sec 1013 1.43 KBytes [ 5] 10.00-11.00 sec 31.3 MBytes 262 Mbits/sec 463 1.43 KBytes [ 5] 11.00-12.00 sec 42.3 MBytes 355 Mbits/sec 544 78.3 KBytes [ 5] 12.00-13.00 sec 41.3 MBytes 346 Mbits/sec 624 1.43 KBytes [ 5] 13.00-14.00 sec 52.7 MBytes 442 Mbits/sec 739 1.43 KBytes [ 5] 14.00-15.00 sec 20.4 MBytes 171 Mbits/sec 384 1.43 KBytes [ 5] 15.00-16.00 sec 74.5 MBytes 625 Mbits/sec 489 299 KBytes [ 5] 16.00-17.00 sec 40.1 MBytes 337 Mbits/sec 835 1.43 KBytes [ 5] 17.00-18.00 sec 43.2 MBytes 363 Mbits/sec 372 1.43 KBytes [ 5] 18.00-19.00 sec 58.9 MBytes 494 Mbits/sec 723 1.43 KBytes [ 5] 19.00-20.00 sec 21.2 MBytes 178 Mbits/sec 300 1.43 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 1.51 GBytes 649 Mbits/sec 12524 sender [ 5] 0.00-20.00 sec 1.51 GBytes 649 Mbits/sec receiver But please be sure, this can change dramatically without making any changes what so ever. Offloading is not turned off in FreeBSD (but on the VIF's as stated before), and tcpdump is running on the host. xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=503<RXCSUM,TXCSUM,TSO4,LRO> ether 12:ec:6a:5c:af:df inet 172.31.16.125 netmask 0xffffff00 broadcast 172.31.16.255 media: Ethernet manual status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=503<RXCSUM,TXCSUM,TSO4,LRO> ether aa:4f:f7:4c:a0:e6 inet 172.31.16.126 netmask 0xffffff00 broadcast 172.31.16.255 media: Ethernet manual status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> Absolutely nothing changed this time (restarted iperf3 -s though since I ran ifconfig do get the info above), tcpdump still running on host (never even aborted): Host: tcpdump still running VIF's: no promisc xn0 on both VM's: no promisc [ 5] local 172.31.16.125 port 30543 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 10.3 MBytes 86.8 Mbits/sec 165 38.4 KBytes [ 5] 1.00-2.00 sec 39.7 MBytes 333 Mbits/sec 175 167 KBytes [ 5] 2.00-3.00 sec 40.0 MBytes 335 Mbits/sec 437 54.1 KBytes [ 5] 3.00-4.00 sec 11.2 MBytes 93.6 Mbits/sec 320 1.43 KBytes [ 5] 4.00-5.00 sec 11.2 MBytes 94.2 Mbits/sec 305 1.43 KBytes [ 5] 5.00-6.00 sec 7.63 MBytes 64.1 Mbits/sec 179 2.84 KBytes [ 5] 6.00-7.00 sec 6.77 MBytes 56.7 Mbits/sec 191 1.43 KBytes [ 5] 7.00-8.00 sec 4.26 MBytes 35.9 Mbits/sec 145 21.3 KBytes [ 5] 8.00-9.00 sec 5.38 MBytes 45.1 Mbits/sec 176 1.43 KBytes [ 5] 9.00-10.00 sec 8.24 MBytes 69.1 Mbits/sec 225 1.43 KBytes [ 5] 10.00-11.00 sec 16.1 MBytes 135 Mbits/sec 422 1.43 KBytes [ 5] 11.00-12.00 sec 3.76 MBytes 31.5 Mbits/sec 113 1.43 KBytes [ 5] 12.00-13.00 sec 6.62 MBytes 55.5 Mbits/sec 161 1.43 KBytes [ 5] 13.00-14.00 sec 5.28 MBytes 44.3 Mbits/sec 156 18.4 KBytes [ 5] 14.00-15.00 sec 5.21 MBytes 43.7 Mbits/sec 146 1.43 KBytes [ 5] 15.00-16.00 sec 6.37 MBytes 53.4 Mbits/sec 199 1.43 KBytes [ 5] 16.00-17.00 sec 10.7 MBytes 89.5 Mbits/sec 193 1.43 KBytes [ 5] 17.00-18.00 sec 9.49 MBytes 79.6 Mbits/sec 390 1.43 KBytes [ 5] 18.00-19.00 sec 5.36 MBytes 44.9 Mbits/sec 125 1.43 KBytes [ 5] 19.00-20.00 sec 3.65 MBytes 30.6 Mbits/sec 58 1.43 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 217 MBytes 91.1 Mbits/sec 4281 sender [ 5] 0.00-20.00 sec 217 MBytes 90.8 Mbits/sec receiver Host: tcpdump still running VIF's: promisc xn0 on both VM's: promisc Connecting to host 172.31.16.126, port 5201 [ 5] local 172.31.16.125 port 61196 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 25.9 MBytes 217 Mbits/sec 271 90.5 KBytes [ 5] 1.00-2.00 sec 56.4 MBytes 473 Mbits/sec 94 255 KBytes [ 5] 2.00-3.00 sec 83.4 MBytes 699 Mbits/sec 872 201 KBytes [ 5] 3.00-4.00 sec 76.4 MBytes 641 Mbits/sec 684 176 KBytes [ 5] 4.00-5.00 sec 80.4 MBytes 674 Mbits/sec 742 1.43 KBytes [ 5] 5.00-6.00 sec 81.4 MBytes 684 Mbits/sec 248 333 KBytes [ 5] 6.00-7.00 sec 86.1 MBytes 722 Mbits/sec 826 160 KBytes [ 5] 7.00-8.00 sec 77.5 MBytes 650 Mbits/sec 209 253 KBytes [ 5] 8.00-9.00 sec 9.64 MBytes 80.9 Mbits/sec 433 12.8 KBytes [ 5] 9.00-10.00 sec 8.31 MBytes 69.6 Mbits/sec 210 1.43 KBytes [ 5] 10.00-11.00 sec 8.64 MBytes 72.5 Mbits/sec 199 1.43 KBytes [ 5] 11.00-12.00 sec 7.02 MBytes 58.9 Mbits/sec 240 1.43 KBytes [ 5] 12.00-13.00 sec 20.1 MBytes 169 Mbits/sec 579 1.43 KBytes [ 5] 13.00-14.00 sec 10.8 MBytes 90.4 Mbits/sec 277 22.7 KBytes [ 5] 14.00-15.00 sec 2.75 MBytes 23.1 Mbits/sec 107 1.43 KBytes [ 5] 15.00-16.00 sec 8.79 MBytes 73.8 Mbits/sec 203 12.8 KBytes [ 5] 16.00-17.00 sec 9.23 MBytes 77.3 Mbits/sec 158 1.43 KBytes [ 5] 17.00-18.00 sec 13.5 MBytes 114 Mbits/sec 381 1.43 KBytes [ 5] 18.00-19.00 sec 41.6 MBytes 349 Mbits/sec 331 99.5 KBytes [ 5] 19.00-20.00 sec 40.6 MBytes 340 Mbits/sec 70 1.43 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 749 MBytes 314 Mbits/sec 7134 sender [ 5] 0.00-20.00 sec 748 MBytes 314 Mbits/sec receiver Host: tcpdump NOT running VIF's: no promisc xn0: no promisc Run1: Connecting to host 172.31.16.126, port 5201 [ 5] local 172.31.16.125 port 20600 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 28.4 MBytes 238 Mbits/sec 451 1.43 KBytes [ 5] 1.00-2.00 sec 21.5 MBytes 181 Mbits/sec 546 2.84 KBytes [ 5] 2.00-3.00 sec 26.7 MBytes 224 Mbits/sec 595 1.43 KBytes [ 5] 3.00-4.00 sec 15.9 MBytes 133 Mbits/sec 347 1.43 KBytes [ 5] 4.00-5.00 sec 4.45 MBytes 37.4 Mbits/sec 187 2.84 KBytes [ 5] 5.00-6.00 sec 15.4 MBytes 129 Mbits/sec 239 66.5 KBytes [ 5] 6.00-7.00 sec 10.6 MBytes 89.1 Mbits/sec 366 1.43 KBytes [ 5] 7.00-8.00 sec 30.8 MBytes 258 Mbits/sec 510 1.43 KBytes [ 5] 8.00-9.00 sec 16.6 MBytes 140 Mbits/sec 332 2.84 KBytes [ 5] 9.00-10.00 sec 12.4 MBytes 104 Mbits/sec 376 1.43 KBytes [ 5] 10.00-11.00 sec 11.3 MBytes 94.9 Mbits/sec 223 1.43 KBytes [ 5] 11.00-12.00 sec 9.00 MBytes 75.5 Mbits/sec 206 1.43 KBytes [ 5] 12.00-13.00 sec 8.95 MBytes 75.1 Mbits/sec 201 1.43 KBytes [ 5] 13.00-14.00 sec 12.3 MBytes 103 Mbits/sec 179 1.43 KBytes [ 5] 14.00-15.00 sec 20.5 MBytes 172 Mbits/sec 445 1.43 KBytes [ 5] 15.00-16.00 sec 3.92 MBytes 32.9 Mbits/sec 163 1.43 KBytes [ 5] 16.00-17.00 sec 6.82 MBytes 57.2 Mbits/sec 122 2.84 KBytes [ 5] 17.00-18.00 sec 14.3 MBytes 120 Mbits/sec 238 1.43 KBytes [ 5] 18.00-19.00 sec 14.2 MBytes 119 Mbits/sec 226 1.43 KBytes [ 5] 19.00-20.00 sec 12.9 MBytes 108 Mbits/sec 297 1.43 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 297 MBytes 125 Mbits/sec 6249 sender [ 5] 0.00-20.00 sec 297 MBytes 124 Mbits/sec receiver Run2 right after a iperf3 -s restart on second VM: Connecting to host 172.31.16.126, port 5201 [ 5] local 172.31.16.125 port 40622 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 6.49 MBytes 54.3 Mbits/sec 163 1.43 KBytes [ 5] 1.00-2.00 sec 9.93 MBytes 83.3 Mbits/sec 263 1.43 KBytes [ 5] 2.00-3.00 sec 7.71 MBytes 64.7 Mbits/sec 173 1.43 KBytes [ 5] 3.00-4.00 sec 5.58 MBytes 47.0 Mbits/sec 83 1.43 KBytes [ 5] 4.00-5.00 sec 14.8 MBytes 124 Mbits/sec 347 1.43 KBytes [ 5] 5.00-6.00 sec 16.8 MBytes 141 Mbits/sec 349 1.43 KBytes [ 5] 6.00-7.00 sec 19.2 MBytes 161 Mbits/sec 205 1.43 KBytes [ 5] 7.00-8.00 sec 6.39 MBytes 53.6 Mbits/sec 79 1.43 KBytes [ 5] 8.00-9.00 sec 7.19 MBytes 60.3 Mbits/sec 133 1.43 KBytes [ 5] 9.00-10.00 sec 18.2 MBytes 153 Mbits/sec 265 1.43 KBytes [ 5] 10.00-11.00 sec 11.0 MBytes 92.6 Mbits/sec 220 1.43 KBytes [ 5] 11.00-12.00 sec 7.93 MBytes 66.6 Mbits/sec 172 1.43 KBytes [ 5] 12.00-13.00 sec 10.1 MBytes 84.8 Mbits/sec 198 1.43 KBytes [ 5] 13.00-14.00 sec 7.92 MBytes 66.5 Mbits/sec 196 1.43 KBytes [ 5] 14.00-15.00 sec 7.63 MBytes 64.0 Mbits/sec 144 1.43 KBytes [ 5] 15.00-16.00 sec 8.70 MBytes 73.0 Mbits/sec 267 1.43 KBytes [ 5] 16.00-17.00 sec 16.0 MBytes 134 Mbits/sec 296 1.43 KBytes [ 5] 17.00-18.00 sec 10.9 MBytes 91.5 Mbits/sec 261 1.43 KBytes [ 5] 18.00-19.00 sec 23.0 MBytes 193 Mbits/sec 444 1.43 KBytes [ 5] 19.00-20.00 sec 8.86 MBytes 74.3 Mbits/sec 224 1.43 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 224 MBytes 94.1 Mbits/sec 4482 sender [ 5] 0.00-20.00 sec 224 MBytes 93.9 Mbits/sec receiver As suggested before, I have set kern.eventtimer.periodic: 1, the above tests are done with that setting. I don't see any difference with it set or not (kern.eventtimer.periodic: 0 was default btw). There was also a suggestion to have a look at kern.timecouter.*, but I'm not sure what to look for here: kern.timecounter.tsc_shift: 1 kern.timecounter.smp_tsc_adjust: 0 kern.timecounter.smp_tsc: 0 kern.timecounter.invariant_tsc: 0 kern.timecounter.fast_gettime: 1 kern.timecounter.tick: 1 kern.timecounter.choice: XENTIMER(950) ACPI-fast(900) i8254(0) HPET(950) TSC-low(-100) dummy(-1000000) kern.timecounter.hardware: XENTIMER kern.timecounter.alloweddeviation: 5 kern.timecounter.stepwarnings: 0 kern.timecounter.tc.XENTIMER.quality: 950 kern.timecounter.tc.XENTIMER.frequency: 1000000000 kern.timecounter.tc.XENTIMER.counter: 3517394506 kern.timecounter.tc.XENTIMER.mask: 4294967295 kern.timecounter.tc.ACPI-fast.quality: 900 kern.timecounter.tc.ACPI-fast.frequency: 3579545 kern.timecounter.tc.ACPI-fast.counter: 1749326798 kern.timecounter.tc.ACPI-fast.mask: 4294967295 kern.timecounter.tc.i8254.quality: 0 kern.timecounter.tc.i8254.frequency: 1193182 kern.timecounter.tc.i8254.counter: 58432 kern.timecounter.tc.i8254.mask: 65535 kern.timecounter.tc.HPET.quality: 950 kern.timecounter.tc.HPET.frequency: 62500000 kern.timecounter.tc.HPET.counter: 3606880145 kern.timecounter.tc.HPET.mask: 4294967295 kern.timecounter.tc.TSC-low.quality: -100 kern.timecounter.tc.TSC-low.frequency: 1200075192 kern.timecounter.tc.TSC-low.counter: 3248698599 kern.timecounter.tc.TSC-low.mask: 4294967295 Regards, Christian Den ons 26 juni 2019 kl 01:14 skrev Eugene Grosbein <eugen@grosbein.net>: > 25.06.2019 19:40, Christian M wrote: > > > I've disabled them all on both VM's with FreeBSD 12.0-RELEASE (ifconfig > xn0 > > -txcsum -rxcsum -lro -tso), and also tried disabling everything for the > VIF > > via XCP-ng earlier. It made no difference unfortunately. > > Then you should try to bring the interface to promiscuous mode manually > with "ifconfig xn0 promisc" any retry. If this does not make a difference, > then you may be facing some timekeeping problem as tcpdump makes additional > CPU load on the system that can force scheduler behaviour to change. > > As next step, you should check your kern.timecounter.* and > kern.eventtimer.* sysctls. > If default value of kern.eventtimer.periodic is 0 (it depends on detected > "hardware"), > you should change it kern.eventtimer.periodic=1 and retry the test. > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAKwR994qWVpY0D8s0n5SOZcY3GzOSV87y7sG0LiMWps3-_ACcg>