Date: Wed, 26 Jun 2019 12:26:09 +0200 From: Christian M <christian.marcos@gmail.com> To: Eugene Grosbein <eugen@grosbein.net> Cc: freebsd-xen@freebsd.org, freebsd-net@freebsd.org Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer Message-ID: <CAKwR995FNv-QjkUQKaHw7Wb2jhA3DoiWzQ7prnA-k56qE8vifw@mail.gmail.com> In-Reply-To: <d89b2b9a-5005-c9ee-5364-9affff5fd1f5@grosbein.net> References: <mailman.7.1560945600.15387.freebsd-xen@freebsd.org> <12994df1-f847-ec92-aae8-43a32e59385f@darco.dk> <CAKwR994kQTRfB0R0_WiTYx4pgOHtR-ge7h0vP8NiM_%2B-MBPGGw@mail.gmail.com> <20190625082103.qiiz2cikrauaqlas@MacBook-Air-de-Roger.local> <CAKwR996s9Y_Bea95GOhPkdEJabmJM0r6Qi43gW7_8SXE2ryddA@mail.gmail.com> <b0d72766-e7a9-08f7-ed8f-f407fe1285e6@grosbein.net> <CAKwR996sPR=qCwk0TO=zhy5J8vOPZtowxFQuK0QooF9bRObOrw@mail.gmail.com> <8ec84f75-2100-ff8a-6337-3c955f31da62@grosbein.net> <CAKwR994qWVpY0D8s0n5SOZcY3GzOSV87y7sG0LiMWps3-_ACcg@mail.gmail.com> <d89b2b9a-5005-c9ee-5364-9affff5fd1f5@grosbein.net>
next in thread | previous in thread | raw e-mail | index | archive | help
I ran ifconfig xn0 -txcsum on both test VM's, and all incorrect checksum disappeared. netstat -sp ip for VM1 (172.31.16.125, running iperf3 -c always) and VM2 (172.31.16.126, running iperf3 -s always): ip: 3664084 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 3664080 packets for this host 0 packets for unknown/unsupported protocol 0 packets forwarded (0 packets fast forwarded) 4 packets not forwardable 0 packets received for unknown multicast group 0 redirects sent 7003245 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header ip: 8217827 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 8217827 packets for this host 0 packets for unknown/unsupported protocol 0 packets forwarded (0 packets fast forwarded) 0 packets not forwardable 0 packets received for unknown multicast group 0 redirects sent 4300027 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header netstat -sp tcp for VM1 (172.31.16.125, running iperf3 -c always) and VM2 (172.31.16.126, running iperf3 -s always): tcp: 7002523 packets sent 6866670 data packets (9939548357 bytes) 109393 data packets (101233068 bytes) retransmitted 0 data packets unnecessarily retransmitted 0 resends initiated by MTU discovery 27039 ack-only packets (90 delayed) 0 URG only packets 0 window probe packets 0 window update packets 125 control packets 3664099 packets received 3387212 acks (for 9939333715 bytes) 270936 duplicate acks 0 acks for unsent data 915 packets (36547 bytes) received in-sequence 4887 completely duplicate packets (208 bytes) 4853 old duplicate packets 0 packets with some dup. data (0 bytes duped) 0 out-of-order packets (0 bytes) 0 packets (0 bytes) of data after window 0 window probes 5694 window update packets 6 packets received after close 0 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 0 discarded due to memory problems 88 connection requests 2 connection accepts 0 bad connection attempts 0 listen queue overflows 0 ignored RSTs in the windows 89 connections established (including accepts) 85 times used RTT from hostcache 85 times used RTT variance from hostcache 85 times used slow-start threshold from hostcache 90 connections closed (including 57 drops) 51 connections updated cached RTT on close 51 connections updated cached RTT variance on close 23 connections updated cached ssthresh on close 1 embryonic connection dropped 3387212 segments updated rtt (of 3313837 attempts) 0 retransmit timeouts 0 connections dropped by rexmit timeout 0 persist timeouts 0 connections dropped by persist timeout 0 Connections (fin_wait_2) dropped because of timeout 0 keepalive timeouts 0 keepalive probes sent 0 connections dropped by keepalive 759139 correct ACK header predictions 70 correct data packet header predictions 2 syncache entries added 0 retransmitted 0 dupsyn 0 dropped 2 completed 0 bucket overflow 0 cache overflow 0 reset 0 stale 0 aborted 0 badack 0 unreach 0 zone failures 2 cookies sent 0 cookies received 1 hostcache entry added 0 bucket overflow 5602 SACK recovery episodes 109329 segment rexmits in SACK recovery episodes 101663360 byte rexmits in SACK recovery episodes 249723 SACK options (SACK blocks) received 0 SACK options (SACK blocks) sent 0 SACK scoreboard overflow 0 packets with ECN CE bit set 0 packets with ECN ECT(0) bit set 0 packets with ECN ECT(1) bit set 0 successful ECN handshakes 0 times ECN reduced the congestion window 0 packets with matching signature received 0 packets with bad signature received 0 times failed to make signature due to no SA 0 times unexpected signature received 0 times no signature provided by segment 0 Path MTU discovery black hole detection activations 0 Path MTU discovery black hole detection min MSS activations 0 Path MTU discovery black hole detection failures TCP connection count by state: 0 connections in CLOSED state 3 connections in LISTEN state 0 connections in SYN_SENT state 0 connections in SYN_RCVD state 2 connections in ESTABLISHED state 0 connections in CLOSE_WAIT state 0 connections in FIN_WAIT_1 state 0 connections in CLOSING state 0 connections in LAST_ACK state 0 connections in FIN_WAIT_2 state 0 connections in TIME_WAIT state tcp: 4299563 packets sent 1510 data packets (132487 bytes) 0 data packets (0 bytes) retransmitted 0 data packets unnecessarily retransmitted 0 resends initiated by MTU discovery 4297568 ack-only packets (2610 delayed) 0 URG only packets 0 window probe packets 392 window update packets 93 control packets 8217839 packets received 1623 acks (for 132605 bytes) 150324 duplicate acks 0 acks for unsent data 7770336 packets (11249613126 bytes) received in-sequence 126189 completely duplicate packets (117973252 bytes) 4853 old duplicate packets 194 packets with some dup. data (59900 bytes duped) 290196 out-of-order packets (415026177 bytes) 0 packets (0 bytes) of data after window 0 window probes 0 window update packets 25 packets received after close 0 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 0 discarded due to memory problems 6 connection requests 102 connection accepts 0 bad connection attempts 0 listen queue overflows 1 ignored RSTs in the window 108 connections established (including accepts) 95 times used RTT from hostcache 95 times used RTT variance from hostcache 0 times used slow-start threshold from hostcache 163 connections closed (including 47 drops) 29 connections updated cached RTT on close 29 connections updated cached RTT variance on close 0 connections updated cached ssthresh on close 0 embryonic connections dropped 1623 segments updated rtt (of 1314 attempts) 0 retransmit timeouts 0 connections dropped by rexmit timeout 0 persist timeouts 0 connections dropped by persist timeout 0 Connections (fin_wait_2) dropped because of timeout 9 keepalive timeouts 8 keepalive probes sent 1 connection dropped by keepalive 593 correct ACK header predictions 7769705 correct data packet header predictions 102 syncache entries added 0 retransmitted 0 dupsyn 0 dropped 102 completed 0 bucket overflow 0 cache overflow 0 reset 0 stale 0 aborted 0 badack 0 unreach 0 zone failures 102 cookies sent 0 cookies received 3 hostcache entries added 0 bucket overflow 0 SACK recovery episodes 0 segment rexmits in SACK recovery episodes 0 byte rexmits in SACK recovery episodes 0 SACK options (SACK blocks) received 293795 SACK options (SACK blocks) sent 0 SACK scoreboard overflow 0 packets with ECN CE bit set 0 packets with ECN ECT(0) bit set 0 packets with ECN ECT(1) bit set 0 successful ECN handshakes 0 times ECN reduced the congestion window 0 packets with matching signature received 0 packets with bad signature received 0 times failed to make signature due to no SA 0 times unexpected signature received 0 times no signature provided by segment 0 Path MTU discovery black hole detection activations 0 Path MTU discovery black hole detection min MSS activations 0 Path MTU discovery black hole detection failures TCP connection count by state: 0 connections in CLOSED state 4 connections in LISTEN state 0 connections in SYN_SENT state 0 connections in SYN_RCVD state 2 connections in ESTABLISHED state 0 connections in CLOSE_WAIT state 0 connections in FIN_WAIT_1 state 0 connections in CLOSING state 0 connections in LAST_ACK state 0 connections in FIN_WAIT_2 state 0 connections in TIME_WAIT state The data collected is after a number of runs with iperf3. I'm really not qualified to interpret any of this, so my hopes are that someone else sees something unusual here. Thanks, Christian Den ons 26 juni 2019 kl 10:37 skrev Eugene Grosbein <eugen@grosbein.net>: > 26.06.2019 15:11, Christian M wrote: > > > Running tcpdump on the host while running iperf3 between the 12.0 VM's > > results in a lot of incorrect cksum like this. > > > > tcpdump -i vif54.0 -v -nn| grep -i incorrect > > 172.31.16.125.63013 > 172.31.16.126.5201: Flags [.], cksum 0x7f08 > > (incorrect -> 0x030f), seq 223153882:223155330, ack 1, win 1026, options > > [nop,nop,TS val 4104002274 ecr 3926764642], length 1448 > > Incorrect checksum shown by tcpdump is normal and expected for outgoing > packets > if txcsum offload is not disabled because networking stack skips checksum > calculation > offloading it to "hardware". It still indicates a bug at sending side if > seen for outgoing packets > when txcsum is disabled. > > Incorrect checksum shown by tcpdump for incoming packets always indicates > a problem > at sending side or intermediate network (if any). > > You should check output of "netstat -sp ip" and "netstat -sp tcp" for > non-zero > and growing checksum error counters at receiving side. > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAKwR995FNv-QjkUQKaHw7Wb2jhA3DoiWzQ7prnA-k56qE8vifw>