From owner-freebsd-net@freebsd.org Fri Jan 17 10:37:58 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E0CD222A276 for ; Fri, 17 Jan 2020 10:37:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 47zcw66RLVz4Lhg for ; Fri, 17 Jan 2020 10:37:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id DCD4822A275; Fri, 17 Jan 2020 10:37:58 +0000 (UTC) Delivered-To: net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DC9AC22A274 for ; Fri, 17 Jan 2020 10:37:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47zcw66GsLz4Lhf for ; Fri, 17 Jan 2020 10:37:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id D2B847579 for ; Fri, 17 Jan 2020 10:37:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 00HAbwtG063353 for ; Fri, 17 Jan 2020 10:37:58 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 00HAbwVp063352 for net@FreeBSD.org; Fri, 17 Jan 2020 10:37:58 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 243392] vmx driver input buffer corruption Date: Fri, 17 Jan 2020 10:37:58 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: alexandr.oleynikov@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jan 2020 10:37:58 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D243392 --- Comment #4 from alexandr.oleynikov@gmail.com --- I did some more tests. Hope this will provide some more information. First one with recompiled kernel with TSO patch. As a network load was a fi= le coping to server using samba=20 ifconfig vmx1 vmx1: flags=3D8943 metric 0= mtu 9000 =20=20=20=20=20=20=20 options=3De403bb ether 00:50:56:be:f0:13 inet 172.31.255.2 netmask 0xffffff00 broadcast 172.31.255.255 media: Ethernet autoselect status: active nd6 options=3D29 # tcpdump -i vmx1 icmp & # tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vmx1, link-type EN10MB (Ethernet), capture size 262144 bytes # ping -s 8000 172.31.255.3 PING 172.31.255.3 (172.31.255.3): 8000 data bytes 11:59:07.108253 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 0, length 8008 11:59:07.108425 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, = seq 0, length 8008 8008 bytes from 172.31.255.3: icmp_seq=3D0 ttl=3D128 time=3D0.226 ms 11:59:08.126583 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 1, length 8008 11:59:08.126754 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, = seq 1, length 8008 8008 bytes from 172.31.255.3: icmp_seq=3D1 ttl=3D128 time=3D0.213 ms --- skipped some lines --- 12:00:20.401492 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 71, length 8008 8008 bytes from 172.31.255.3: icmp_seq=3D71 ttl=3D128 time=3D0.550 ms 12:00:20.402010 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, = seq 71, length 8008 12:00:21.408758 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 72, length 8008 8008 bytes from 172.31.255.3: icmp_seq=3D72 ttl=3D128 time=3D2.303 ms 12:00:21.410995 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, = seq 72, length 8008 12:00:24.527165 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 73, length 8008 8008 bytes from 172.31.255.3: icmp_seq=3D73 ttl=3D128 time=3D133.291 ms 12:00:24.592341 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, = seq 73, length 8008 12:00:25.569300 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 74, length 8008 12:00:25.662953 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, = seq 74, length 8008 --- after seqnum 73 packets received by kernel and seen with tcpdump but not returned to ping process --- skipped some lines ---=20 12:01:27.114142 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 134, length 8008 12:01:27.160943 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, = seq 134, length 8008 12:01:28.125972 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 135, length 8008 12:01:28.126346 IP truncated-ip - 7982 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 135, length 8008 --- received malformed L2 frame from seqnum >=3D 135 12:01:29.198552 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 136, length 8008 12:01:29.223302 IP truncated-ip - 7810 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 136, length 8008 12:01:30.214849 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 137, length 8008 12:01:30.221687 IP truncated-ip - 7822 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 137, length 8008 12:01:31.246460 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 138, length 8008 --- skip some lines 12:01:37.514942 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 144, length 8008 12:01:37.517865 IP truncated-ip - 7808 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 144, length 8008 12:01:38.579626 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 145, length 8008 12:01:38.615120 IP truncated-ip - 7928 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 145, length 8008 12:01:39.603253 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 146, length 8008 12:01:40.614996 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 147, length 8008 12:01:40.615183 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, = seq 146, length 8008 --- difference in 1 second between sending and receiveng reply from seqnum = 146 12:01:40.615201 IP truncated-ip - 7928 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 147, length 8008 8008 bytes from 172.31.255.3: icmp_seq=3D146 ttl=3D128 time=3D1011.985 ms 12:01:41.657600 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 148, length 8008 12:01:42.701072 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 149, length 8008 12:01:42.701321 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, = seq 148, length 8008 8008 bytes from 172.31.255.3: icmp_seq=3D148 ttl=3D128 time=3D1043.763 ms 12:01:43.615120 IP truncated-ip - 7928 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 149, length 8008 12:01:43.714982 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 150, length 8008 12:01:43.988367 IP truncated-ip - 7808 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 150, length 8008 12:01:44.787457 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 151, length 8008 12:01:44.788966 IP truncated-ip - 7782 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 151, length 8008 12:01:45.815011 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 152, length 8008 12:01:45.970727 IP truncated-ip - 7976 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 152, length 8008 12:01:46.834089 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 153, length 8008 12:01:47.615212 IP truncated-ip - 7928 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 153, length 8008 12:01:47.897600 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 154, length 8008 12:01:48.914981 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 155, length 8008 12:01:48.915192 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, = seq 154, length 8008 8008 bytes from 172.31.255.3: icmp_seq=3D154 ttl=3D128 time=3D1017.638 ms --- some packet reveived undamaged but with delay in 1 second When i try using iperf as network load source in most cases was kernel pani= c as result: Fatal trap 12: page fault while in kernel mode cpuid =3D 1; apic id =3D 02 fault virtual address =3D 0x0 fault code =3D supervisor write data, page not present instruction pointer =3D 0x20:0xffffffff80cef252 stack pointer =3D 0x28:0xfffffe00753547c0 frame pointer =3D 0x28:0xfffffe00753548a0 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 0 (if_io_tqg_1) trap number =3D 12 panic: page fault cpuid =3D 1 time =3D 1579255990 KDB: stack backtrace: #0 0xffffffff80c1d297 at kdb_backtrace+0x67 #1 0xffffffff80bd05cd at vpanic+0x19d #2 0xffffffff80bd0423 at panic+0x43 #3 0xffffffff810a7d2c at trap_fatal+0x39c #4 0xffffffff810a7d79 at trap_pfault+0x49 #5 0xffffffff810a736f at trap+0x29f #6 0xffffffff81081a0c at calltrap+0x8 #7 0xffffffff80ce9be5 at _task_fn_rx+0x75 #8 0xffffffff80c1bb54 at gtaskqueue_run_locked+0x144 #9 0xffffffff80c1b7b8 at gtaskqueue_thread_loop+0x98 #10 0xffffffff80b90c23 at fork_exit+0x83 #11 0xffffffff81082a4e at fork_trampoline+0xe Uptime: 16m42s Then reverting to default kernel, disabling tso and reboot: # uname -a FreeBSD ******************* 12.1-RELEASE-p1 FreeBSD 12.1-RELEASE-p1 GENERI= C=20 amd64 # ifconfig vmx1 vmx1: flags=3D8843 metric 0 mtu 9000 =20=20=20=20=20=20=20 options=3De400bb ether 00:50:56:be:f0:13 inet 172.31.255.2 netmask 0xffffff00 broadcast 172.31.255.255 media: Ethernet autoselect status: active nd6 options=3D29 # iperf3 -c 172.31.255.2 -p 1234 Connecting to host 172.31.255.2, port 1234 [ 5] local 172.31.255.5 port 32466 connected to 172.31.255.2 port 1234 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.03 sec 497 MBytes 4.05 Gbits/sec 11 8.74 KBytes [ 5] 1.03-2.07 sec 0.00 Bytes 0.00 bits/sec 3 8.74 KBytes [ 5] 2.07-3.06 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes [ 5] 3.06-4.02 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes [ 5] 4.02-5.01 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes [ 5] 5.01-6.03 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes [ 5] 6.03-7.06 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes [ 5] 7.06-8.04 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes [ 5] 8.04-9.07 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes [ 5] 9.07-10.01 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.01 sec 497 MBytes 416 Mbits/sec 18 sender [ 5] 0.00-10.60 sec 496 MBytes 393 Mbits/sec recei= ver # ping -s 8000 172.31.255.5 PING 172.31.255.5 (172.31.255.5): 8000 data bytes 8008 bytes from 172.31.255.5: icmp_seq=3D0 ttl=3D64 time=3D0.322 ms 12:22:09.903151 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 0, length 8008 12:22:09.903253 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 0, length 8008 12:22:10.922205 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 1, length 8008 12:22:10.922300 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 1, length 8008 8008 bytes from 172.31.255.5: icmp_seq=3D1 ttl=3D64 time=3D0.147 ms 12:22:11.969930 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 2, length 8008 12:22:11.970035 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 2, length 8008 8008 bytes from 172.31.255.5: icmp_seq=3D2 ttl=3D64 time=3D0.159 ms 12:22:12.997254 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 3, length 8008 12:22:12.997386 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 3, length 8008 8008 bytes from 172.31.255.5: icmp_seq=3D3 ttl=3D64 time=3D0.175 ms 12:22:14.029823 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 4, length 8008 12:22:14.030017 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 4, length 8008 8008 bytes from 172.31.255.5: icmp_seq=3D4 ttl=3D64 time=3D0.237 ms 12:22:15.058570 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 5, length 8008 12:22:15.058769 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 5, length 8008 8008 bytes from 172.31.255.5: icmp_seq=3D5 ttl=3D64 time=3D0.241 ms 12:22:16.096803 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 6, length 8008 12:22:16.096896 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 6, length 8008 8008 bytes from 172.31.255.5: icmp_seq=3D6 ttl=3D64 time=3D0.139 ms 12:22:17.136966 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 7, length 8008 12:22:17.137224 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 7, length 8008 12:22:18.164014 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 8, length 8008 12:22:18.164194 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 8, length 8008 --- packets stops sending to ping process --- skip some lines --- -- but after some time packets againg sending to ping process 12:26:15.636917 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 238, length 8008 12:26:15.637147 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 238, length 8008 12:26:16.696907 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 239, length 8008 12:26:16.697100 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 239, length 8008 8008 bytes from 172.31.255.5: icmp_seq=3D239 ttl=3D64 time=3D0.256 ms 12:26:17.756044 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 240, length 8008 12:26:17.756178 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 240, length 8008 8008 bytes from 172.31.255.5: icmp_seq=3D240 ttl=3D64 time=3D0.190 ms 12:26:18.796861 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 241, length 8008 12:26:18.796982 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 241, length 8008 8008 bytes from 172.31.255.5: icmp_seq=3D241 ttl=3D64 time=3D0.176 ms 12:26:19.836847 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 242, length 8008 12:26:19.836981 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 242, length 8008 8008 bytes from 172.31.255.5: icmp_seq=3D242 ttl=3D64 time=3D0.192 ms --=20 You are receiving this mail because: You are the assignee for the bug.=