From owner-freebsd-net@freebsd.org Sat Dec 15 14:19:04 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B66BE13168DF for ; Sat, 15 Dec 2018 14:19:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 3C28472424 for ; Sat, 15 Dec 2018 14:19:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id F0E5713168DA; Sat, 15 Dec 2018 14:19:03 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B5D7413168D2 for ; Sat, 15 Dec 2018 14:19:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 55BB272422 for ; Sat, 15 Dec 2018 14:19:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id AA34BE060 for ; Sat, 15 Dec 2018 14:19:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wBFEJ2ao024487 for ; Sat, 15 Dec 2018 14:19:02 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wBFEJ2mi024486 for net@FreeBSD.org; Sat, 15 Dec 2018 14:19:02 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 221919] ixl: TX queue hang when using TSO and having a high and mixed network load Date: Sat, 15 Dec 2018 14:18:58 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-STABLE X-Bugzilla-Keywords: IntelNetworking X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: peter.x.eriksson@liu.se X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: erj@freebsd.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Dec 2018 14:19:05 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221919 --- Comment #20 from Peter Eriksson --- Just a quick note that we're still seeing the same problem on our production servers if we enable "tso" on the 10G interfaces. FreeBSD 11.2-RELEASE-p6. Haven't been able to reproduce it on the test servers (identical hardware) running 11.2-RELEASE-p5 (and 12-0-RELEASE) so far though (but they don't see any traffic)... Driver version: > dev.ixl.0.%desc: Intel(R) Ethernet Connection 700 Series PF Driver, Versi= on - 1.9.9-k Firmware: > dev.ixl.0.fw_version: fw 6.80.48812 api 1.7 nvm 6.00 etid 80003751 oem 18= .4608.17 Watch Events in the output from sysctl -a > dev.ixl.0.watchdog_events: 4 Dmesg errors: > ixl0: WARNING: queue 3 appears to be hung! > ixl0: WARNING: queue 2 appears to be hung! > ixl2: WARNING: queue 2 appears to be hung! > ixl2: WARNING: queue 4 appears to be hung! > ixl2: WARNING: queue 7 appears to be hung! > ixl2: WARNING: queue 3 appears to be hung! > ixl0: WARNING: queue 7 appears to be hung! > ixl2: WARNING: queue 3 appears to be hung! > ixl0: WARNING: queue 4 appears to be hung! (Output from ifconfig with TSO disabled) > # ifconfig lagg0 > lagg0: flags=3D8843 metric 0 mtu = 1500 > > options=3D6404bb > ether 3c:fd:fe:25:47:a0 > inet6 fe80::3efd:feff:fe25:47a0%lagg0 prefixlen 64 scopeid 0xa > inet6 2001:6b0:17:2400::8:43 prefixlen 64 > inet 130.236.8.43 netmask 0xffffffe0 broadcast 130.236.8.63 > nd6 options=3D21 > media: Ethernet autoselect > status: active > groups: lagg > laggproto lacp lagghash l2,l3,l4 > laggport: ixl0 flags=3D1c > laggport: ixl2 flags=3D1c iperf3 output with TSO disabled: > # iperf3 -c filur00 -t4 > Connecting to host filur00, port 5201 > [ 5] local 2001:6b0:17:2400::8:43 port 51226 connected to 2001:6b0:17:24= 00::8:40 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 318 MBytes 2.66 Gbits/sec 0 561 KBytes > [ 5] 1.00-2.00 sec 350 MBytes 2.94 Gbits/sec 0 1.11 MBytes > [ 5] 2.00-3.00 sec 392 MBytes 3.28 Gbits/sec 0 1.67 MBytes > [ 5] 3.00-4.00 sec 351 MBytes 2.94 Gbits/sec 0 1.77 MBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-4.00 sec 1.38 GBytes 2.95 Gbits/sec 0 sen= der > [ 5] 0.00-4.00 sec 1.38 GBytes 2.95 Gbits/sec rec= eiver >=20 > iperf Done. With TSO enabled (when things work): > # ifconfig lagg0 tso ; iperf3 -c filur00 -t4 > Connecting to host filur00, port 5201 > [ 5] local 2001:6b0:17:2400::8:43 port 51237 connected to 2001:6b0:17:24= 00::8:40 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 976 MBytes 8.19 Gbits/sec 0 492 KBytes > [ 5] 1.00-2.00 sec 1.08 GBytes 9.29 Gbits/sec 0 1021 KBytes > [ 5] 2.00-3.00 sec 1.08 GBytes 9.29 Gbits/sec 0 1.50 MBytes > [ 5] 3.00-4.00 sec 1.08 GBytes 9.28 Gbits/sec 0 1.75 MBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-4.00 sec 4.20 GBytes 9.01 Gbits/sec 0 sen= der > [ 5] 0.00-4.00 sec 4.19 GBytes 9.01 Gbits/sec rec= eiver >=20 > iperf Done. But often queues get stuck and freezes. Hmm.. I just noticed that it was IP= v6 that stopped working when I tried to enable it on a production server and r= an iperf3 on it - IPv4 traffic was still passing thru.=20 Can it be that there still are IPv6 (TSO6)-related bugs and that the IPv4 o= nes are solved? Too bad I can't find a way to force it to happen on the test servers... --=20 You are receiving this mail because: You are on the CC list for the bug.=