From owner-freebsd-net@FreeBSD.ORG Tue Mar 25 23:06:44 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8D59AD1D; Tue, 25 Mar 2014 23:06:44 +0000 (UTC) Received: from mail-qc0-x234.google.com (mail-qc0-x234.google.com [IPv6:2607:f8b0:400d:c01::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 23B4112E; Tue, 25 Mar 2014 23:06:44 +0000 (UTC) Received: by mail-qc0-f180.google.com with SMTP id w7so1660938qcr.39 for ; Tue, 25 Mar 2014 16:06:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=zoD0KE9aAbgbUW0clh2h+FVgn/h46nQ2aDImti0DVuw=; b=O6mqLGY7yvA6X1lRZd+lvZF1aZAQGQoxq9fPS688PZysbDrltRY7TKbLp1qgtYoPvP NeFa3/NyCirNNviqei5/4TTj4fBzj9x72mvEXZHmMvlBrSRQ0kj34xZ8tgKZXwEcZHDp h7C/eeFTpa26jk0CJmWjGEhJE1xZvwfsf6YvA+zyTtf9sO3kj3xrvf/LYf9AbJ1z7Tld xBxBd9Ui81Ff1K2RmWwBIIRuvns96ViVRPh8dV6nab0sxVyQBLe+Pj95IyLC0ll1eMjx CKBjEVyubO65Sd0qXzHXU14dRxZhvOay5AHh6vYWKlP2OF8g1lmAsiRQsksTIr5g/cDS pZtw== MIME-Version: 1.0 X-Received: by 10.140.50.231 with SMTP id s94mr26017545qga.33.1395788803361; Tue, 25 Mar 2014 16:06:43 -0700 (PDT) Received: by 10.96.79.97 with HTTP; Tue, 25 Mar 2014 16:06:43 -0700 (PDT) In-Reply-To: References: <0BC10908-2081-45AC-A1C8-14220D81EC0A@hostpoint.ch> <1236110257.2510701.1395709458870.JavaMail.root@uoguelph.ca> Date: Tue, 25 Mar 2014 20:06:43 -0300 Message-ID: Subject: Re: 9.2 ixgbe tx queue hang From: Christopher Forgeron To: Rick Macklem Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: FreeBSD Net , Garrett Wollman , Jack Vogel , Markus Gebert X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Mar 2014 23:06:44 -0000 Update: I'm changing my mind, and I believe Rick's TSO patch is fixing things (sorry). In looking at my notes, it's possible I had lagg on for those tests. lagg does seem to negate the TSO patch in my case. kernel.10stable_basicTSO_65535/ - IP_MAXPACKET = 65535; - manually forced (no if statement) ifp->if_hw_tsomax = IP_MAXPACKET - (ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN); - Verified on boot via printf that ifp->if_hw_tsomax = 65517 - Boot in a NON LAGG environment. ix0 only. ixgbe's printf is showing packets up to 65530. Haven't run long enough yet to see if anything will go over 65535 I have this tcpdump running to check packet size. tcpdump -ennvvXS -i ix0 greater 65518 I do expect to get packets over 65518, but I was just curious to see if any of them would go over 65535. Time will tell. In a separate test, If I enable lagg, we have LOTS of oversized packet problems. It looks like tsomax is definitely not making it through in if_lagg.c - Any recommendations there? I will eventually need lagg, as I'm sure will others. With dtrace, it's showing t_tsomax >= 65518. Shouldn't that not be happening? dtrace -n 'fbt::tcp_output:entry / args[0]->t_tsomax != 0 && args[0]->t_tsomax >= 65518 / { printf("unexpected tp->t_tsomax: %i\n", args[0]->t_tsomax); stack(); }' 6 31403 tcp_output:entry unexpected tp->t_tsomax: 65535 kernel`tcp_do_segment+0x2c99 kernel`tcp_input+0x11a2 kernel`ip_input+0xa2 kernel`netisr_dispatch_src+0x5e kernel`ether_demux+0x12a kernel`ether_nh_input+0x35f kernel`netisr_dispatch_src+0x5e kernel`bce_intr+0x765 kernel`intr_event_execute_handlers+0xab kernel`ithread_loop+0x96 kernel`fork_exit+0x9a kernel`0xffffffff80c75b2e 3 31403 tcp_output:entry unexpected tp->t_tsomax: 65535 kernel`tcp_do_segment+0x2c99 kernel`tcp_input+0x11a2 kernel`ip_input+0xa2 kernel`netisr_dispatch_src+0x5e kernel`ether_demux+0x12a kernel`ether_nh_input+0x35f kernel`netisr_dispatch_src+0x5e kernel`bce_intr+0x765 kernel`intr_event_execute_handlers+0xab kernel`ithread_loop+0x96 kernel`fork_exit+0x9a kernel`0xffffffff80c75b2e 6 31403 tcp_output:entry unexpected tp->t_tsomax: 65535 kernel`tcp_do_segment+0x2c99 kernel`tcp_input+0x11a2 kernel`ip_input+0xa2 kernel`netisr_dispatch_src+0x5e kernel`ether_demux+0x12a kernel`ether_nh_input+0x35f kernel`netisr_dispatch_src+0x5e kernel`bce_intr+0x765 kernel`intr_event_execute_handlers+0xab kernel`ithread_loop+0x96 kernel`fork_exit+0x9a kernel`0xffffffff80c75b2e 1 31403 tcp_output:entry unexpected tp->t_tsomax: 65535 kernel`tcp_do_segment+0x2c99 kernel`tcp_input+0x11a2 kernel`ip_input+0xa2 kernel`netisr_dispatch_src+0x5e kernel`ether_demux+0x12a kernel`ether_nh_input+0x35f kernel`netisr_dispatch_src+0x5e kernel`bce_intr+0x765 kernel`intr_event_execute_handlers+0xab kernel`ithread_loop+0x96 kernel`fork_exit+0x9a kernel`0xffffffff80c75b2e