From owner-freebsd-stable@FreeBSD.ORG Sat Nov 26 01:29:55 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 418BE1065670 for ; Sat, 26 Nov 2011 01:29:55 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id F28D68FC13 for ; Sat, 26 Nov 2011 01:29:54 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap8EANZA0E6DaFvO/2dsb2JhbABEhQOnAoFyAQEBAwEBAQEgKyALBRYYAgINGQIpAQkmBggHBAEcBIdkCKRjkTCBMIgcgRYEiCGKCYIgkik X-IronPort-AV: E=Sophos;i="4.69,575,1315195200"; d="scan'208";a="145662711" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 25 Nov 2011 20:29:53 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id E79A7B3F86; Fri, 25 Nov 2011 20:29:53 -0500 (EST) Date: Fri, 25 Nov 2011 20:29:53 -0500 (EST) From: Rick Macklem To: George Mitchell Message-ID: <1629066195.410848.1322270993899.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <4ECFD8D2.4030604@m5p.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-stable@freebsd.org Subject: Re: TCP Reassembly Issues X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Nov 2011 01:29:55 -0000 George Mitchell wrote: > On 11/24/11 21:00, Jeremy Chadwick wrote: > >[...] > > If none of this solves the problem, then I consider this a priority > > 0 > > blocker (read: "all hands on deck") issue with the IP stack in > > FreeBSD > > 9.x and will need immediate attention. > > > > I would strongly recommend a developer or clueful end-user begin > > tracking down who committed all of these bits and CC them into the > > thread. I would start by looking who implemented the > > net.inet.tcp.reass.cursegments sysctl, because that isn't in > > RELENG_8 at > > all. > > > > I've tried out the 9.0 release candidates, and what I notice is that > for > a few minutes after the system starts, I get wonderful NFS read > throughput (7+ MB/s over a 100 megabit interface) -- more than twice > as > fast as 7.n or 8.n on the same hardware -- quickly degrading to > abysmal > (less than 0.5 MB/s). Is this possibly related to the problem under > discussion? -- George Mitchell > Well, when I've seen NFS perf. degrade like this, it has usually been related to RPC transport (and TCP is the default for 9.0). Just from reading some of the thread, it sounds like this problem will result in the FAIL count (the last #) for "vmstat -z" for tcpreass will increase and/or net.inet.tcp.reass.cursegments increases to net.inet.tcp.reass.maxsegments. I'd suggest that, after the NFS perf has degrades, you: # vmstat -z | fgrep tcpreass - and see how big the last # is # sysctl -a | fgrep reass - and see how cursegments compares with maxsegments If these don't indicate that is the TCP Reassembly Issue, then... There are many other possibilities w.r.t. the NFS perf. degradation. Most often I've seen it when the net interface hardware/device driver starts dropping packets (like happens on this laptop with an el-cheapo re net interface in it). You can capture a packet trace after the performance has degraded with tcpdump and look to see if TCP segments are being lost/retransmitted. (Although wireshark knows NFS and is nice for this, because it shows relative sequence numbers, the TCP dump will show you the TCP level retries, etc.) Good luck with it, rick > P.S. A lot of other 9.0 features look very nice indeed! > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to > "freebsd-stable-unsubscribe@freebsd.org"