From owner-freebsd-net@FreeBSD.ORG Fri Jan 31 23:20:58 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B456DBA; Fri, 31 Jan 2014 23:20:58 +0000 (UTC) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 676DD13E2; Fri, 31 Jan 2014 23:20:58 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: X-IronPort-AV: E=Sophos;i="4.95,760,1384318800"; d="scan'208";a="92702741" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 31 Jan 2014 18:20:56 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id A57A0B3F43; Fri, 31 Jan 2014 18:20:56 -0500 (EST) Date: Fri, 31 Jan 2014 18:20:56 -0500 (EST) From: Rick Macklem To: J David Message-ID: <1609454808.1083115.1391210456671.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: Terrible NFS performance under 9.2-RELEASE? MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: freebsd-net@freebsd.org, Garrett Wollman X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Jan 2014 23:20:58 -0000 J David wrote: > On Fri, Jan 31, 2014 at 1:18 AM, wrote: > > This is almost entirely wrong in its description of the non-offload > > case. > > Yes, you're quite right; I confused myself. GSO works a little > differently, but FreeBSD doesn't use that. > > > The whole mess is then passed on to the hardware for > > offload, if it fits. > > That's the point, NFS is creating a situation where it never fits. > It > can't shove 65k into 64k, so it ends up looping back through the > whole > output routine again for a tiny tail of data, and then the same for > the input routine on the other side. Arguably that makes rsize/wsize > 65536 negligibly different than rsize/wsize 32768 in the long run > because the average data output per pass is about the same (64k + 1k > vs 33k + 33k). Except, of course, in the case where almost all files > are between 32k and 60k. > Oh, and remember to try setting readahead=8 in your mounts, too. NFS will do a read + N readaheads (where N == 1 by default) and then wait for replies to those before continuing on. If the product of rsize * readahead isn't enough bits to fill the pipe (bandwidth * transit delay), then you won't be using the bandwidth your network interface provides. rick ps: Any you probably want your nfsd threads to be at least 16 instead of the default of 4. > Please don't get me wrong, I'm not suggesting there's anything more > than a small CPU reduction to be obtained by changing this. Which is > not nothing if the client is CPU-limited due to the other work it's > doing, but it's not much. To get real speedups from NFS would > require > a change to the punishing read-before-write behavior, which is pretty > clearly not going to happen. > > > RPC responses will only get smushed together if > > tcp_output() wasn't able to schedule the transmit immediately, and > > if > > the network is working properly, that will only happen if there's > > more > > than one client-side-receive-window's-worth of data to be > > transmitted. > > This is something I have seen live in tcpdump, but then I have had so > many problems with NFS and congestion control that the "network is > working properly" condition probably isn't satisfied. Hopefully the > jumbo cluster changes will resolve that once and for all. > > Thanks! > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org" >