From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 00:04:08 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 06254252 for ; Thu, 3 Jul 2014 00:04:08 +0000 (UTC) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id BF4AB2287 for ; Thu, 3 Jul 2014 00:04:07 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqoEAHQItFODaFve/2dsb2JhbABag19agm6oUgEBAQaTFIZtUwGBInWEAwEBAQMBAQEBIAQnIAsFFhgCAg0ZAikBCSYGCAcEARwEiBkIDapZmzAXgSyERIhhAQEbNAeCd4FMBZgChDOSQYNfIS8GgQU5 X-IronPort-AV: E=Sophos;i="5.01,591,1400040000"; d="scan'208";a="137915737" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 02 Jul 2014 20:04:06 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 177C6B3F47; Wed, 2 Jul 2014 20:04:06 -0400 (EDT) Date: Wed, 2 Jul 2014 20:04:06 -0400 (EDT) From: Rick Macklem To: "Russell L. Carter" Message-ID: <371130768.6608219.1404345846086.JavaMail.root@uoguelph.ca> In-Reply-To: <53B49AE0.4030902@pinyon.org> Subject: Re: NFS client READ performance on -current MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2014 00:04:08 -0000 Russell L. Carter wrote: > Greetings! > It's been 14 years. OMG do I love poudriere and zfs. > > But apropos of this post from last January: > > http://lists.freebsd.org/pipermail/freebsd-net/2014-January/037547.html > > I am going to capitalize READ to emphasize that all I am looking at > here is reading on the client. In some of posts I have read people > have been confusing read vs. write performance and losing the thread. > > I have fresh current server and client, each with otherwise quiescent > em NICs. I also have debian kernel 3.14-1 server and client. The > freebsd hardware is large enough, 16GB RAM + vishera 8 cores. (The > debian systems are much smaller/slower) > > Upshot is that with a -current NFSV4 (4.1) serving a -current client > and a linux client, I see today client READ performance of: > > -current -> -current: 2.7MB/s READ nice and steady > -current -> linux: 80MB/s READ very bursty > > And for the weird (to me) result: > > linux -> -current: 55.74MB/s READ > > So to be very clear, the freebsd -current server seems capable of of > supporting excellent client READ bandwidth. But something is > dramatically awry with the freebsd client. (scp, rsync over ssh, all > are near wire speed when I get rid of disk overhead). The -current > server has got an 11TB raidz2 behind it but I'm only READing, not > writing, so all of the discussion about ZFS + NFS sync write problems > seems immaterial, possibly. > > Here's the mount info: > > -current client's output from nfsstat -m (terp is current server, ari > is linux server): > > terp:/raid/library on /mnt/terp/library > nfsv4,minorversion=1,tcp,resvport,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=2798255,timeout=120,retrans=2147483647 > > ari:/d1/library on /mnt/ari > nfsv4,minorversion=0,tcp,resvport,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=2798255,timeout=120,retrans=2147483647 > > Linux client's excerpt from mount output: > > terp:/raid/library on /mnt/terp/library type nfs4 > (rw,relatime,vers=4.0,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.0.10.3,local_lock=none,addr=10.0.10.4) > > BTW, linux -> linux is about 80MB/s > > I tried nfsv3 -current -> -current and still got 2.7MB/s READ. > > I have spent about 5 hours strolling through the archives of the last > several years of discussion about this issue in various forums, but I > didn't see a reference once since Jaunary's thread on freebsd-net, > and > I didn't find a bug entry, so I thought I'd keep it alive, so to > speak. > > I've tried all of the various sysctl tweaks that have been suggested > over time, that would make sense for reads, to no effect. > Please try disabling TSO on the FreeBSD systems and also try an rsize=32768 to see if either of those have any effect. There was also a recent case where disabling msix interrupt handling for the network interface fixed a slow NFS I/O rate issue. (I don't know what the loader conf variable was, but look for something with msix in it for the net device driver. It might be hw.em.enable_msix=0 or something like that?) rick > Russell > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org" >