From owner-freebsd-stable@FreeBSD.ORG Sat Sep 27 07:20:40 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 483461065678; Sat, 27 Sep 2008 07:20:40 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from cs1.cs.huji.ac.il (cs1.cs.huji.ac.il [132.65.16.10]) by mx1.freebsd.org (Postfix) with ESMTP id EBAFF8FC1B; Sat, 27 Sep 2008 07:20:39 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by cs1.cs.huji.ac.il with esmtp id 1KjU6X-0005dk-On; Sat, 27 Sep 2008 10:20:37 +0300 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.2 To: Matthew Dillon In-reply-to: <200809261818.m8QII6jF004333@apollo.backplane.com> References: <20080926081806.GA19055@icarus.home.lan> <20080926095230.GA20789@icarus.home.lan> <20080926143153.GA26978@icarus.home.lan> <200809261818.m8QII6jF004333@apollo.backplane.com> Comments: In-reply-to Matthew Dillon message dated "Fri, 26 Sep 2008 11:18:06 -0700." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sat, 27 Sep 2008 10:20:37 +0300 From: Danny Braniss Message-ID: Cc: freebsd-hackers@freebsd.org, Jeremy Chadwick , freebsd-stable@freebsd.org Subject: Re: bad NFS/UDP performance X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Sep 2008 07:20:40 -0000 > > :> -vfs.nfs.realign_test: 22141777 > :> +vfs.nfs.realign_test: 498351 > :> > :> -vfs.nfsrv.realign_test: 5005908 > :> +vfs.nfsrv.realign_test: 0 > :> > :> +vfs.nfsrv.commit_miss: 0 > :> +vfs.nfsrv.commit_blks: 0 > :> > :> changing them did nothing - or at least with respect to nfs throughput :-) > : > :I'm not sure what any of these do, as NFS is a bit out of my league. > ::-) I'll be following this thread though! > : > :-- > :| Jeremy Chadwick jdc at parodius.com | > > A non-zero nfs_realign_count is bad, it means NFS had to copy the > mbuf chain to fix the alignment. nfs_realign_test is just the > number of times it checked. So nfs_realign_test is irrelevant. > it's nfs_realign_count that matters. > it's zero, so I guess I'm ok there. funny though, on my 'good' machine, vfs.nfsrv.realign_test: 5862999 and on the slow one, it's 0 - but then again the good one has been up for several days. > Several things can cause NFS payloads to be improperly aligned. > Anything from older network drivers which can't start DMA on a > 2-byte boundary, resulting in the 14-byte encapsulation header > causing improper alignment of the IP header & payload, to rpc > embedded in NFS TCP streams winding up being misaligned. > > Modern network hardware either support 2-byte-aligned DMA, allowing > the encapsulation to be 2-byte aligned so the payload winds up being > 4-byte aligned, or support DMA chaining allowing the payload to be > placed in its own mbuf, or pad, etc. > > -- > > One thing I would check is to be sure a couple of nfsiod's are running > on the client when doing your tests. If none are running the RPCs wind > up being more synchronous and less pipelined. Another thing I would > check is IP fragment reassembly statistics (for UDP) - there should be > none for TCP connections no matter what the NFS I/O size selected. > ahh, nfsiod, it seems that it's now dynamicaly started! at least none show when host is idle, after i run my tests there are 20! with ppid 0 need to refresh my NFS knowledge. how can I see the IP fragment reassembly statistics? > (It does seem more likely to be scheduler-related, though). > tend to agree, I tried bith ULE/BSD, but the badness is there. > -Matt > thanks, danny