From owner-freebsd-fs@freebsd.org Wed Jul 1 23:44:56 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D53FF992639 for ; Wed, 1 Jul 2015 23:44:56 +0000 (UTC) (envelope-from email.ahmedkamal@googlemail.com) Received: from mail-wi0-x229.google.com (mail-wi0-x229.google.com [IPv6:2a00:1450:400c:c05::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6434B276C for ; Wed, 1 Jul 2015 23:44:56 +0000 (UTC) (envelope-from email.ahmedkamal@googlemail.com) Received: by widjy10 with SMTP id jy10so70650509wid.1 for ; Wed, 01 Jul 2015 16:44:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=FBKvBo/OQV72Y4PKhLud00weTd0h/WJ+Pq2VeqXdum8=; b=IoTTh/rrC1/4+5XkefXS1RJgIWTvN9NvjvrRsmVWShgMDitFlFpjUNcxI3nx2gPGgY cS2dv8Q51OKBGnMGj0+qLgLQWbdSkMMaRP1dWAlLUglAElH9+2Prth3P3j25+R29M7aV BVCzbhf2Dk8jCkLqwmG2224uxrFaZJ1+os2tlpXe9BGL9nvI/hC801DEx70Ggl4pC7BO 7dAqJ7j43TGaHK/NNy2h9YBBtrw/cP+NzcCOf4DXM1BANI+iIG5Veexcyby5o7sQMjAl BkYNuwGlganBhcNlynH4lYk6uodLN4/kaueaweETuBHxdBmx9q/9e4JCMjx9AS5E8a3w eGLA== X-Received: by 10.194.6.229 with SMTP id e5mr6207641wja.158.1435794294836; Wed, 01 Jul 2015 16:44:54 -0700 (PDT) MIME-Version: 1.0 Received: by 10.28.6.143 with HTTP; Wed, 1 Jul 2015 16:44:35 -0700 (PDT) In-Reply-To: <684628776.2772174.1435793776748.JavaMail.zimbra@uoguelph.ca> References: <684628776.2772174.1435793776748.JavaMail.zimbra@uoguelph.ca> From: Ahmed Kamal Date: Thu, 2 Jul 2015 01:44:35 +0200 Message-ID: Subject: Re: Linux NFSv4 clients are getting (bad sequence-id error!) To: Rick Macklem Cc: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Jul 2015 23:44:57 -0000 Thanks a lot Rick .. Actually I did reboot the nas box after setting the below in /etc/sysctl.conf (just to clean things up) vfs.nfsd.tcphighwater=61000 vfs.nfsd.tcpcachetimeout=300 Since clients are rhel5 only, they don't seem to support v4.1, only v4.0 .. So this one is not an option for now. The good news, is that after raising tcphighwater, these messages (nfsd server cache flooded, try increasing vfs.nfsd.tcphighwater) have completely stopped appearing in freebsd dmesg The not so great news is, after updating sysctl and rebooting the nas box, I still saw a few (NFS: v4 server nas returned a bad sequence-id error!) lines in logs. Users have already left, so I don't know how bad is it .. Could you share more info on what this error means? RedHat seems to think the client can skip-by-1 and choose larger IDs and that would be totally fine ? Also how serious is this error, would it cause NFS session stall like that ? On Thu, Jul 2, 2015 at 1:36 AM, Rick Macklem wrote: > Ahmed Kamal wrote: > > Hi all, > > > > I'm a refugee from linux land. I just set up my first freebsd 10.1 zfs > box, > > sharing /home over nfs. Since every home directory is its own zfs > dataset, > > I chose to use nfsv4 to enable recursively sharing/mounting any directory > > under /home (I understand nfs4 is a must in this scenario!) > > > > I'm able to mount form linux (rhel5 latest kernel) successfully. Users > are > > working fine. However every now and then a user screams that his session > is > > frozen. Usually the processes are stuck in nfs_wait or rpc_* state. I > tried > > using a much newer linux kernel (3.2 however it still faced the same > > problem). The errors in Linux log files are mostly: > > Jul 1 17:41:47 mammoth kernel: NFS: v4 server nas returned a *bad > > sequence-id error*! > > Jul 1 17:52:32 mammoth kernel: nfs4_reclaim_locks: unhandled error -11. > > Zeroing state > > Jul 1 17:52:32 mammoth kernel: nfs4_reclaim_open_state: Lock reclaim > > failed! > > > Btw, a client should only do "reclaim" operations after the server has > replied with NFS4ERR_STALE_CLIENTID or NFS4ERR_STALE_STATEID. I am pretty > certain that the FreeBSD NFSv4 server only generates these replies after > it has rebooted, so assuming the server didn't reboot, I have no idea why > the client would attempt these and am not surprised they failed. > > I'm guessing that the DRC constipation somehow caused the Linux client > to go into recovery mode? > > rick > > > My search led me to (https://access.redhat.com/solutions/1328073) a > > detailed analysis of the issue, which you can read over here > > https://dl.dropboxusercontent.com/u/51939288/nfs4-bad-seq.pdf .. NetApp > > confirmed this was a bug for them (I'm wondering if this is still in > > FreeBSD?!) > > > > PS: Right before sending this, I saw dmesg on the freebsd box advising > > increasing vfs.nfsd.tcphighwater .. So I up'ed that to 64000. I also > up'ed > > the number of nfs server threads (-t) from 10 to 60 (we're roughly 40 > linux > > machines) > > > > Any advice is most appreciated! > > > > Thanks > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > >