Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 8 Mar 2016 19:49:30 -0500 (EST)
From:      Rick Macklem <rmacklem@uoguelph.ca>
To:        Paul Mather <paul@gromit.dlib.vt.edu>
Cc:        Ronald Klop <ronald-lists@klop.ws>, freebsd-fs@freebsd.org,  freebsd-arm@freebsd.org
Subject:   Re: Unstable NFS on recent CURRENT
Message-ID:  <1290552239.10146172.1457484570450.JavaMail.zimbra@uoguelph.ca>
In-Reply-To: <08710728-3130-49BE-8BD7-AFE85A31C633@gromit.dlib.vt.edu>
References:  <3DAB3639-8FB8-43D3-9517-94D46EDEC19E@gromit.dlib.vt.edu> <op.ydylazgukndu52@ronaldradial.radialsg.local> <1482595660.8940439.1457405756110.JavaMail.zimbra@uoguelph.ca> <08710728-3130-49BE-8BD7-AFE85A31C633@gromit.dlib.vt.edu>

next in thread | previous in thread | raw e-mail | index | archive | help
Paul Mather wrote:
> On Mar 7, 2016, at 9:55 PM, Rick Macklem <rmacklem@uoguelph.ca> wrote:
> 
> > Paul Mather (forwarded by Ronald Klop) wrote:
> >> On Sun, 06 Mar 2016 02:57:03 +0100, Paul Mather <paul@gromit.dlib.vt.edu>
> >> wrote:
> >> 
> >>> On my BeagleBone Black running 11-CURRENT (r296162) lately I have been
> >>> having trouble with NFS.  I have been doing a buildworld and buildkernel
> >>> with /usr/src and /usr/obj mounted via NFS.  Recently, this process has
> >>> resulted in the buildworld failing at some point, with a variety of
> >>> errors (Segmentation fault; Permission denied; etc.).  Even a "ls -alR"
> >>> of /usr/src doesn't manage to complete.  It errors out thus:
> >>> 
> >>> =====
> >>> [[...]]
> >>> total 0
> >>> ls: ./.svn/pristine/fe: Permission denied
> >>> 
> >>> ./.svn/pristine/ff:
> >>> total 0
> >>> ls: ./.svn/pristine/ff: Permission denied
> >>> ls: fts_read: Permission denied
> >>> =====
> >>> 
> >>> On the console, I get the following:
> >>> 
> >>> newnfs: server 'chumby.chumby.lan' error: fileid changed. fsid
> >>> 94790777:a4385de: expected fileid 0x4, got 0x2. (BROKEN NFS SERVER OR
> >>> MIDDLEWARE)
> >>> 
Oh, I had forgotten this. Here's the comment related to this error.
(about line#445 in sys/fs/nfsclient/nfs_clport.c):
446                      * BROKEN NFS SERVER OR MIDDLEWARE
447 	                 *
448 	                 * Certain NFS servers (certain old proprietary filers ca.
449 	                 * 2006) or broken middleboxes (e.g. WAN accelerator products)
450 	                 * will respond to GETATTR requests with results for a
451 	                 * different fileid.
452 	                 *
453 	                 * The WAN accelerator we've observed not only serves stale
454 	                 * cache results for a given file, it also occasionally serves
455 	                 * results for wholly different files.  This causes surprising
456 	                 * problems; for example the cached size attribute of a file
457 	                 * may truncate down and then back up, resulting in zero
458 	                 * regions in file contents read by applications.  We observed
459 	                 * this reliably with Clang and .c files during parallel build.
460 	                 * A pcap revealed packet fragmentation and GETATTR RPC
461 	                 * responses with wholly wrong fileids.

If you can connect the client->server with a simple switch (or just an RJ45 cable), it
might be worth testing that way. (I don't recall the name of the middleware product, but
I think it was shipped by one of the major switch vendors. I also don't know if the product
supports NFSv4?)

rick

> >>> 
> >>> I am using a FreeBSD/amd64 10.3-PRERELEASE (r296412) as the NFS server.
> >>> On the BeagleBone Black, I am mounting /usr/src and /usr/obj via
> >>> /etc/fstab as follows:
> >>> 
> >>> chumby.chumby.lan:/build/src/head /usr/src nfs rw,nfsv4 0 0
> >>> chumby.chumby.lan:/build/obj/bbb /usr/obj nfs rw,nfsv4 0 0
> >>> 
> >>> 
> >>> /build/src/head and /build/obj/bbb are both ZFS file systems.
> >>> 
> > Is it possible that a ZFS file system has gotten to the point where the
> > i-node# exceeds 32bits? ZFS does support more than 32bits for i-node#s,
> > but FreeBSD does not (it truncates to the low order 32bits).
> > I know diddly about ZFS, so I don't know if you actually have to create
> > more than 4billion files to get the i-node# to exceed 32bits or ???
> > 
> > There has been work done on making ino_t 64bits, but it hasn't made it
> > into FreeBSD-current and I have no idea when it might.
> > 
> > If you could try a build on newly created file systems (or UFS ones
> > instead of ZFS), that would tell you if the above might be the problem.
> 
> 
> I don't think I have that big of a ZFS pool (it's 2 TB). :-)
> 
> It doesn't seem that there are excessive numbers of inodes, and the counts
> match up between the NFS client and server sides.
> 
> In the information below, chumby is the NFS server and beaglebone the client:
> 
> pmather@beaglebone:~ % mount
> /dev/mmcsd0s2a on / (ufs, local, noatime, soft-updates)
> devfs on /dev (devfs, local)
> /dev/mmcsd0s1 on /boot/msdos (msdosfs, local, noatime)
> tmpfs on /tmp (tmpfs, local)
> tmpfs on /var/log (tmpfs, local)
> tmpfs on /var/tmp (tmpfs, local)
> chumby.chumby.lan:/build/src/head on /usr/src (nfs, nfsv4acls)
> chumby.chumby.lan:/build/obj/bbb on /usr/obj (nfs, nfsv4acls)
> pmather@beaglebone:~ % df -i /usr/src /usr/obj
> Filesystem                        1K-blocks    Used     Avail Capacity iused
> ifree %iused  Mounted on
> chumby.chumby.lan:/build/src/head   2097152 1344484    752668    64%  147835
> 1505336    9%   /usr/src
> chumby.chumby.lan:/build/obj/bbb  530875884 1949364 528926520     0%   70814
> 1057853040    0%   /usr/obj
> 
> 
> paul@chumby:/home/paul> df -i /build/src/head /build/obj/bbb
> Filesystem                  1K-blocks    Used     Avail Capacity iused
> ifree %iused  Mounted on
> zroot/SHARED/build/src/head   2097152 1344484    752668    64%  147835
> 1505336    9%   /build/src/head
> zroot/SHARED/build/obj/bbb  530876268 1949364 528926904     0%   70814
> 1057853808    0%   /build/obj/bbb
> 
> 
> On the NFS client system, these are the only NFS-related settings I have in
> /etc/rc.conf:
> 
> nfsuserd_enable="YES"
> nfscbd_enable="YES"
> 
> 
> Would you recommend I try it with nfscbd_enable="NO"?
> 
> I will try NFS from other clients to see whether it's just this FreeBSD/arm
> system that's having problems.
> 
> Cheers,
> 
> Paul.
> 
> 
> > 
> > rick
> > 
> >>> Has anyone else encountered this?  It has only started happening
> >>> recently for me, it seems.  Prior to this, I have been able to do a
> >>> buildworld and buildkernel successfully over NFS.
> >>> 
> >>> Cheers,
> >>> 
> >>> Paul.
> >> 
> >> I cc this to freebsd-fs for you.
> >> 
> >> Ronald.
> >> _______________________________________________
> >> freebsd-fs@freebsd.org mailing list
> >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
> >> 
> > _______________________________________________
> > freebsd-arm@freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-arm
> > To unsubscribe, send any mail to "freebsd-arm-unsubscribe@freebsd.org"
> 
> 



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1290552239.10146172.1457484570450.JavaMail.zimbra>