From owner-freebsd-arm@freebsd.org Wed Mar 9 00:49:33 2016 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4A30CAC77E0; Wed, 9 Mar 2016 00:49:33 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id C58306AF; Wed, 9 Mar 2016 00:49:32 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:C9qBfB2lXbfTlulqsmDT+DRfVm0co7zxezQtwd8ZsekVI/ad9pjvdHbS+e9qxAeQG96LtLQU1qGM7OjJYi8p39WoiDg6aptCVhsI2409vjcLJ4q7M3D9N+PgdCcgHc5PBxdP9nC/NlVJSo6lPwWB6kO74TNaIBjjLw09fr2zQd6NyZTqnLrts7ToICx2xxOFKYtoKxu3qQiD/uI3uqBFbpgL9x3Sv3FTcP5Xz247bXianhL7+9vitMU7q3cYk7sb+sVBSaT3ebgjBfwdVWx+cjN92cvwqBOWTReT/mBOFSISkwFUGE7L9hz3VIz99Czgua140SieOMTwCrQ1Qiij6alsDxHyhSoNLDJ8+XvS2fB32ZpSvRbpghVjw4POKNWNPed6VqzHetYbWSxNWsdbETJdRI6wct1cIfAGOLNiroL+734Hphi6CAzkUPnqwzRLgnLz9bA93PksFRnGmgcpSYFd+E/Ipcn4Yf9BGdu+y7PFmHCaN6tb X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DQAQAjct9W/61jaINcDoN+bQa6WAENgWkXCoUkSgKBfxQBAQEBAQEBAWMngi2CFQEBBAEBASAEJyALEAIBCBgCAg0ZAgInAQkmAgQIBwQBHASIAw6vW48lAQEBAQEBAQECAQEBAQEBAQEUBHuFHIF7gkeEGwEBBRaDAoE6BYdWhVp0PYhJhWOCcIIyhE2HaYUujlQCHgEBQoIDGYENWR4uAQaIRjR+AQEB X-IronPort-AV: E=Sophos;i="5.22,559,1449550800"; d="scan'208";a="269926196" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 08 Mar 2016 19:49:31 -0500 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 4710715F56D; Tue, 8 Mar 2016 19:49:31 -0500 (EST) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 0P5xFr62_XHf; Tue, 8 Mar 2016 19:49:30 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 9418815F571; Tue, 8 Mar 2016 19:49:30 -0500 (EST) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id b3VHMlovJOMe; Tue, 8 Mar 2016 19:49:30 -0500 (EST) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 7366715F56D; Tue, 8 Mar 2016 19:49:30 -0500 (EST) Date: Tue, 8 Mar 2016 19:49:30 -0500 (EST) From: Rick Macklem To: Paul Mather Cc: Ronald Klop , freebsd-fs@freebsd.org, freebsd-arm@freebsd.org Message-ID: <1290552239.10146172.1457484570450.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <08710728-3130-49BE-8BD7-AFE85A31C633@gromit.dlib.vt.edu> References: <3DAB3639-8FB8-43D3-9517-94D46EDEC19E@gromit.dlib.vt.edu> <1482595660.8940439.1457405756110.JavaMail.zimbra@uoguelph.ca> <08710728-3130-49BE-8BD7-AFE85A31C633@gromit.dlib.vt.edu> Subject: Re: Unstable NFS on recent CURRENT MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.12] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF44 (Win)/8.0.9_GA_6191) Thread-Topic: Unstable NFS on recent CURRENT Thread-Index: k8ThePUeTUqowV1bL9aF3N1wliHkwA== X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Mar 2016 00:49:33 -0000 Paul Mather wrote: > On Mar 7, 2016, at 9:55 PM, Rick Macklem wrote: > > > Paul Mather (forwarded by Ronald Klop) wrote: > >> On Sun, 06 Mar 2016 02:57:03 +0100, Paul Mather > >> wrote: > >> > >>> On my BeagleBone Black running 11-CURRENT (r296162) lately I have been > >>> having trouble with NFS. I have been doing a buildworld and buildkernel > >>> with /usr/src and /usr/obj mounted via NFS. Recently, this process has > >>> resulted in the buildworld failing at some point, with a variety of > >>> errors (Segmentation fault; Permission denied; etc.). Even a "ls -alR" > >>> of /usr/src doesn't manage to complete. It errors out thus: > >>> > >>> ===== > >>> [[...]] > >>> total 0 > >>> ls: ./.svn/pristine/fe: Permission denied > >>> > >>> ./.svn/pristine/ff: > >>> total 0 > >>> ls: ./.svn/pristine/ff: Permission denied > >>> ls: fts_read: Permission denied > >>> ===== > >>> > >>> On the console, I get the following: > >>> > >>> newnfs: server 'chumby.chumby.lan' error: fileid changed. fsid > >>> 94790777:a4385de: expected fileid 0x4, got 0x2. (BROKEN NFS SERVER OR > >>> MIDDLEWARE) > >>> Oh, I had forgotten this. Here's the comment related to this error. (about line#445 in sys/fs/nfsclient/nfs_clport.c): 446 * BROKEN NFS SERVER OR MIDDLEWARE 447 * 448 * Certain NFS servers (certain old proprietary filers ca. 449 * 2006) or broken middleboxes (e.g. WAN accelerator products) 450 * will respond to GETATTR requests with results for a 451 * different fileid. 452 * 453 * The WAN accelerator we've observed not only serves stale 454 * cache results for a given file, it also occasionally serves 455 * results for wholly different files. This causes surprising 456 * problems; for example the cached size attribute of a file 457 * may truncate down and then back up, resulting in zero 458 * regions in file contents read by applications. We observed 459 * this reliably with Clang and .c files during parallel build. 460 * A pcap revealed packet fragmentation and GETATTR RPC 461 * responses with wholly wrong fileids. If you can connect the client->server with a simple switch (or just an RJ45 cable), it might be worth testing that way. (I don't recall the name of the middleware product, but I think it was shipped by one of the major switch vendors. I also don't know if the product supports NFSv4?) rick > >>> > >>> I am using a FreeBSD/amd64 10.3-PRERELEASE (r296412) as the NFS server. > >>> On the BeagleBone Black, I am mounting /usr/src and /usr/obj via > >>> /etc/fstab as follows: > >>> > >>> chumby.chumby.lan:/build/src/head /usr/src nfs rw,nfsv4 0 0 > >>> chumby.chumby.lan:/build/obj/bbb /usr/obj nfs rw,nfsv4 0 0 > >>> > >>> > >>> /build/src/head and /build/obj/bbb are both ZFS file systems. > >>> > > Is it possible that a ZFS file system has gotten to the point where the > > i-node# exceeds 32bits? ZFS does support more than 32bits for i-node#s, > > but FreeBSD does not (it truncates to the low order 32bits). > > I know diddly about ZFS, so I don't know if you actually have to create > > more than 4billion files to get the i-node# to exceed 32bits or ??? > > > > There has been work done on making ino_t 64bits, but it hasn't made it > > into FreeBSD-current and I have no idea when it might. > > > > If you could try a build on newly created file systems (or UFS ones > > instead of ZFS), that would tell you if the above might be the problem. > > > I don't think I have that big of a ZFS pool (it's 2 TB). :-) > > It doesn't seem that there are excessive numbers of inodes, and the counts > match up between the NFS client and server sides. > > In the information below, chumby is the NFS server and beaglebone the client: > > pmather@beaglebone:~ % mount > /dev/mmcsd0s2a on / (ufs, local, noatime, soft-updates) > devfs on /dev (devfs, local) > /dev/mmcsd0s1 on /boot/msdos (msdosfs, local, noatime) > tmpfs on /tmp (tmpfs, local) > tmpfs on /var/log (tmpfs, local) > tmpfs on /var/tmp (tmpfs, local) > chumby.chumby.lan:/build/src/head on /usr/src (nfs, nfsv4acls) > chumby.chumby.lan:/build/obj/bbb on /usr/obj (nfs, nfsv4acls) > pmather@beaglebone:~ % df -i /usr/src /usr/obj > Filesystem 1K-blocks Used Avail Capacity iused > ifree %iused Mounted on > chumby.chumby.lan:/build/src/head 2097152 1344484 752668 64% 147835 > 1505336 9% /usr/src > chumby.chumby.lan:/build/obj/bbb 530875884 1949364 528926520 0% 70814 > 1057853040 0% /usr/obj > > > paul@chumby:/home/paul> df -i /build/src/head /build/obj/bbb > Filesystem 1K-blocks Used Avail Capacity iused > ifree %iused Mounted on > zroot/SHARED/build/src/head 2097152 1344484 752668 64% 147835 > 1505336 9% /build/src/head > zroot/SHARED/build/obj/bbb 530876268 1949364 528926904 0% 70814 > 1057853808 0% /build/obj/bbb > > > On the NFS client system, these are the only NFS-related settings I have in > /etc/rc.conf: > > nfsuserd_enable="YES" > nfscbd_enable="YES" > > > Would you recommend I try it with nfscbd_enable="NO"? > > I will try NFS from other clients to see whether it's just this FreeBSD/arm > system that's having problems. > > Cheers, > > Paul. > > > > > > rick > > > >>> Has anyone else encountered this? It has only started happening > >>> recently for me, it seems. Prior to this, I have been able to do a > >>> buildworld and buildkernel successfully over NFS. > >>> > >>> Cheers, > >>> > >>> Paul. > >> > >> I cc this to freebsd-fs for you. > >> > >> Ronald. > >> _______________________________________________ > >> freebsd-fs@freebsd.org mailing list > >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs > >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > >> > > _______________________________________________ > > freebsd-arm@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-arm > > To unsubscribe, send any mail to "freebsd-arm-unsubscribe@freebsd.org" > >