From owner-freebsd-fs@freebsd.org Tue Mar 29 18:58:35 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D89FFAE2401 for ; Tue, 29 Mar 2016 18:58:35 +0000 (UTC) (envelope-from nishida@asusa.net) Received: from asusam.asj-hosting.net (asusa.asj-hosting.net [219.118.222.245]) (using TLSv1.2 with cipher CAMELLIA256-SHA (256/256 bits)) (Client CN "*.asj-hosting.net", Issuer "Go Daddy Secure Certificate Authority - G2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 869BC1D2F for ; Tue, 29 Mar 2016 18:58:34 +0000 (UTC) (envelope-from nishida@asusa.net) Received: (qmail 63393 invoked by uid 89); 30 Mar 2016 03:58:26 +0900 X-ASJ-Track-ID: <20160329185826.63393.qmail@asusam.asj-hosting.net> X-Spam-Checker-Version: ASJ KMsrv Spam Check Process Internal X-Spam-Status: No, hits=0.0 X-Spam-Flag: No X-Virus-Scanned: ASJ KMsrv Virus Check Process 08041001 X-ASJ-SMTP-Authentication: nishida@asusa.net X-ASJ-Arrival-IP: 50.207.112.201 X-ASJ-SPF-Info: auth X-ASJ-Scan-ID: <1459277905.973411.63386@asusam.asj-hosting.net> X-ASJ-Received-SPF: pass (send with smtp authentication by nishida@asusa.net@50.207.112.201) Received: from gw.asusa.net (HELO rd03.asusa-internal.net) (nishida@asusa.net@50.207.112.201) by asusams.asj-hosting.net with ESMTPS (AES128-SHA encrypted); 30 Mar 2016 03:58:25 +0900 Subject: Re: Problem with FUSE + fts To: freebsd-fs@freebsd.org References: <56F42EF4.5000505@asusa.net> <1294209833.31699182.1458950014610.JavaMail.zimbra@uoguelph.ca> <56F6148D.2030706@asusa.net> From: Hiroshi Nishida Message-ID: <56FAD050.2080707@asusa.net> Date: Tue, 29 Mar 2016 11:58:24 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.7.0 MIME-Version: 1.0 In-Reply-To: <56F6148D.2030706@asusa.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Mar 2016 18:58:35 -0000 Now I figured out what causes this error. Originally, fts outputs an ENOENT error when: FTSENT *p; struct stat sb; stat(p->fts_accpath, &sb); p->fts_ino != sb.st_ino i.e., the inode of p is different from sb.st_ino. They are usually same but sometimes a new inode is allocated to p while scanning the dir in the following way: 1. FUSE lowelevel's forget() is called for some reason and removes all entries from the entry tables, as well as clears all inodes, while scanning the dir. 2. Since p is already removed from FUSE's entry table, FUSE adds it again and allocates a new inode. I don't know why forget() is called for the directory which is still open and clears all inodes, but according to fuse_lowlevel.h /** * Forget about an inode * * This function is called when the kernel removes an inode * from its internal caches. * * The inode's lookup count increases by one for every call to * fuse_reply_entry and fuse_reply_create. The nlookup parameter * indicates by how much the lookup count should be decreased. * * Inodes with a non-zero lookup count may receive request from * the kernel even after calls to unlink, rmdir or (when * overwriting an existing file) rename. Filesystems must handle * such requests properly and it is recommended to defer removal * of the inode until the lookup count reaches zero. Calls to * unlink, remdir or rename will be followed closely by forget * unless the file or directory is open, in which case the * kernel issues forget only after the release or releasedir * calls. * removing the inode should be deferred until the dir is closed. I haven't checked the ref count of each node yet but there seems to be a bug in the above process. Also, there is a suggestion for the hash table but I will post later. Any feedback is appreciated on it. Thank you. On 2016/03/25 21:48, Hiroshi Nishida wrote: > Thank you for your response. > > On 3/25/16 4:53 PM, Rick Macklem wrote: >> I think I see the same thing when doing an "rm -r" on a fuse/GlusterFS volume. > > Unfortunately, it happens also with "find XXX -print", though I have experienced a similar "rm -r" + "XXX: No such file or directory" problem with UFS + SUJ. > And I also verified with truss that in > > _fstat(fd, &sb); > p->fts_ino != sb.st_ino > > stat() system call is called with the same path as p's. > > Anyway, the following patch for lib/libc/gen/fts.c prevents the error but is far from a good solution. > https://github.com/scopedog/FUSE-Test/blob/master/fts.c.patch > It assumes that the filesystem id (f_type in struct statfs) of FUSE is 0xed but I am not sure if it's applicable to all FUSE filesystems. > > I'll look into FUSE source code next week. >> To be honest, I just add a "-f" to the command to shut it up and then it deleted >> the tree. >> >> I think, in general, what readdir() returns after an entry is unlink'd is undefined >> behaviour. As such, the safe way to delete all of a directory is something like: >> - in a loop until readdir() returns EOF >> - opendir() >> - readdir() the first entry >> - unlink() that entry >> - closedir() >> --> So that all you ever do is readdir() the first entry after an opendir(). > > By the way, could you delete all the files with "-f"? > I am testing with a pretty big directory containing 81,000 files/dirs and have never used "-f", but have to "rm -r" again for undeleted entries. > However, the offset problem is very interesting as it seems to be applicable to all filesystems. > > Thank you. > -- Hiroshi Nishida nishida@asusa.net