From owner-freebsd-fs@freebsd.org Mon Sep 11 05:15:57 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E987AE1BCE6 for ; Mon, 11 Sep 2017 05:15:57 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [70.36.157.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D003470A5D for ; Mon, 11 Sep 2017 05:15:57 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (localhost [IPv6:::1]) by chez.mckusick.com (8.15.2/8.15.2) with ESMTP id v8B5JVmf060773; Sun, 10 Sep 2017 22:19:33 -0700 (PDT) (envelope-from mckusick@chez.mckusick.com) Message-Id: <201709110519.v8B5JVmf060773@chez.mckusick.com> From: Kirk McKusick To: Andreas Longwitz Subject: Re: fsync: giving up on dirty on ufs partitions running vfs_write_suspend() cc: freebsd-fs@freebsd.org In-reply-to: <59B29E35.9000506@incore.de> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <60771.1505107171.1@chez.mckusick.com> Content-Transfer-Encoding: quoted-printable Date: Sun, 10 Sep 2017 22:19:31 -0700 X-Spam-Status: No, score=0.1 required=5.0 tests=MISSING_MID, UNPARSEABLE_RELAY autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on chez.mckusick.com X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Sep 2017 05:15:58 -0000 > Date: Fri, 08 Sep 2017 15:42:13 +0200 > From: Andreas Longwitz > To: freebsd-fs@freebsd.org > Subject: fsync: giving up on dirty on ufs partitions running vfs_write_s= uspend() > = > I try to describe the cause for the "fsync: given up on dirty" problem > described in > = > https://lists.freebsd.org/pipermail/freebsd-fs/2012-February/013804.html > or > https://lists.freebsd.org/pipermail/freebsd-fs/2013-August/018163.html > > Now I run FreeBSD 10.3 Stable r317936 and sometimes I see messages like > = > dssbkp4 kernel: fsync: giving up on dirty > dssbkp4 kernel: 0xfffff80040d6c938: tag devfs, type VCHR > dssbkp4 kernel: usecount 1, writecount 0, refcount 47 > mountedhere 0xfffff8004083a200 > dssbkp4 kernel: flags (VI_ACTIVE) > dssbkp4 kernel: v_object 0xfffff800409b3500 ref 0 pages > 1138 cleanbuf 42 dirtybuf 4 > dssbkp4 kernel: lock type devfs: EXCL by thread > 0xfffff800403a8a00 (pid 26, g_journal switcher, tid 100181) > dssbkp4 kernel: dev mirror/gmbkp4p5.journal > dssbkp4 kernel: GEOM_JOURNAL: Cannot suspend file system > /home (error=3D35). > = > on all of my servers running gjournal. Similar messages can be seen when > a snapshot is taken (e.g. dump -L) on a arbitrary ufs partition. In all > these cases the function vfs_write_suspend() was called which returned > EAGAIN. This error code is set in vop_stdfsync(), when the above > messages are created. > = > First I was confused about the "mountedhere" address, because the given > address does not point to a "struct mount" but (as type =3D VCHR > indicates) to a "struct cdev". Threfore I suggest the following patch to > improve the output of vn_printf() using the textstrings from defines in > /sys/sys/vnode.h: > = > --- vfs_subr.c.orig 2017-05-08 14:17:38.000000000 +0200 > +++ vfs_subr.c 2017-08-30 10:45:47.549740000 +0200 > @@ -3003,6 +3003,8 @@ > static char *typename[] =3D > {"VNON", "VREG", "VDIR", "VBLK", "VCHR", "VLNK", "VSOCK", "VFIFO", "VBA= D", > "VMARKER"}; > +static char *typetext[] =3D > +{"", "", "mountedhere", "", "rdev", "", "socket", "fifoinfo", "", ""}; > = > void > vn_printf(struct vnode *vp, const char *fmt, ...) > @@ -3016,8 +3018,9 @@ > va_end(ap); > printf("%p: ", (void *)vp); > printf("tag %s, type %s\n", vp->v_tag, typename[vp->v_type]); > - printf(" usecount %d, writecount %d, refcount %d mountedhere > %p\n", > - vp->v_usecount, vp->v_writecount, vp->v_holdcnt, > vp->v_mountedhere); > + printf(" usecount %d, writecount %d, refcount %d %s %p\n", > + vp->v_usecount, vp->v_writecount, vp->v_holdcnt, > typetext[vp->v_type], > + vp->v_mountedhere); > buf[0] =3D '\0'; > buf[1] =3D '\0'; > if (vp->v_vflag & VV_ROOT) I concur with the above change and will make it. > Second I found, that the "dirty" situation during vfs_write_suspend() > only occurs when a big file (more than 10G on a partition of 116G) is > removed. If vfs_write_suspend() is called immediately after "rm > bigfile", then in vop_stdfsync() 1000 tries (maxretry) are done to wait > for the "rm bigfile" to complete. Because a lot of bitmap writes must be > done, the value 1000 is not sufficient on my servers. I have increased > maxretry and in the worst case I saw 8650 tries to complete without > "dirty". In this case the time spent in vop_stdfsync() was about 0,5 > seconds. The following patch solves the "dirty problem" for me: > = > --- vfs_default.c.orig 2016-10-24 12:26:57.000000000 +0200 > +++ vfs_default.c 2017-09-08 12:49:18.059970000 +0200 > @@ -644,7 +644,7 @@ > struct bufobj *bo; > struct buf *nbp; > int error =3D 0; > - int maxretry =3D 1000; /* large, arbitrarily chosen */ > + int maxretry =3D 100000; /* large, arbitrarily chosen */ > = > bo =3D &vp->v_bufobj; > BO_LOCK(bo); > = > --- > Andreas Longwitz This message has plagued me for years. It started out as a panic, then got changed to a printf because I could not get rid of it. I was never able to figure out why it should take more than five iterations to finish, but obviously it takes more. The 1000 number was picked because that just seemed insanely large and I did not want to iterate forever. I have no problem with bumping up the iteration count if there is some way to figure out that each iteration is making forward progress (so we know that we are not in an infinite loop). Can you come up with a scheme that can measure forward progress? I would much prefer that to just making this number ever bigger. Kirk McKusick From owner-freebsd-fs@freebsd.org Mon Sep 11 10:20:55 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6D06FE028E5 for ; Mon, 11 Sep 2017 10:20:55 +0000 (UTC) (envelope-from Nikolaus@rath.org) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3C3BE7F276 for ; Mon, 11 Sep 2017 10:20:54 +0000 (UTC) (envelope-from Nikolaus@rath.org) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 7D15220C21 for ; Mon, 11 Sep 2017 06:20:53 -0400 (EDT) Received: from frontend2 ([10.202.2.161]) by compute1.internal (MEProxy); Mon, 11 Sep 2017 06:20:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-sender:x-me-sender:x-sasl-enc:x-sasl-enc; s= fm1; bh=zkIiic3t2LXSBV4PMMefP9ZORKWuAaI70v7sJ4MPCWM=; b=Ear0xZp2 wzeyCfG4G8LaeYLln6Pyw/d0TrN72piwNi/f/SvD+eVmVFGKZHMvEML3n7UVC4/L t+uvwGkacek0x/cJ+SGh6HJAcphPCr82h70QAnoL0ueC7EFiFzQlBKHSYYgaN6sA zDYwSxCVxA/HWZwvrr7311kqlUKG5dgF4sry0ZRfGG9M9yzAfszJdntOrLb+hVmk ympe3ng30ajIPxV4ObfmN23LoqnhZNvDDQNVsYMaRtabw6bd8hTZB5GCK+MEq7iU 8cG2gnHdosAlFEq1J2zDJ6fDBGxGh17waDEJ98JYWGix5q2T69rnLioV/lQV/ZMS 13fmcm+kJsbIxw== X-ME-Sender: X-Sasl-enc: XgyprrMOPev39c+AS3DWgEaQp18hGunnbBMYobJTWR5B 1505125253 Received: from ebox.rath.org (ebox.rath.org [45.79.69.51]) by mail.messagingengine.com (Postfix) with ESMTPA id 399E5240B3 for ; Mon, 11 Sep 2017 06:20:53 -0400 (EDT) Received: from vostro.rath.org (vostro [192.168.12.4]) by ebox.rath.org (Postfix) with ESMTPS id 40E04113 for ; Mon, 11 Sep 2017 10:20:52 +0000 (UTC) Received: by vostro.rath.org (Postfix, from userid 1000) id DF775102310; Mon, 11 Sep 2017 11:20:50 +0100 (BST) From: Nikolaus Rath To: freebsd-fs@freebsd.org Subject: Re: umount() taking minutes for FUSE filesystems References: <87bmn44ruu.fsf@vostro.rath.org> <87o9qyrbs8.fsf@vostro.rath.org> <2FAD66DE-031B-4B36-9E85-C7BC6B52B5E6@gmail.com> <29de6425-9f92-3bd8-f446-1c9dded33b15@freebsd.org> <87k21dzdrp.fsf@thinkpad.rath.org> <201709051811.v85IBmbO005440@higson.cam.lispworks.com> <87ingugw2v.fsf@thinkpad.rath.org> <201709081143.v88BhEOn001626@higson.cam.lispworks.com> <87lglp6zj8.fsf@vostro.rath.org> Mail-Copies-To: never Mail-Followup-To: freebsd-fs@freebsd.org Date: Mon, 11 Sep 2017 11:20:50 +0100 In-Reply-To: <87lglp6zj8.fsf@vostro.rath.org> (Nikolaus Rath's message of "Fri, 08 Sep 2017 15:22:19 +0100") Message-ID: <87a821a64d.fsf@vostro.rath.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Sep 2017 10:20:55 -0000 On Sep 08 2017, Nikolaus Rath wrote: > On Sep 08 2017, Martin Simmons wrote: >>>>>>> On Thu, 07 Sep 2017 21:14:32 +0200, Nikolaus Rath said: >>>=20 >>> On Sep 05 2017, Martin Simmons wrote: >>> >> Probably the crucial difference is that the test that takes long exi= ts >>> >> its main loop on its own and then informs the FUSE kernel module abo= ut >>> >> that, while the other tests terminate the main loop because the kern= el >>> >> module tells them to do so. >>> > >>> > What does "informs the FUSE kernel module about that" do to inform it? >>>=20 >>> It calls unmount. >>> https://github.com/libfuse/libfuse/blob/master/lib/mount_bsd.c#L127 >> >> It seems to me that the following occurs: >> >> 1. The user program exits the main loop. >> 2. The user program sends unmount to the kernel. >> 3. The kernel sends FUSE_DESTROY to the user program. >> 4. The kernel waits for the user program to respond to FUSE_DESTROY. >> 5. The user program does not respond because it has exited the main loop. >> 6. The kernel wait times out after 60 seconds. >> >> Perhaps the solution is to close the fd before calling unmount(), like t= he >> Linux code in https://github.com/libfuse/libfuse/blob/master/lib/mount.c= #L275 >> does? > > That sounds reasonable, will give it a shot. This indeed fixed the problem. Thanks! Best, -Nikolaus --=20 GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F =C2=BBTime flies like an arrow, fruit flies like a Banana.=C2= =AB From owner-freebsd-fs@freebsd.org Mon Sep 11 17:10:41 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A4CFBE172F1 for ; Mon, 11 Sep 2017 17:10:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 937D3692EE for ; Mon, 11 Sep 2017 17:10:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v8BHAf6N061770 for ; Mon, 11 Sep 2017 17:10:41 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219972] Unable to zpool export following some zfs recv Date: Mon, 11 Sep 2017 17:10:41 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: matthias.pfaller@familie-pfaller.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Sep 2017 17:10:41 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219972 --- Comment #5 from Matthias Pfaller = --- The same just happened to me. dtrace: [root@zork ~]# dtrace -n 'sdt:::set-error /arg0 =3D=3D EBUSY/ { printf("fun= c %s", probefunc); stack(); }' dtrace: description 'sdt:::set-error ' matched 1 probe CPU ID FUNCTION:NAME 0 63932 none:set-error func none zfs.ko`zfs_ioc_pool_export+0x30 zfs.ko`zfsdev_ioctl+0x6d6 kernel`devfs_ioctl_f+0x128 kernel`kern_ioctl+0x255 kernel`sys_ioctl+0x16f kernel`amd64_syscall+0x6c4 kernel`0xffffffff80ec392b [root@zork ~]# zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT backup 1.81T 638G 1.19T - 15% 34% 1.04x ONLINE /backup data 2.72T 2.29T 438G - 28% 84% 1.00x ONLINE - root 111G 51.0G 60.0G - 48% 45% 1.00x ONLINE - [root@zork ~]# zdb backup zdb: can't open 'backup': No such file or directory ZFS_DBGMSG(zdb): Any additional information I could provide? This will happen again on next monday... --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Sep 11 17:12:19 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 89DCCE17546 for ; Mon, 11 Sep 2017 17:12:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7839C6959B for ; Mon, 11 Sep 2017 17:12:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v8BHCJkC071446 for ; Mon, 11 Sep 2017 17:12:19 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219972] Unable to zpool export following some zfs recv Date: Mon, 11 Sep 2017 17:12:19 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: matthias.pfaller@familie-pfaller.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Sep 2017 17:12:19 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219972 --- Comment #6 from Matthias Pfaller = --- I forgot: [root@zork ~]# zpool list -v backup NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALT= H=20 ALTROOT backup 1.81T 638G 1.19T - 15% 34% 1.04x ONLIN= E=20 /backup gpt/backup.eli 1.81T 638G 1.19T - 15% 34% --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Sep 12 17:08:54 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7EEBCE12E9C; Tue, 12 Sep 2017 17:08:54 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 601A27DB1F; Tue, 12 Sep 2017 17:08:53 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (106-69-250-203.dyn.iinet.net.au [106.69.250.203]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id v8CH8gIk023598 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Tue, 12 Sep 2017 10:08:45 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: extending the maximum filename length (pointer to patch)[request for input] To: cem@freebsd.org Cc: freebsd-current , "freebsd-fs@freebsd.org" References: <0154558d-b2ad-af97-3960-3e392678f709@freebsd.org> <8d04540b-6daf-aa13-5648-0ec2541cbae6@freebsd.org> From: Julian Elischer Message-ID: Date: Wed, 13 Sep 2017 01:08:36 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Sep 2017 17:08:54 -0000 On 12/9/17 2:17 pm, Conrad Meyer wrote: > On Sat, Sep 9, 2017 at 9:09 AM, Julian Elischer wrote: >> maybe we could get it into -current. >> It'd be silly to have to have people re-inventing hte wheel all the time. >> How about you put those changes into the reviews.freebsd.org and we can get >> some general consensus on them. >> We'll have to do similar for the Asian customers and anyone who uses UTF-8. >> So it >> would be silly to have to develop it all again (but subtly different of >> course). >> >> The key issue is how many system calls and other APIs would be broken, >> and how many would be broken in a non backwards compatible way? >> >> We would need it in a stable/10 and 11 branch but if the patch is isolated >> enough we could carry it forward until we get to 12. >> >> One has to allow people to do whatever they are used to with Windows. >> And in this case the issue is serving files over samba to windows machines. > Hey Julian, > > I've thrown the patch up at https://reviews.freebsd.org/D12330 . I > haven't actually tested it on FreeBSD, but it does compile. We also > have some patches against contrib/pjdfstest to fix those tests against > long file names, but I think we can hold off on those changes until > we've nailed down what the architectural change will be (if any). thanks! that looks a lot like a proof -of concept patch we derived a while back but never really tested. The issue for us is that using UTF-8 the filenames become too short for common usage in China and Japan. Apparently they routinely nae files with the contents of a small novella. e.g. “这是一个测试多字符文件名长度的文件目的是命名一个文件用中文或者日文或者韩文字符并且要求字符长度超过八十五个字符然后拷贝该文件到我们的共享文件夹看看是否能够拷贝该文件到我.txt” (I have no idea what that says but apparently it's a real filename from a windows machine that blew up when written via samba.) Does anyone else have any thoughts about whether FreeBSD 12 might grow longer path/filename support?  (I'm told Linux uses 1K and 4096 for filenames and path length.) Julian > > It's quite possible this accidentally breaks even more APIs than > expected and we should do some fine tuning to reduce the damage. Our > $WORK product mostly doesn't care about ABI, so we may not have > noticed some ABI breakage. > > If anyone else is interested, please subscribe or add yourself as a > reviewer on the phabricator revision. > > Best, > Conrad > From owner-freebsd-fs@freebsd.org Tue Sep 12 17:16:14 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 18DFDE13766; Tue, 12 Sep 2017 17:16:14 +0000 (UTC) (envelope-from jonathan@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E94127E165; Tue, 12 Sep 2017 17:16:13 +0000 (UTC) (envelope-from jonathan@FreeBSD.org) Received: from [134.153.27.124] (unknown [127.0.1.132]) by freefall.freebsd.org (Postfix) with ESMTP id DD88B18567; Tue, 12 Sep 2017 17:16:12 +0000 (UTC) (envelope-from jonathan@FreeBSD.org) From: "Jonathan Anderson" To: "Julian Elischer" Cc: cem@freebsd.org, freebsd-current , "freebsd-fs@freebsd.org" Subject: Re: extending the maximum filename length (pointer to patch)[request for input] Date: Tue, 12 Sep 2017 14:46:12 -0230 Message-ID: <3C985916-3641-45B7-A721-C4F89A699B70@FreeBSD.org> In-Reply-To: References: <0154558d-b2ad-af97-3960-3e392678f709@freebsd.org> <8d04540b-6daf-aa13-5648-0ec2541cbae6@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: base64 X-Mailer: MailMate (1.9.6r5347) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Sep 2017 17:16:14 -0000 T24gMTIgU2VwIDIwMTcsIGF0IDE0OjM4LCBKdWxpYW4gRWxpc2NoZXIgd3JvdGU6Cgo+IOKAnOi/ meaYr+S4gOS4qua1i+ivleWkmuWtl+espuaWh+S7tuWQjemVv+W6pueahOaWh+S7tuebrueahOaY r+WRveWQjeS4gOS4quaWh+S7tueUqOS4reaWh+aIluiAheaXpeaWh+aIluiAhemfqeaWh+Wtl+es puW5tuS4lOimgeaxguWtl+espumVv+W6pui2hei/h+WFq+WNgeS6lOS4quWtl+espueEtuWQjuaL t+i0neivpeaWh+S7tuWIsOaIkeS7rOeahOWFseS6q+aWh+S7tuWkueeci+eci+aYr+WQpuiDveWk n+aLt+i0neivpeaWh+S7tuWIsOaIkS50eHTigJ0KPiAoSSBoYXZlIG5vIGlkZWEgd2hhdCB0aGF0 IHNheXMgYnV0IGFwcGFyZW50bHkgaXQncyBhIHJlYWwgZmlsZW5hbWUgCj4gZnJvbSBhIHdpbmRv d3MgbWFjaGluZSB0aGF0IGJsZXcgdXAgd2hlbiB3cml0dGVuIHZpYSBzYW1iYS4pCgpHb29nbGUg VHJhbnNsYXRlIHNheXMsIGFtdXNpbmdseToKIlRoaXMgaXMgYSB0ZXN0IGZpbGUgZm9yIHRoZSBs ZW5ndGggb2YgdGhlIGZpbGUgbmFtZS4gVGhlIHB1cnBvc2UgaXMgdG8gCm5hbWUgYSBmaWxlIGlu IENoaW5lc2Ugb3IgSmFwYW5lc2Ugb3IgS29yZWFuIGNoYXJhY3RlcnMgYW5kIHJlcXVpcmUgdGhl IApjaGFyYWN0ZXIgdG8gYmUgbG9uZ2VyIHRoYW4gODUgY2hhcmFjdGVycyBhbmQgdGhlbiBjb3B5 IHRoZSBmaWxlIGludG8gCm91ciBzaGFyZWQgZm9sZGVyIHRvIHNlZSBpZiBpdCBjYW4gY29weSB0 aGUgZmlsZSBUbyBtZSIgKC50eHQsIEkgZ3Vlc3MpCgpObyBtYXR0ZXIgd2hhdCBudW1iZXIgeW91 IGNob29zZSBmb3IgYSBwYXRoIGxlbmd0aCwgeW91J3JlIG5ldmVyIGdvaW5nIAp0byB3aW4gYWdh aW5zdCB0aGF0IHNwZWNpZmljIHVzZXIuIDopCgoKSm9uCi0tCkpvbmF0aGFuIEFuZGVyc29uCmpv bmF0aGFuQEZyZWVCU0Qub3JnCg== From owner-freebsd-fs@freebsd.org Tue Sep 12 21:11:57 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 74476E1E40A; Tue, 12 Sep 2017 21:11:57 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x231.google.com (mail-wm0-x231.google.com [IPv6:2a00:1450:400c:c09::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 101C538B9; Tue, 12 Sep 2017 21:11:57 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x231.google.com with SMTP id g206so257915wme.0; Tue, 12 Sep 2017 14:11:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=fGuCK/04/QEVpa2YAd3EskymPkxZWvObz4jlqF5X0c0=; b=IVhQiEvNcnTBntRUgPW74eGffZHGFYryeUaxTNbTe4WCAJWCREfsBuqsRFf/oxwAA8 vC07emE0mPA/S0+n8gYuTgph/68VoaJbnvL6b+1r9pEAwXzUOjR1SzBHzImTcJwyjRWg yMpWIUMWUsK6i4ILhT/JmgOCWY8fXX83t8kTBKyv7wVUejpqTWll9Mde2dEloJA1MFI5 TvGV+mXxeT+dWFbMI/4LDzo0MnXqJkErVUgx16ni7hxlRbt8G53kELtdPp4H1KASpprU 6OB1Qf6NBFY3IOhGYgLxjB1sfiXoGD0vYFt44P0KjJsIAYfWOAw0QJiCcgXTpQPNR2PL j+Uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=fGuCK/04/QEVpa2YAd3EskymPkxZWvObz4jlqF5X0c0=; b=bawjEDcb7CDLwDjMhk5BsToHN1CF7i/rKqp5Iw4z3Jyztft+RI+BmpI32Rp4r0KWma ndP+V6jpQ3S+MtE+m7fD40FXGOx12bas5XETDa6PIZV73ivXry01YYyxhqB73MhDcC5g rOZSkYX2LM3SRnyM/nZ52XnWiRg13OEd4U0FMJxBJu1SEREjQa/eOktwYPg9+ElvN7IS 61jppSwuHzzmsD/oBrH4ElRc/IzF2apz22QPJRM7PsK0eZxJ5r5sESEmtEV9PZzI/CDE Z8WTcnVIJ/+64jwMGgYP4i2oJ9QsmNVi7MS/w+9ipV3Ve9lBfuFEj7OuHKkQyVqXE5Nj EIdw== X-Gm-Message-State: AHPjjUhW66GKCfZvcLkH7AtJqxOVmVVB1KgRw3uBzaJo+INDc86cL1tW cCOpBjcnMCl/eb2Gw14= X-Google-Smtp-Source: AOwi7QBdCHkBER+Q3QkVr3vdN7RA0ZKUD1+I8+G/+H4K4t/jcb3Hyz7z4XUi/do/nroZSu9VAd9Nug== X-Received: by 10.28.156.203 with SMTP id f194mr700439wme.149.1505250714447; Tue, 12 Sep 2017 14:11:54 -0700 (PDT) Received: from ben-3.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id v5sm9378395wmb.0.2017.09.12.14.11.52 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 12 Sep 2017 14:11:53 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: extending the maximum filename length (pointer to patch)[request for input] From: Ben RUBSON In-Reply-To: Date: Tue, 12 Sep 2017 23:11:52 +0200 Cc: "freebsd-fs@freebsd.org" , freebsd-current , Julian Elischer Content-Transfer-Encoding: quoted-printable Message-Id: <041393BA-94E3-4D22-95C8-EF0634746DC3@gmail.com> References: <0154558d-b2ad-af97-3960-3e392678f709@freebsd.org> <8d04540b-6daf-aa13-5648-0ec2541cbae6@freebsd.org> To: cem@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Sep 2017 21:11:57 -0000 On 12/9/17 2:17 pm, Conrad Meyer wrote: > On Sat, Sep 9, 2017 at 9:09 AM, Julian Elischer = wrote: >> maybe we could get it into -current. >> It'd be silly to have to have people re-inventing hte wheel all the = time. >> How about you put those changes into the reviews.freebsd.org and we = can get >> some general consensus on them. >> We'll have to do similar for the Asian customers and anyone who uses = UTF-8. >> So it >> would be silly to have to develop it all again (but subtly different = of >> course). >>=20 >> The key issue is how many system calls and other APIs would be = broken, >> and how many would be broken in a non backwards compatible way? >>=20 >> We would need it in a stable/10 and 11 branch but if the patch is = isolated >> enough we could carry it forward until we get to 12. >>=20 >> One has to allow people to do whatever they are used to with Windows. >> And in this case the issue is serving files over samba to windows = machines. > Hey Julian, >=20 > I've thrown the patch up at https://reviews.freebsd.org/D12330 . I > haven't actually tested it on FreeBSD, but it does compile. We also > have some patches against contrib/pjdfstest to fix those tests against > long file names, but I think we can hold off on those changes until > we've nailed down what the architectural change will be (if any). Hi Conrad, This patch makes me think about another related bug #184340 : https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D184340 It is about PATH_MAX which in some cases can be too small. Not sure if it's the case / and how to do it, but perhaps it is time to raise some other limits / think about a global solution regarding these length limits ? Many thanks ! Ben From owner-freebsd-fs@freebsd.org Tue Sep 12 21:24:03 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 10345E1EDBA; Tue, 12 Sep 2017 21:24:03 +0000 (UTC) (envelope-from rainer@ultra-secure.de) Received: from connect.ultra-secure.de (connect.ultra-secure.de [88.198.71.201]) by mx1.freebsd.org (Postfix) with ESMTP id 2EECA634E9; Tue, 12 Sep 2017 21:24:01 +0000 (UTC) (envelope-from rainer@ultra-secure.de) Received: (Haraka outbound); Tue, 12 Sep 2017 23:23:11 +0200 Authentication-Results: connect.ultra-secure.de; iprev=pass; auth=pass (plain); spf=none smtp.mailfrom=ultra-secure.de Received-SPF: None (connect.ultra-secure.de: domain of ultra-secure.de does not designate 217.71.83.52 as permitted sender) receiver=connect.ultra-secure.de; identity=mailfrom; client-ip=217.71.83.52; helo=[192.168.1.200]; envelope-from= Received: from [192.168.1.200] (217-071-083-052.ip-tech.ch [217.71.83.52]) by connect.ultra-secure.de (Haraka/2.6.2-toaster) with ESMTPSA id F2D42BBD-4578-47E4-B239-97A38462D027.1 envelope-from (authenticated bits=0) (version=TLSv1/SSLv3 cipher=AES256-SHA verify=NO); Tue, 12 Sep 2017 23:23:08 +0200 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: extending the maximum filename length (pointer to patch)[request for input] From: Rainer Duffner In-Reply-To: <041393BA-94E3-4D22-95C8-EF0634746DC3@gmail.com> Date: Tue, 12 Sep 2017 23:23:56 +0200 Cc: cem@freebsd.org, "freebsd-fs@freebsd.org" , freebsd-current Message-Id: References: <0154558d-b2ad-af97-3960-3e392678f709@freebsd.org> <8d04540b-6daf-aa13-5648-0ec2541cbae6@freebsd.org> <041393BA-94E3-4D22-95C8-EF0634746DC3@gmail.com> To: Ben RUBSON X-Mailer: Apple Mail (2.3124) X-Haraka-GeoIP: EU, CH, 451km X-Haraka-ASN: 24951 X-Haraka-GeoIP-Received: X-Haraka-ASN: 24951 217.71.80.0/20 X-Haraka-ASN-CYMRU: asn=24951 net=217.71.80.0/20 country=CH assignor=ripencc date=2003-08-07 X-Haraka-FCrDNS: 217-071-083-052.ip-tech.ch X-Haraka-p0f: os="Mac OS X " link_type="DSL" distance=15 total_conn=5 shared_ip=N X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on spamassassin X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, HTML_MESSAGE autolearn=ham autolearn_force=no version=3.4.1 X-Haraka-Karma: score: 6, good: 3919, bad: 1, connections: 4345, history: 3918, asn_score: 452, asn_connections: 464, asn_good: 452, asn_bad: 0, pass:asn, asn_all_good, relaying Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Sep 2017 21:24:03 -0000 > Am 12.09.2017 um 23:11 schrieb Ben RUBSON : >=20 > On 12/9/17 2:17 pm, Conrad Meyer wrote: >> On Sat, Sep 9, 2017 at 9:09 AM, Julian Elischer = wrote: >>> maybe we could get it into -current. >>> It'd be silly to have to have people re-inventing hte wheel all the = time. >>> How about you put those changes into the reviews.freebsd.org and we = can get >>> some general consensus on them. >>> We'll have to do similar for the Asian customers and anyone who uses = UTF-8. >>> So it >>> would be silly to have to develop it all again (but subtly different = of >>> course). >>>=20 >>> The key issue is how many system calls and other APIs would be = broken, >>> and how many would be broken in a non backwards compatible way? >>>=20 >>> We would need it in a stable/10 and 11 branch but if the patch is = isolated >>> enough we could carry it forward until we get to 12. >>>=20 >>> One has to allow people to do whatever they are used to with = Windows. >>> And in this case the issue is serving files over samba to windows = machines. >> Hey Julian, >>=20 >> I've thrown the patch up at https://reviews.freebsd.org/D12330 . I >> haven't actually tested it on FreeBSD, but it does compile. We also >> have some patches against contrib/pjdfstest to fix those tests = against >> long file names, but I think we can hold off on those changes until >> we've nailed down what the architectural change will be (if any). >=20 > Hi Conrad, >=20 > This patch makes me think about another related bug #184340 : > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D184340 = > It is about PATH_MAX which in some cases can be too small. >=20 > Not sure if it's the case / and how to do it, > but perhaps it is time to raise some other limits / > think about a global solution regarding these length limits ? >=20 > Many thanks ! >=20 > Ben And there=E2=80=99s also this bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D215067 = "g_dev_taste: make_dev_p() failed on importing pool with snapshots with = long names=E2=80=9C But maybe that has nothing to do with it. From owner-freebsd-fs@freebsd.org Tue Sep 12 22:54:03 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 32A88E23298; Tue, 12 Sep 2017 22:54:03 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: from mail-yw0-f179.google.com (mail-yw0-f179.google.com [209.85.161.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E99DD66DCA; Tue, 12 Sep 2017 22:54:02 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: by mail-yw0-f179.google.com with SMTP id v72so29278632ywa.3; Tue, 12 Sep 2017 15:54:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc; bh=Cu6pZe6V5Dn7lfbAu6cOclFpi14j/v7n96PUR5lNWj8=; b=hJkOow3WMh5QAIZ4g+65jhA3YmBZPvEJy5VdpvtnRGlis82mQyamcdUaP9w+wUvyqK MTwdcC3ZhsJUmlTtfzwY5ggsyCy09W8ScRmYnAMtwbwCm5KCv2G6Wj1/QNdy4Swyr/2h CNVgwx+B1B+J+wnp713XnGCTNI/pqZ9YFtWRyJ9q704UPb3auzo6apmI54wGM52sdgNG JQ8UTEhweFYfFJwz/MoLjvad4ah4cy8O82zoDSw2ai5xfaq6H0a+Np/2oRexqBFfxM9s KI3LPCvHn5GFWGuh88iMSUtYbe1K97imPlhE5J3gGpkV1x5pBXYiBMMu7yYq7qUqcgCB qZqg== X-Gm-Message-State: AHPjjUikkCBxPS2urD+U3YL3a9ZIv4sFn5Ds1rUnqVcuYxuml5FJrwXS qnQrSD7iA2qYp32KcHE= X-Received: by 10.13.209.6 with SMTP id t6mr13754245ywd.60.1505256835583; Tue, 12 Sep 2017 15:53:55 -0700 (PDT) Received: from mail-io0-f179.google.com (mail-io0-f179.google.com. [209.85.223.179]) by smtp.gmail.com with ESMTPSA id g72sm4331652ywe.26.2017.09.12.15.53.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Sep 2017 15:53:55 -0700 (PDT) Received: by mail-io0-f179.google.com with SMTP id v36so43241345ioi.1; Tue, 12 Sep 2017 15:53:55 -0700 (PDT) X-Google-Smtp-Source: ADKCNb4oA9KkamK0dy8LOKyjyZ2m2lGbwO7B/yMN6gyx9Xlgjd/UrtKl2QjJAg3TdA74LlzyP5srOIYo4oas2rw/GMs= X-Received: by 10.107.11.102 with SMTP id v99mr12980164ioi.260.1505256834858; Tue, 12 Sep 2017 15:53:54 -0700 (PDT) MIME-Version: 1.0 Reply-To: cem@freebsd.org Received: by 10.2.81.131 with HTTP; Tue, 12 Sep 2017 15:53:54 -0700 (PDT) In-Reply-To: <041393BA-94E3-4D22-95C8-EF0634746DC3@gmail.com> References: <0154558d-b2ad-af97-3960-3e392678f709@freebsd.org> <8d04540b-6daf-aa13-5648-0ec2541cbae6@freebsd.org> <041393BA-94E3-4D22-95C8-EF0634746DC3@gmail.com> From: Conrad Meyer Date: Tue, 12 Sep 2017 15:53:54 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: extending the maximum filename length (pointer to patch)[request for input] To: Ben RUBSON Cc: "freebsd-fs@freebsd.org" , freebsd-current , Julian Elischer Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Sep 2017 22:54:03 -0000 On Tue, Sep 12, 2017 at 2:11 PM, Ben RUBSON wrote: > Hi Conrad, > > This patch makes me think about another related bug #184340 : > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=184340 > It is about PATH_MAX which in some cases can be too small. > > Not sure if it's the case / and how to do it, > but perhaps it is time to raise some other limits / > think about a global solution regarding these length limits ? > > Many thanks ! Hi Ben, Yes, I think that bug is about basically the same issue under discussion here. Best, Conrad From owner-freebsd-fs@freebsd.org Tue Sep 12 22:57:02 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 773A0E23591; Tue, 12 Sep 2017 22:57:02 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: from mail-yw0-f181.google.com (mail-yw0-f181.google.com [209.85.161.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 392B467127; Tue, 12 Sep 2017 22:57:01 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: by mail-yw0-f181.google.com with SMTP id s62so31421052ywg.0; Tue, 12 Sep 2017 15:57:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc:content-transfer-encoding; bh=tZJ8/HkoP5ODUqO0rqMDQqW115BBMDlK7JFNItlqolg=; b=Drbm0ny0gnWx7RrfCF2yAw3/gbnbgsqw4UnAgVavkWSf0OM03Kvm9sJQxsts//H1+U XaCnuamUFzmFnfXR0GlzTE8s3Ga4xBM5WSUYwYdL/FvNQM9iIxEvtWG6kOiyfyc2dEKM azbU6ixVqyC4z3Z8opYURy4Vi9lROrLEfOY3tmfzJREZ71ts6ZtMVM2KFYjITWvPNcsC Y8pUv0M8JElkUTkKavnnYeGJJ73bSGXpjLZjHGbc2Q++Vz+j54sfae9rnkmkjp1JYwa7 oAJQSx0krnt1RGmqlk00OV9sfbubQlVuAPkmJcRLo3yuIsrYEuNG5ZkdteNec9U/GcTb UASg== X-Gm-Message-State: AHPjjUgTkzckH+1V1wOPkmHKHKjocXPb7U3fREvg740FLHCK6LJ91HtY 4F0pmeWiFdL4bfWXLZo= X-Received: by 10.37.188.129 with SMTP id e1mr13764327ybk.139.1505257014764; Tue, 12 Sep 2017 15:56:54 -0700 (PDT) Received: from mail-it0-f49.google.com (mail-it0-f49.google.com. [209.85.214.49]) by smtp.gmail.com with ESMTPSA id g10sm4286361ywl.54.2017.09.12.15.56.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Sep 2017 15:56:54 -0700 (PDT) Received: by mail-it0-f49.google.com with SMTP id o200so2638664itg.0; Tue, 12 Sep 2017 15:56:54 -0700 (PDT) X-Google-Smtp-Source: AOwi7QA5bM/7EXE2oaVyGbyiywncFoN7wFmjhhQDsoE0YdVrZBRDR8UoPNnXm6NspzhWNUSWInP6GPlY5bcLJR8Xl8k= X-Received: by 10.36.110.4 with SMTP id w4mr2193089itc.38.1505257013637; Tue, 12 Sep 2017 15:56:53 -0700 (PDT) MIME-Version: 1.0 Reply-To: cem@freebsd.org Received: by 10.2.81.131 with HTTP; Tue, 12 Sep 2017 15:56:52 -0700 (PDT) In-Reply-To: References: <0154558d-b2ad-af97-3960-3e392678f709@freebsd.org> <8d04540b-6daf-aa13-5648-0ec2541cbae6@freebsd.org> <041393BA-94E3-4D22-95C8-EF0634746DC3@gmail.com> From: Conrad Meyer Date: Tue, 12 Sep 2017 15:56:52 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: extending the maximum filename length (pointer to patch)[request for input] To: Rainer Duffner Cc: Ben RUBSON , "cem@FreeBSD.org" , "freebsd-fs@freebsd.org" , freebsd-current Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Sep 2017 22:57:02 -0000 On Tue, Sep 12, 2017 at 2:23 PM, Rainer Duffner wr= ote: > And there=E2=80=99s also this bug: > > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D215067 > > "g_dev_taste: make_dev_p() failed on importing pool with snapshots with l= ong > names=E2=80=9C > > > > But maybe that has nothing to do with it. Hi Rainer, Yes, I think that one is mostly orthogonal to the filesystem path lengths. Still, it might also be nice to expand that one while we're there. ino64 was nice enough to bump MNAMELEN from 88 to 1024 and d_namlen to 16 bits, so we don't have to expand those. Best, Conrad From owner-freebsd-fs@freebsd.org Wed Sep 13 00:58:53 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 18C83E02880; Wed, 13 Sep 2017 00:58:53 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E2E6C6BC6C; Wed, 13 Sep 2017 00:58:52 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (106-69-250-203.dyn.iinet.net.au [106.69.250.203]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id v8D0wknK025227 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Tue, 12 Sep 2017 17:58:50 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: extending the maximum filename length (pointer to patch)[request for input] To: Jonathan Anderson Cc: cem@freebsd.org, freebsd-current , "freebsd-fs@freebsd.org" References: <0154558d-b2ad-af97-3960-3e392678f709@freebsd.org> <8d04540b-6daf-aa13-5648-0ec2541cbae6@freebsd.org> <3C985916-3641-45B7-A721-C4F89A699B70@FreeBSD.org> From: Julian Elischer Message-ID: <9c450074-c599-f968-61e4-23f0350b0466@freebsd.org> Date: Wed, 13 Sep 2017 08:58:40 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <3C985916-3641-45B7-A721-C4F89A699B70@FreeBSD.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Sep 2017 00:58:53 -0000 On 13/9/17 1:16 am, Jonathan Anderson wrote: > On 12 Sep 2017, at 14:38, Julian Elischer wrote: > >> “这是一个测试多字符文件名长度的文件目的是命名一个文件用中文或者日文或者韩文字符并且要求字符长度超过八十五个字符然后拷贝该文件到我们的共享文件夹看看是否能够拷贝该文件到我.txt” >> >> (I have no idea what that says but apparently it's a real filename >> from a windows machine that blew up when written via samba.) > > Google Translate says, amusingly: > "This is a test file for the length of the file name. The purpose is > to name a file in Chinese or Japanese or Korean characters and > require the character to be longer than 85 characters and then copy > the file into our shared folder to see if it can copy the file To > me" (.txt, I guess) > > No matter what number you choose for a path length, you're never > going to win against that specific user. :) ha! it was give as an example to show us the limit. Apparently however our company has difficulty selling in China due to this limit because Microsoft has a much larger limit and people use it. > > > Jon > -- > Jonathan Anderson > jonathan@FreeBSD.org > > From owner-freebsd-fs@freebsd.org Wed Sep 13 16:18:04 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 38F7EE03BB5 for ; Wed, 13 Sep 2017 16:18:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 271CC6C293 for ; Wed, 13 Sep 2017 16:18:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v8DGI3LY061275 for ; Wed, 13 Sep 2017 16:18:04 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 222288] g_bio leak after zfs ABD commit Date: Wed, 13 Sep 2017 16:18:04 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: avg@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status cc assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Sep 2017 16:18:04 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D222288 Andriy Gapon changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |Open CC| |freebsd-fs@FreeBSD.org Assignee|freebsd-bugs@FreeBSD.org |avg@FreeBSD.org --- Comment #1 from Andriy Gapon --- (In reply to Dan Nelson from comment #0) Thank you very much for the report! As soon as I read it I noticed that I have the same kind of issue. I dug through the biozone to examine the leaked bio-s and they all seem to have bio_cmd =3D 5 and bio_flags =3D 8. So, they seem to be BIO_FLUSH bio-s. T= heir zio-s must have been ZIO_TYPE_IOCTL. Now, those zio-s use ZIO_IOCTL_PIPELINE and it is defined as: #define ZIO_IOCTL_PIPELINE \ (ZIO_INTERLOCK_STAGES | \ ZIO_STAGE_VDEV_IO_START | \ ZIO_STAGE_VDEV_IO_ASSESS) So, ZIO_STAGE_VDEV_IO_START is in the pipeline, but ZIO_STAGE_VDEV_IO_DONE = is not as you have correctly theorised. The normal I/O pipelines always include ZIO_VDEV_IO_STAGES #define ZIO_VDEV_IO_STAGES \ (ZIO_STAGE_VDEV_IO_START | \ ZIO_STAGE_VDEV_IO_DONE | \ ZIO_STAGE_VDEV_IO_ASSESS) so the problem does not affect them. I will double-check why the ioctl pipeline omitted ZIO_STAGE_VDEV_IO_DONE a= nd will test adding that stage. --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Wed Sep 13 21:21:13 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 75B19E102C4 for ; Wed, 13 Sep 2017 21:21:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 63DDE770AE for ; Wed, 13 Sep 2017 21:21:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v8DLLCoj055624 for ; Wed, 13 Sep 2017 21:21:13 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 221064] zfs should not be able to shadow mount on root directory from userspace Date: Wed, 13 Sep 2017 21:21:13 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: emaste@freebsd.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Sep 2017 21:21:13 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221064 --- Comment #10 from Ed Maste --- (In reply to Poul-Henning Kamp from comment #5) I agree that using 'mount' to mount a new fs over an existing one is probab= ly intentional, but the issue with ZFS here is that the mount happens as a side effect of 'zpool import' without the user explicitly asking for this. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Sep 13 23:12:37 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6E844E14ADD for ; Wed, 13 Sep 2017 23:12:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5D1567EE18 for ; Wed, 13 Sep 2017 23:12:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v8DNCbJn062670 for ; Wed, 13 Sep 2017 23:12:37 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 221064] zfs should not be able to shadow mount on root directory from userspace Date: Wed, 13 Sep 2017 23:12:37 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: allanjude@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Sep 2017 23:12:37 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221064 --- Comment #11 from Allan Jude --- (In reply to Ed Maste from comment #10) Maybe we could create a new mount option, noshadow, and make 'zfs mount -a'= and maybe even regular 'mount -a' use it by default, but manual mounts would no= t. When noshadow is specified, only mount if the target mountpoint is an empty directory. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Sep 14 12:46:12 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DB6C6E15B99; Thu, 14 Sep 2017 12:46:12 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citapm.icyb.net.ua (citapm.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id DA38472BC7; Thu, 14 Sep 2017 12:46:11 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citapm.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id PAA18631; Thu, 14 Sep 2017 15:46:03 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1dsTWt-00007d-5E; Thu, 14 Sep 2017 15:46:03 +0300 Subject: Re: mount / unmount and mountcheckdirs() To: Kirk McKusick Cc: freebsd-arch@FreeBSD.org, freebsd-fs References: <201605220640.u4M6enEo017327@chez.mckusick.com> From: Andriy Gapon Message-ID: <134c7c6e-f4f1-ef38-cc50-0e56c27c9fb8@FreeBSD.org> Date: Thu, 14 Sep 2017 15:45:07 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <201605220640.u4M6enEo017327@chez.mckusick.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Sep 2017 12:46:13 -0000 On 22/05/2016 09:40, Kirk McKusick wrote: > I added the checkdirs functionality in the mount direction only > (I actually did it in 4.4BSD-Lite and it got swept in with commit > 22521). The reason is that when a directory that is not empty is > mounted on, the expectation is that the entries in that directory > should no longer be present; rather they should be replaced by the > entries in the newly mounted directory. Thus all processes sitting > in the mounted on directory should see the newly mounted directory > as if they had come to it using a lookup after the mount had been > done. If a process had proceeded through the mounted on directory > into one of its other entries, then they are left alone until such > time as they chdir back into the mount point directory through ".." > at which time they will be passed up to the mounted directory using > the same mechanism that would put them there if they traversed into > the mount point from above it in the tree. I believe this is the > correct behavior, is not a security threat, and should be left alone. I almost dropped a ball on this issue, but I am now picking it up again. At the moment I am moving forward with the dounmount change as it seems to be non-contentious and rather simple to do and test. Regarding the mount part, I am not sure that I completely agree with you. Even if mountcheckdirs() does not cause any problems in the mount path, I still fail to see its usefulness. Specifically, I still do not see any significant difference between the covered directory and any directory below it. So, if we leave the lower directories alone, while bother with the covered directory... The covered directory: - absolute paths work correctly - relative paths with enough ".." (one) can access the actual namespace - other relative paths operate on the shadowed sub-tree of the original filesystem The lower directories: - absolute paths work correctly - relative paths with enough ".." (> 1) can access the actual namespace - other relative paths operate on the shadowed sub-tree of the original filesystem The only difference I can think of is that the root of the mounted filesystem cannot be reached with just ".."-s from the covered directory. But is this difference of any significance? Mateusz also raised some interesting points. On the other hand, it seems that illumos and probably Solaris has interesting parallels to the FreeBSD behavior. It does not allow to mount over a directory that is a current directory for any process ("Device busy"), but does not object against processes in directories below the mount point. So, probably it's just I who misses something about that scenario :-) > I was not aware that the functionality had been added at unmount > time, and I do not believe that it should have been done. Normally > an unmount will not succeed if any vnodes are busy (for example, if > any directory in the filesystem is a current directory). The only > way that it can succeed in such a case is if a forcible unmount is > done. The forcible unmount will effectively do a revoke(2) on all > current directory vnodes in the unmounted filesystem. Further attempts > to access them will fail with "." not found errors. The only way to > get a valid current directory is to chdir to an absolute pathname. > Gratuitously fixing this if you happen to be in the former root of > the filesystem is wrong. And as you note can lead to unintensionally > giving an escape path from a prison. So I concur with your removing > this added functionality. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Thu Sep 14 12:59:37 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 62B6EE168E5; Thu, 14 Sep 2017 12:59:37 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citapm.icyb.net.ua (citapm.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 7A5AD73231; Thu, 14 Sep 2017 12:59:35 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citapm.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id PAA18650; Thu, 14 Sep 2017 15:59:34 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1dsTjy-00008D-4Q; Thu, 14 Sep 2017 15:59:34 +0300 Subject: Re: mount / unmount and mountcheckdirs() To: Mateusz Guzik Cc: freebsd-arch@FreeBSD.org, freebsd-fs References: <5c01bf62-b7b2-2e1d-bca5-859e6bf1f0e5@FreeBSD.org> <20160515165332.GA27836@dft-labs.eu> From: Andriy Gapon Message-ID: <69d80f6d-da25-7b94-7774-564cecd0a8c2@FreeBSD.org> Date: Thu, 14 Sep 2017 15:58:13 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <20160515165332.GA27836@dft-labs.eu> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Sep 2017 12:59:37 -0000 On 15/05/2016 19:53, Mateusz Guzik wrote: > On Sun, May 15, 2016 at 04:37:05PM +0300, Andriy Gapon wrote: >> >> I am curious about the purpose of mountcheckdirs() called when mounting and >> unmounting a filesystem. >> >> The function is described as such: >> /* >> * Scan all active processes and prisons to see if any of them have a current >> * or root directory of `olddp'. If so, replace them with the new mount point. >> */ >> and it seems to be used to "lift" processes and jails to a root of a new >> filesystem when it is mounted and to "lower" them onto a covered vnode (if any) >> when a filesystem is unmounted. >> >> What's the purpose of those actions? >> It's strange that the machinations are done at all, but it is stranger that they >> are applied only to processes and jails at exactly a covered vnode and a root >> vnode. Anything below in a filesystem's tree is left alone. Is there anything >> so very special about being at exactly those points? >> >> IMO, the machinations can have unexpected security consequences. >> > > I don't know why this was implemented. It is also being done in NetBSD. > It is not done in Solaris nor Linux. I've done some testing on illumos and while it indeed does not do any "lifting", it does not permit to mount over a directory which is a current directory for any process. I haven't tested with zones, but suspect that it would also be prohibited. > Replacement is buggy in at least 2 ways: > 1. the process vs jail vnode replacement leaves a time window where > these 2 don't match, which screws up with the look up > 2. on fork we can have a 'struct filedesc' object copied but not > assigned to the new process yet, so it ends up with the old vnode > > And indeed, interested parties still have access to old vnodes by means > of having a file descriptor. Yeah. > That said, this likely needs to be simply changed to /deny/ mount > operations which would alter jail roots. That's probably what we should do. Or rather, Someone Should Do It. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Thu Sep 14 14:22:54 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D8199E1B550 for ; Thu, 14 Sep 2017 14:22:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C68DE75E7E for ; Thu, 14 Sep 2017 14:22:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v8EEMqbK057726 for ; Thu, 14 Sep 2017 14:22:54 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219049] ftruncate() slow on UFS+journaled softupdates Date: Thu, 14 Sep 2017 14:22:53 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: njt@ayvali.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Sep 2017 14:22:55 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219049 N. J. Thomas changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |njt@ayvali.org --- Comment #4 from N. J. Thomas --- I'm having the same issue with a box I upgraded from 10.3 to 11.1 (via 11.0) on a UFS filesystem. I'm pretty sure extracting packages wasn't this slow earlier. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Sep 15 03:11:00 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E2B7FE20030; Fri, 15 Sep 2017 03:11:00 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [70.36.157.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B2D4A74F22; Fri, 15 Sep 2017 03:11:00 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (localhost [IPv6:::1]) by chez.mckusick.com (8.15.2/8.15.2) with ESMTP id v8F3Ea6B085072; Thu, 14 Sep 2017 20:14:36 -0700 (PDT) (envelope-from mckusick@mckusick.com) Message-Id: <201709150314.v8F3Ea6B085072@chez.mckusick.com> From: Kirk McKusick To: Andriy Gapon Subject: Re: mount / unmount and mountcheckdirs() Cc: Konstantin Belousov , freebsd-arch@FreeBSD.org, freebsd-fs In-reply-to: <134c7c6e-f4f1-ef38-cc50-0e56c27c9fb8@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <85070.1505445276.1@chez.mckusick.com> Content-Transfer-Encoding: quoted-printable Date: Thu, 14 Sep 2017 20:14:36 -0700 X-Spam-Status: No, score=0.1 required=5.0 tests=MISSING_MID, UNPARSEABLE_RELAY autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on chez.mckusick.com X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Sep 2017 03:11:01 -0000 > To: Kirk McKusick > Subject: Re: mount / unmount and mountcheckdirs() > Cc: freebsd-arch@FreeBSD.org, freebsd-fs > From: Andriy Gapon > Date: Thu, 14 Sep 2017 15:45:07 +0300 > = > On 22/05/2016 09:40, Kirk McKusick wrote: >> I added the checkdirs functionality in the mount direction only >> (I actually did it in 4.4BSD-Lite and it got swept in with commit >> 22521). The reason is that when a directory that is not empty is >> mounted on, the expectation is that the entries in that directory >> should no longer be present; rather they should be replaced by the >> entries in the newly mounted directory. Thus all processes sitting >> in the mounted on directory should see the newly mounted directory >> as if they had come to it using a lookup after the mount had been >> done. If a process had proceeded through the mounted on directory >> into one of its other entries, then they are left alone until such >> time as they chdir back into the mount point directory through ".." >> at which time they will be passed up to the mounted directory using >> the same mechanism that would put them there if they traversed into >> the mount point from above it in the tree. I believe this is the >> correct behavior, is not a security threat, and should be left alone. > = > I almost dropped a ball on this issue, but I am now picking it up again. > At the moment I am moving forward with the dounmount change as it seems = to be > non-contentious and rather simple to do and test. > = > Regarding the mount part, I am not sure that I completely agree with you= . > Even if mountcheckdirs() does not cause any problems in the mount path, = I still > fail to see its usefulness. Specifically, I still do not see any signif= icant > difference between the covered directory and any directory below it. So= , if we > leave the lower directories alone, while bother with the covered directo= ry... > = > The covered directory: > - absolute paths work correctly > - relative paths with enough ".." (one) can access the actual namespace > - other relative paths operate on the shadowed sub-tree of the original > filesystem > = > = > The lower directories: > - absolute paths work correctly > - relative paths with enough ".." (> 1) can access the actual namespace > - other relative paths operate on the shadowed sub-tree of the original > filesystem > = > The only difference I can think of is that the root of the mounted files= ystem > cannot be reached with just ".."-s from the covered directory. But is t= his > difference of any significance? > = > Mateusz also raised some interesting points. > = > On the other hand, it seems that illumos and probably Solaris has > interesting parallels to the FreeBSD behavior. It does not allow > to mount over a directory that is a current directory for any process > ("Device busy"), but does not object against processes in directories > below the mount point. > = > So, probably it's just I who misses something about that scenario :-) > = >> I was not aware that the functionality had been added at unmount >> time, and I do not believe that it should have been done. Normally >> an unmount will not succeed if any vnodes are busy (for example, if >> any directory in the filesystem is a current directory). The only >> way that it can succeed in such a case is if a forcible unmount is >> done. The forcible unmount will effectively do a revoke(2) on all >> current directory vnodes in the unmounted filesystem. Further attempts >> to access them will fail with "." not found errors. The only way to >> get a valid current directory is to chdir to an absolute pathname. >> Gratuitously fixing this if you happen to be in the former root of >> the filesystem is wrong. And as you note can lead to unintensionally >> giving an escape path from a prison. So I concur with your removing >> this added functionality. > -- = > Andriy Gapon I had to dig back through some *really* old emails to find out what triggered the addition of mountcheckdirs(). The problem that it was specifically solving was that as part of the startup script a minimal root directory was replaced by the real root directory. The shell running the startup script needed to be moved to the new mounted-on root so that the rest of the script would not fail. That disaster of a hack has been replaced with the much more functional code that deals with setting up the root and the devfs filesystem on /dev. So the need for which it was designed no longer exists. But I still believe that it is the correct thing to do. For example, if you are using automount code and chdir into your home directory triggering an auto-mount, you should just be in your home directory after the mount rather than having to do cd ../$USER to get there. ~Kirk From owner-freebsd-fs@freebsd.org Fri Sep 15 09:20:13 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 29CB1E092B9; Fri, 15 Sep 2017 09:20:13 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C2FE58378C; Fri, 15 Sep 2017 09:20:12 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id v8F9K1Ai048801 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 15 Sep 2017 12:20:01 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua v8F9K1Ai048801 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id v8F9K1Ar048800; Fri, 15 Sep 2017 12:20:01 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 15 Sep 2017 12:20:01 +0300 From: Konstantin Belousov To: Kirk McKusick Cc: Andriy Gapon , freebsd-fs , freebsd-arch@FreeBSD.org Subject: Re: mount / unmount and mountcheckdirs() Message-ID: <20170915092001.GK78693@kib.kiev.ua> References: <134c7c6e-f4f1-ef38-cc50-0e56c27c9fb8@FreeBSD.org> <201709150314.v8F3Ea6B085072@chez.mckusick.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201709150314.v8F3Ea6B085072@chez.mckusick.com> User-Agent: Mutt/1.9.0 (2017-09-02) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Sep 2017 09:20:13 -0000 On Thu, Sep 14, 2017 at 08:14:36PM -0700, Kirk McKusick wrote: > > To: Kirk McKusick > > Subject: Re: mount / unmount and mountcheckdirs() > > Cc: freebsd-arch@FreeBSD.org, freebsd-fs > > From: Andriy Gapon > > Date: Thu, 14 Sep 2017 15:45:07 +0300 > > > > On 22/05/2016 09:40, Kirk McKusick wrote: > >> I added the checkdirs functionality in the mount direction only > >> (I actually did it in 4.4BSD-Lite and it got swept in with commit > >> 22521). The reason is that when a directory that is not empty is > >> mounted on, the expectation is that the entries in that directory > >> should no longer be present; rather they should be replaced by the > >> entries in the newly mounted directory. Thus all processes sitting > >> in the mounted on directory should see the newly mounted directory > >> as if they had come to it using a lookup after the mount had been > >> done. If a process had proceeded through the mounted on directory > >> into one of its other entries, then they are left alone until such > >> time as they chdir back into the mount point directory through ".." > >> at which time they will be passed up to the mounted directory using > >> the same mechanism that would put them there if they traversed into > >> the mount point from above it in the tree. I believe this is the > >> correct behavior, is not a security threat, and should be left alone. > > > > I almost dropped a ball on this issue, but I am now picking it up again. > > At the moment I am moving forward with the dounmount change as it seems to be > > non-contentious and rather simple to do and test. > > > > Regarding the mount part, I am not sure that I completely agree with you. > > Even if mountcheckdirs() does not cause any problems in the mount path, I still > > fail to see its usefulness. Specifically, I still do not see any significant > > difference between the covered directory and any directory below it. So, if we > > leave the lower directories alone, while bother with the covered directory... > > > > The covered directory: > > - absolute paths work correctly > > - relative paths with enough ".." (one) can access the actual namespace > > - other relative paths operate on the shadowed sub-tree of the original > > filesystem > > > > > > The lower directories: > > - absolute paths work correctly > > - relative paths with enough ".." (> 1) can access the actual namespace > > - other relative paths operate on the shadowed sub-tree of the original > > filesystem > > > > The only difference I can think of is that the root of the mounted filesystem > > cannot be reached with just ".."-s from the covered directory. But is this > > difference of any significance? > > > > Mateusz also raised some interesting points. > > > > On the other hand, it seems that illumos and probably Solaris has > > interesting parallels to the FreeBSD behavior. It does not allow > > to mount over a directory that is a current directory for any process > > ("Device busy"), but does not object against processes in directories > > below the mount point. > > > > So, probably it's just I who misses something about that scenario :-) > > > >> I was not aware that the functionality had been added at unmount > >> time, and I do not believe that it should have been done. Normally > >> an unmount will not succeed if any vnodes are busy (for example, if > >> any directory in the filesystem is a current directory). The only > >> way that it can succeed in such a case is if a forcible unmount is > >> done. The forcible unmount will effectively do a revoke(2) on all > >> current directory vnodes in the unmounted filesystem. Further attempts > >> to access them will fail with "." not found errors. The only way to > >> get a valid current directory is to chdir to an absolute pathname. > >> Gratuitously fixing this if you happen to be in the former root of > >> the filesystem is wrong. And as you note can lead to unintensionally > >> giving an escape path from a prison. So I concur with your removing > >> this added functionality. > > -- > > Andriy Gapon > > I had to dig back through some *really* old emails to find out what > triggered the addition of mountcheckdirs(). The problem that it was > specifically solving was that as part of the startup script a minimal > root directory was replaced by the real root directory. The shell > running the startup script needed to be moved to the new mounted-on > root so that the rest of the script would not fail. If the mountcheckdirs() code not going away, please add your spelunking results as a comment before the function. This theme is recurring, and it would be highly beneficial to not loose the non-trivial reasoning behind the code existence. > > That disaster of a hack has been replaced with the much more functional > code that deals with setting up the root and the devfs filesystem on > /dev. So the need for which it was designed no longer exists. But I > still believe that it is the correct thing to do. For example, if you > are using automount code and chdir into your home directory triggering > an auto-mount, you should just be in your home directory after the > mount rather than having to do cd ../$USER to get there. I believe that the current autofs does not allow a process to get into this situation at all. In fact, the behavior implemented by mountcheckdirs() is surprising as well. For instance, I did expected that the system would operate as if mountcheckdirs() does not exist, and it caused me some head-scratching when I see it first time. From owner-freebsd-fs@freebsd.org Fri Sep 15 10:08:32 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ECB19E0B8A8; Fri, 15 Sep 2017 10:08:32 +0000 (UTC) (envelope-from etnapierala@gmail.com) Received: from mail-ua0-x22e.google.com (mail-ua0-x22e.google.com [IPv6:2607:f8b0:400c:c08::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A1CB613CB; Fri, 15 Sep 2017 10:08:32 +0000 (UTC) (envelope-from etnapierala@gmail.com) Received: by mail-ua0-x22e.google.com with SMTP id k63so989334uak.12; Fri, 15 Sep 2017 03:08:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=WvemsmAcbVVWcb6anF3UZZ+QeMxdOBfA88FTxcKo+ws=; b=VZWi7kTq2IAgUQE+gA0YYcj9cdAMsVCfx1cDT1J8zY8k3U6U3Ja43x4dQADan18dE7 c2Tfl9wUktOd1EbBgArhs1b7jsI2F5PUu61chWeBzMxgu5Qju4/7sZptGmlKJS+eXoJ3 lLQ3wJ1IDvVNQhES1oA0NctUW+wyC0IRCxhinpfQMlcylg4kQJUI64S07UIoQcEGk+e1 07SkwVvhJ6qhPw3oPe9pRdFYvuTl4exFJ5mIe/LSsW8tgXoqkFcrqX4Uu++JVh03PkiX AM0paIIQ5et3rAMav01kUNEgkkiRF5qQgcSDV0oPvm1UrRVpWBF1GmaMtqaWoUfyqmzw QOcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=WvemsmAcbVVWcb6anF3UZZ+QeMxdOBfA88FTxcKo+ws=; b=kOSaB8SMIZXmB6iz4d0pCoJHwzuFFRpa54LJILC96AvRjr90PdHwjIO0hqZXZm4zTQ Mz1mOBG2RVj3HyYJr4NkdgINqkpacatNxsMG7czaeiswE3ujdtDU2E0s8YbQaCAMSH+2 +CMhdTNWoRGO7LkyNiDQ6D2d/Uo8ojPYl4VsSOONo2EVzMAq+qdFG+XDOwATByQJD5b0 BDni/6fU9DA6zNEDVpkh76/vKmtK1cPvx/tRteqNvef5ZHyeOFWAGOlw+ZsmkYB4YOBm 01aaWv6I3dRRvxJ1CCJwelrz1J+4Xeyv29eNWMQIqWPEfrsbA3iLrd1AZNAIHOVTjgOr lCfw== X-Gm-Message-State: AHPjjUhyxv4vqZQrdVzzBepZ+KLzfhr9mS8sySZKYcG6P5EvOTwWDlRd WhZFeTIB8rrKPPww8p4O8fEz4mZUKi1bp3pe278= X-Google-Smtp-Source: ADKCNb5R0j+c+YzO6Bucq8sonl/3eRgkdBN5qOndAXN9iO1TjCIxwv02Qlx/4fY9BcOqHCn/6mX3lR7JqVL1Kyeh2E0= X-Received: by 10.176.21.109 with SMTP id p42mr12368078uae.127.1505470111627; Fri, 15 Sep 2017 03:08:31 -0700 (PDT) MIME-Version: 1.0 Sender: etnapierala@gmail.com Received: by 10.159.51.232 with HTTP; Fri, 15 Sep 2017 03:08:31 -0700 (PDT) In-Reply-To: <20170915092001.GK78693@kib.kiev.ua> References: <134c7c6e-f4f1-ef38-cc50-0e56c27c9fb8@FreeBSD.org> <201709150314.v8F3Ea6B085072@chez.mckusick.com> <20170915092001.GK78693@kib.kiev.ua> From: Edward Napierala Date: Fri, 15 Sep 2017 11:08:31 +0100 X-Google-Sender-Auth: q1_4RPx-uknuYeYqYQVra7HLIj8 Message-ID: Subject: Re: mount / unmount and mountcheckdirs() To: Konstantin Belousov Cc: Kirk McKusick , freebsd-fs , Andriy Gapon , "freebsd-arch@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Sep 2017 10:08:33 -0000 2017-09-15 10:20 GMT+01:00 Konstantin Belousov : > On Thu, Sep 14, 2017 at 08:14:36PM -0700, Kirk McKusick wrote: > > > To: Kirk McKusick > > > Subject: Re: mount / unmount and mountcheckdirs() > > > Cc: freebsd-arch@FreeBSD.org, freebsd-fs > > > From: Andriy Gapon > > > Date: Thu, 14 Sep 2017 15:45:07 +0300 > > > > > > On 22/05/2016 09:40, Kirk McKusick wrote: > > >> I added the checkdirs functionality in the mount direction only > > >> (I actually did it in 4.4BSD-Lite and it got swept in with commit > > >> 22521). The reason is that when a directory that is not empty is > > >> mounted on, the expectation is that the entries in that directory > > >> should no longer be present; rather they should be replaced by the > > >> entries in the newly mounted directory. Thus all processes sitting > > >> in the mounted on directory should see the newly mounted directory > > >> as if they had come to it using a lookup after the mount had been > > >> done. If a process had proceeded through the mounted on directory > > >> into one of its other entries, then they are left alone until such > > >> time as they chdir back into the mount point directory through ".." > > >> at which time they will be passed up to the mounted directory using > > >> the same mechanism that would put them there if they traversed into > > >> the mount point from above it in the tree. I believe this is the > > >> correct behavior, is not a security threat, and should be left alone. > > > > > > I almost dropped a ball on this issue, but I am now picking it up > again. > > > At the moment I am moving forward with the dounmount change as it > seems to be > > > non-contentious and rather simple to do and test. > > > > > > Regarding the mount part, I am not sure that I completely agree with > you. > > > Even if mountcheckdirs() does not cause any problems in the mount > path, I still > > > fail to see its usefulness. Specifically, I still do not see any > significant > > > difference between the covered directory and any directory below it. > So, if we > > > leave the lower directories alone, while bother with the covered > directory... > > > > > > The covered directory: > > > - absolute paths work correctly > > > - relative paths with enough ".." (one) can access the actual namespace > > > - other relative paths operate on the shadowed sub-tree of the original > > > filesystem > > > > > > > > > The lower directories: > > > - absolute paths work correctly > > > - relative paths with enough ".." (> 1) can access the actual namespace > > > - other relative paths operate on the shadowed sub-tree of the original > > > filesystem > > > > > > The only difference I can think of is that the root of the mounted > filesystem > > > cannot be reached with just ".."-s from the covered directory. But is > this > > > difference of any significance? > > > > > > Mateusz also raised some interesting points. > > > > > > On the other hand, it seems that illumos and probably Solaris has > > > interesting parallels to the FreeBSD behavior. It does not allow > > > to mount over a directory that is a current directory for any process > > > ("Device busy"), but does not object against processes in directories > > > below the mount point. > > > > > > So, probably it's just I who misses something about that scenario :-) > > > > > >> I was not aware that the functionality had been added at unmount > > >> time, and I do not believe that it should have been done. Normally > > >> an unmount will not succeed if any vnodes are busy (for example, if > > >> any directory in the filesystem is a current directory). The only > > >> way that it can succeed in such a case is if a forcible unmount is > > >> done. The forcible unmount will effectively do a revoke(2) on all > > >> current directory vnodes in the unmounted filesystem. Further attempts > > >> to access them will fail with "." not found errors. The only way to > > >> get a valid current directory is to chdir to an absolute pathname. > > >> Gratuitously fixing this if you happen to be in the former root of > > >> the filesystem is wrong. And as you note can lead to unintensionally > > >> giving an escape path from a prison. So I concur with your removing > > >> this added functionality. > > > -- > > > Andriy Gapon > > > > I had to dig back through some *really* old emails to find out what > > triggered the addition of mountcheckdirs(). The problem that it was > > specifically solving was that as part of the startup script a minimal > > root directory was replaced by the real root directory. The shell > > running the startup script needed to be moved to the new mounted-on > > root so that the rest of the script would not fail. > If the mountcheckdirs() code not going away, please add your spelunking > results as a comment before the function. This theme is recurring, and > it would be highly beneficial to not loose the non-trivial reasoning > behind the code existence. > > > > > That disaster of a hack has been replaced with the much more functional > > code that deals with setting up the root and the devfs filesystem on > > /dev. So the need for which it was designed no longer exists. But I > > still believe that it is the correct thing to do. For example, if you > > are using automount code and chdir into your home directory triggering > > an auto-mount, you should just be in your home directory after the > > mount rather than having to do cd ../$USER to get there. > I believe that the current autofs does not allow a process to get into > this situation at all. > It does. For example: [trasz@v2:~]% cd /media/md0 [trasz@v2:/media/md0]% mount /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) devfs on /dev (devfs, local, multilabel) map -hosts on /net (autofs) map -media on /media (autofs) [trasz@v2:/media/md0]% ls [trasz@v2:/media/md0]% mount /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) devfs on /dev (devfs, local, multilabel) map -hosts on /net (autofs) map -media on /media (autofs) /dev/md0 on /media/md0 (ufs, local, noatime, nosuid, automounted) Getting rid of mountcheckdirs() in the unmount path should be fine, I think. From owner-freebsd-fs@freebsd.org Fri Sep 15 10:30:43 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7D9DBE0C9CC; Fri, 15 Sep 2017 10:30:43 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 125A12012; Fri, 15 Sep 2017 10:30:42 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id v8FAUb7f065316 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 15 Sep 2017 13:30:37 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua v8FAUb7f065316 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id v8FAUbZt065314; Fri, 15 Sep 2017 13:30:37 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 15 Sep 2017 13:30:37 +0300 From: Konstantin Belousov To: Edward Napierala Cc: Kirk McKusick , freebsd-fs , Andriy Gapon , "freebsd-arch@freebsd.org" Subject: Re: mount / unmount and mountcheckdirs() Message-ID: <20170915103037.GM78693@kib.kiev.ua> References: <134c7c6e-f4f1-ef38-cc50-0e56c27c9fb8@FreeBSD.org> <201709150314.v8F3Ea6B085072@chez.mckusick.com> <20170915092001.GK78693@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.0 (2017-09-02) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Sep 2017 10:30:43 -0000 On Fri, Sep 15, 2017 at 11:08:31AM +0100, Edward Napierala wrote: > 2017-09-15 10:20 GMT+01:00 Konstantin Belousov : > > I believe that the current autofs does not allow a process to get into > > this situation at all. > > > > It does. For example: > > [trasz@v2:~]% cd /media/md0 > [trasz@v2:/media/md0]% mount > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > devfs on /dev (devfs, local, multilabel) > map -hosts on /net (autofs) > map -media on /media (autofs) > [trasz@v2:/media/md0]% ls > [trasz@v2:/media/md0]% mount > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > devfs on /dev (devfs, local, multilabel) > map -hosts on /net (autofs) > map -media on /media (autofs) > /dev/md0 on /media/md0 (ufs, local, noatime, nosuid, automounted) > > Getting rid of mountcheckdirs() in the unmount path should be fine, I think. How the example proves that mountcheckdirs() can be removed ? How can we see which directory content was printed by ls, the covered or mounted ? From owner-freebsd-fs@freebsd.org Fri Sep 15 10:49:19 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1409EE0D7EE; Fri, 15 Sep 2017 10:49:19 +0000 (UTC) (envelope-from etnapierala@gmail.com) Received: from mail-ua0-x22a.google.com (mail-ua0-x22a.google.com [IPv6:2607:f8b0:400c:c08::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BCCC029EE; Fri, 15 Sep 2017 10:49:18 +0000 (UTC) (envelope-from etnapierala@gmail.com) Received: by mail-ua0-x22a.google.com with SMTP id t5so1033493uad.13; Fri, 15 Sep 2017 03:49:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=JvUxFc3iQf2tK20/Xg7cXWHxAE9QNx105L4hLj9hTJU=; b=tBeEV1zKSsrO7PT47ysShzYNIolFkAW0CfJylCSFmnEhDY6ZT9kgNiz2XR0l48c4Ts Bmv5hig88Eb9MiSxhSNF2xmtEsEH5AsoZmUsx5O6HdLMcOd9dp3nBDJmDY90mB9OG6TJ KbnTBQVHOGsT+bIYJR3zis1TH1y2gkgE6BAkjjJ1jKCywNr6YwUde/tpZKBld1Lw23bM PZWKNDWyGl+nc9uKKZJjkFuAplB27knxVh9cTJy6qr1Yfbw/7C8665qCheO6Cpw3Ymuf iy1RVYuvWYvo3L1KvvcpSmHEscbMhRubOmNAJc9HjgXDEboebCSHRwpYMSohMxTtu58M qMuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=JvUxFc3iQf2tK20/Xg7cXWHxAE9QNx105L4hLj9hTJU=; b=V2ONm50Kj2WHgWwlSXwq8YLS2nl6pEgpBfBCyNUUsgdN6F/v3OovJjV9UMf8K+xAty IAXSp15WeqvyDc80PuJda4AdzUVgaFbYFVHcwyjTaLu9svO/od7+hIHuNBHjItzRBEYn NsMTtBEZKAD8/eUrTRt8eBqp6V1FNUjdL8RTchSDQ8R9UazXmKZ/sMGN6srHm+/OEfys ZKncfxKtxRMjpZovSHmVFxQaeniKivzFeKMcQStL+frfaXk1sTXXRT8mQaD8xAEFFQFB 5lZylaB703Dp66/Zk+hsMh2UkluJIvpa+Cyc+JYd0e+46u4beXy/EG7k9nBCN9VZhts+ qcHw== X-Gm-Message-State: AHPjjUgqlD46efSK1UR/oACLLeCSz97JF8xanz+1SINtbdxZVce+58V/ JKyqxqJmNx0XtwOAcbzqTiv8ytXyAKhCOnB1fRw= X-Google-Smtp-Source: AOwi7QAKe5oLEoNiGKpgmJKxqY5j9fk7JoVcuwqj+WtI4dflqwHlboZ0ayKkTBSAuTTquJTl41gK506818mwW6KPt78= X-Received: by 10.176.83.123 with SMTP id y56mr21175614uay.172.1505472557839; Fri, 15 Sep 2017 03:49:17 -0700 (PDT) MIME-Version: 1.0 Sender: etnapierala@gmail.com Received: by 10.159.51.232 with HTTP; Fri, 15 Sep 2017 03:49:17 -0700 (PDT) In-Reply-To: <20170915103037.GM78693@kib.kiev.ua> References: <134c7c6e-f4f1-ef38-cc50-0e56c27c9fb8@FreeBSD.org> <201709150314.v8F3Ea6B085072@chez.mckusick.com> <20170915092001.GK78693@kib.kiev.ua> <20170915103037.GM78693@kib.kiev.ua> From: Edward Napierala Date: Fri, 15 Sep 2017 11:49:17 +0100 X-Google-Sender-Auth: wZ_JDot4INtpYyYVBEu9HHbFkw4 Message-ID: Subject: Re: mount / unmount and mountcheckdirs() To: Konstantin Belousov Cc: Kirk McKusick , freebsd-fs , Andriy Gapon , "freebsd-arch@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Sep 2017 10:49:19 -0000 2017-09-15 11:30 GMT+01:00 Konstantin Belousov : > On Fri, Sep 15, 2017 at 11:08:31AM +0100, Edward Napierala wrote: > > 2017-09-15 10:20 GMT+01:00 Konstantin Belousov : > > > I believe that the current autofs does not allow a process to get into > > > this situation at all. > > > > > > > It does. For example: > > > > [trasz@v2:~]% cd /media/md0 > > [trasz@v2:/media/md0]% mount > > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > > devfs on /dev (devfs, local, multilabel) > > map -hosts on /net (autofs) > > map -media on /media (autofs) > > [trasz@v2:/media/md0]% ls > > [trasz@v2:/media/md0]% mount > > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > > devfs on /dev (devfs, local, multilabel) > > map -hosts on /net (autofs) > > map -media on /media (autofs) > > /dev/md0 on /media/md0 (ufs, local, noatime, nosuid, automounted) > > > > Getting rid of mountcheckdirs() in the unmount path should be fine, I > think. > How the example proves that mountcheckdirs() can be removed ? In the unmount path? It doesn't; I just don't think autofs would be affected, since having a current working directory in a mountpoint prevents the unmount (unless it's forced). In the mount path? See below. > How can > we see which directory content was printed by ls, the covered or mounted ? > Well, it would be pretty useless if you got the covered directory. But ok, here's a better example, one that actually shows that: [trasz@v2:/media/md0]% mount /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) devfs on /dev (devfs, local, multilabel) map -hosts on /net (autofs) map -media on /media (autofs) [trasz@v2:/media/md0]% ls -al total 9 drwxr-xr-x 3 root wheel 512 Sep 10 10:16 . drwxr-xr-x 3 root wheel 512 Sep 8 11:42 .. drwxrwxr-x 2 root operator 512 Sep 10 10:16 .snap [trasz@v2:/media/md0]% mount /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) devfs on /dev (devfs, local, multilabel) map -hosts on /net (autofs) map -media on /media (autofs) /dev/md0 on /media/md0 (ufs, local, noatime, nosuid, automounted) Notice the .snap/ directory; it's an empty UFS filesystem. From owner-freebsd-fs@freebsd.org Fri Sep 15 11:00:38 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E806BE0E2AD; Fri, 15 Sep 2017 11:00:38 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8EB9E3131; Fri, 15 Sep 2017 11:00:38 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id v8FB0XC3072038 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 15 Sep 2017 14:00:34 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua v8FB0XC3072038 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id v8FB0XmQ072037; Fri, 15 Sep 2017 14:00:33 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 15 Sep 2017 14:00:33 +0300 From: Konstantin Belousov To: Edward Napierala Cc: Kirk McKusick , freebsd-fs , Andriy Gapon , "freebsd-arch@freebsd.org" Subject: Re: mount / unmount and mountcheckdirs() Message-ID: <20170915110033.GO78693@kib.kiev.ua> References: <134c7c6e-f4f1-ef38-cc50-0e56c27c9fb8@FreeBSD.org> <201709150314.v8F3Ea6B085072@chez.mckusick.com> <20170915092001.GK78693@kib.kiev.ua> <20170915103037.GM78693@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.0 (2017-09-02) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Sep 2017 11:00:39 -0000 On Fri, Sep 15, 2017 at 11:49:17AM +0100, Edward Napierala wrote: > 2017-09-15 11:30 GMT+01:00 Konstantin Belousov : > > > On Fri, Sep 15, 2017 at 11:08:31AM +0100, Edward Napierala wrote: > > > 2017-09-15 10:20 GMT+01:00 Konstantin Belousov : > > > > I believe that the current autofs does not allow a process to get into > > > > this situation at all. > > > > > > > > > > It does. For example: > > > > > > [trasz@v2:~]% cd /media/md0 > > > [trasz@v2:/media/md0]% mount > > > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > > > devfs on /dev (devfs, local, multilabel) > > > map -hosts on /net (autofs) > > > map -media on /media (autofs) > > > [trasz@v2:/media/md0]% ls > > > [trasz@v2:/media/md0]% mount > > > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > > > devfs on /dev (devfs, local, multilabel) > > > map -hosts on /net (autofs) > > > map -media on /media (autofs) > > > /dev/md0 on /media/md0 (ufs, local, noatime, nosuid, automounted) > > > > > > Getting rid of mountcheckdirs() in the unmount path should be fine, I > > think. > > How the example proves that mountcheckdirs() can be removed ? > > > In the unmount path? It doesn't; I just don't think autofs would be > affected, > since having a current working directory in a mountpoint prevents the > unmount > (unless it's forced). > > In the mount path? See below. > > > > How can > > we see which directory content was printed by ls, the covered or mounted ? > > > > Well, it would be pretty useless if you got the covered directory. This is exactly what is being discussed. > But ok, here's > a better example, one that actually shows that: > > [trasz@v2:/media/md0]% mount > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > devfs on /dev (devfs, local, multilabel) > map -hosts on /net (autofs) > map -media on /media (autofs) > [trasz@v2:/media/md0]% ls -al > total 9 > drwxr-xr-x 3 root wheel 512 Sep 10 10:16 . > drwxr-xr-x 3 root wheel 512 Sep 8 11:42 .. > drwxrwxr-x 2 root operator 512 Sep 10 10:16 .snap > [trasz@v2:/media/md0]% mount > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > devfs on /dev (devfs, local, multilabel) > map -hosts on /net (autofs) > map -media on /media (autofs) > /dev/md0 on /media/md0 (ufs, local, noatime, nosuid, automounted) > > Notice the .snap/ directory; it's an empty UFS filesystem. And the .snap is from the /dev/md0 volume, right ? If yes, then this demonstration indeed proves that autofs behaves as intended on mount. From owner-freebsd-fs@freebsd.org Fri Sep 15 11:31:07 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 77BF2E0F8FB; Fri, 15 Sep 2017 11:31:07 +0000 (UTC) (envelope-from etnapierala@gmail.com) Received: from mail-ua0-x229.google.com (mail-ua0-x229.google.com [IPv6:2607:f8b0:400c:c08::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 299D33F35; Fri, 15 Sep 2017 11:31:07 +0000 (UTC) (envelope-from etnapierala@gmail.com) Received: by mail-ua0-x229.google.com with SMTP id 72so1070216uas.8; Fri, 15 Sep 2017 04:31:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=BCI+uCZ99PDTwjj4fnECwhYjMbmHlI4kV7UO85WAj8k=; b=U0vg7NkxyyqeGXIDr/C6/jmM6AJ5gVozWuK9AgDgydTm6oHMOZiKNM4eVkfPnqtw96 ahfiOjiFXo+n35bbiJufPvHGgCO35W/el8DTt7syEWMmmhuYgc1A9KA8iBXnk+PcY0wH lqc6z8XdwC/Yx5DuM8GP8DHUqejEyExjexGrW+8J6Lwblkv59vT4O/XsJrS8SCEzpQNt QaPLHpbPWBYq8rMr/cnS0M74x1cyiLoewU6DLrFC3C+kZ+ooU2KKK9VRaw7AhJ/CyYQx UUnlxAMGeta+eczBC6nx+WjFVl1AcGIAlaStYXQnBdIo01i9jPTmIzngkRpozi5koJG0 F0eQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=BCI+uCZ99PDTwjj4fnECwhYjMbmHlI4kV7UO85WAj8k=; b=hL+wHv79DnMU3zQFG9+BJALOlB9x/m1CAJkYow0jz8HUh6c74eZuJfR+XADos/YhiT 97CwOKNgjkHrqURfNGOQ+MKU+4hlMHyA+yLv/+ESrZsNPazVkZagW6/OqJHxakjOvR9t /I+5O9Dycog+MRZ59IUN7BEEoVUYdEa4m2LTvfIbeJlufdjtl+9EmczU56s7eE/rm1UR /IR6uXSZmmMCGMeEtsGN7m1adCmUkAQ2WFd0L7wENSMrStEDSomA2sxb19sc8QZp+5No TMaf/P7J8fnrCpt4U1ZRYiSSBqmOCyxhJa8qzXgyyjgXL/LJIRB4us/LgkK0trrAJllS IEJQ== X-Gm-Message-State: AHPjjUiM2BPESQUGSOPP6JSuHolU1n4MZSyMoJ1KZqsQGYMkF4YH/Xfn GckE8S7Z5Bpbp/w+WMYibtQ4g0y4n9H0t9hy1bQ= X-Google-Smtp-Source: ADKCNb5aTm0GnEozTU1lPnkNjCT3fdmbp/nT0ji7/w9OI6+wGuezGPLbtCkGeU02divGOZokZ3ZkYIurzFdL7sIU3W8= X-Received: by 10.176.92.111 with SMTP id a47mr21213593uag.111.1505475065951; Fri, 15 Sep 2017 04:31:05 -0700 (PDT) MIME-Version: 1.0 Sender: etnapierala@gmail.com Received: by 10.159.51.232 with HTTP; Fri, 15 Sep 2017 04:31:05 -0700 (PDT) In-Reply-To: <20170915110033.GO78693@kib.kiev.ua> References: <134c7c6e-f4f1-ef38-cc50-0e56c27c9fb8@FreeBSD.org> <201709150314.v8F3Ea6B085072@chez.mckusick.com> <20170915092001.GK78693@kib.kiev.ua> <20170915103037.GM78693@kib.kiev.ua> <20170915110033.GO78693@kib.kiev.ua> From: Edward Napierala Date: Fri, 15 Sep 2017 12:31:05 +0100 X-Google-Sender-Auth: MfR-gur5i6VTgT8RqaOUE7DyMKA Message-ID: Subject: Re: mount / unmount and mountcheckdirs() To: Konstantin Belousov Cc: Kirk McKusick , freebsd-fs , Andriy Gapon , "freebsd-arch@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Sep 2017 11:31:07 -0000 2017-09-15 12:00 GMT+01:00 Konstantin Belousov : > On Fri, Sep 15, 2017 at 11:49:17AM +0100, Edward Napierala wrote: > > 2017-09-15 11:30 GMT+01:00 Konstantin Belousov : > > > > > On Fri, Sep 15, 2017 at 11:08:31AM +0100, Edward Napierala wrote: > > > > 2017-09-15 10:20 GMT+01:00 Konstantin Belousov >: > > > > > I believe that the current autofs does not allow a process to get > into > > > > > this situation at all. > > > > > > > > > > > > > It does. For example: > > > > > > > > [trasz@v2:~]% cd /media/md0 > > > > [trasz@v2:/media/md0]% mount > > > > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > > > > devfs on /dev (devfs, local, multilabel) > > > > map -hosts on /net (autofs) > > > > map -media on /media (autofs) > > > > [trasz@v2:/media/md0]% ls > > > > [trasz@v2:/media/md0]% mount > > > > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > > > > devfs on /dev (devfs, local, multilabel) > > > > map -hosts on /net (autofs) > > > > map -media on /media (autofs) > > > > /dev/md0 on /media/md0 (ufs, local, noatime, nosuid, automounted) > > > > > > > > Getting rid of mountcheckdirs() in the unmount path should be fine, I > > > think. > > > How the example proves that mountcheckdirs() can be removed ? > > > > > > In the unmount path? It doesn't; I just don't think autofs would be > > affected, > > since having a current working directory in a mountpoint prevents the > > unmount > > (unless it's forced). > > > > In the mount path? See below. > > > > > > > How can > > > we see which directory content was printed by ls, the covered or > mounted ? > > > > > > > Well, it would be pretty useless if you got the covered directory. > This is exactly what is being discussed. > > > But ok, here's > > a better example, one that actually shows that: > > > > [trasz@v2:/media/md0]% mount > > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > > devfs on /dev (devfs, local, multilabel) > > map -hosts on /net (autofs) > > map -media on /media (autofs) > > [trasz@v2:/media/md0]% ls -al > > total 9 > > drwxr-xr-x 3 root wheel 512 Sep 10 10:16 . > > drwxr-xr-x 3 root wheel 512 Sep 8 11:42 .. > > drwxrwxr-x 2 root operator 512 Sep 10 10:16 .snap > > [trasz@v2:/media/md0]% mount > > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > > devfs on /dev (devfs, local, multilabel) > > map -hosts on /net (autofs) > > map -media on /media (autofs) > > /dev/md0 on /media/md0 (ufs, local, noatime, nosuid, automounted) > > > > Notice the .snap/ directory; it's an empty UFS filesystem. > And the .snap is from the /dev/md0 volume, right ? > Yes. > If yes, then this demonstration indeed proves that autofs behaves as > intended on mount. > Exactly. And, from what I understand, that would break if mountcheckdirs() got removed from the mount path. From owner-freebsd-fs@freebsd.org Fri Sep 15 11:45:39 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 872F4E105CB; Fri, 15 Sep 2017 11:45:39 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2C80F63B1C; Fri, 15 Sep 2017 11:45:39 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id v8FBjXxI081996 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 15 Sep 2017 14:45:33 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua v8FBjXxI081996 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id v8FBjX85081994; Fri, 15 Sep 2017 14:45:33 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 15 Sep 2017 14:45:33 +0300 From: Konstantin Belousov To: Edward Napierala Cc: Kirk McKusick , freebsd-fs , Andriy Gapon , "freebsd-arch@freebsd.org" Subject: Re: mount / unmount and mountcheckdirs() Message-ID: <20170915114533.GS78693@kib.kiev.ua> References: <134c7c6e-f4f1-ef38-cc50-0e56c27c9fb8@FreeBSD.org> <201709150314.v8F3Ea6B085072@chez.mckusick.com> <20170915092001.GK78693@kib.kiev.ua> <20170915103037.GM78693@kib.kiev.ua> <20170915110033.GO78693@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.0 (2017-09-02) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Sep 2017 11:45:39 -0000 On Fri, Sep 15, 2017 at 12:31:05PM +0100, Edward Napierala wrote: > 2017-09-15 12:00 GMT+01:00 Konstantin Belousov : > > > On Fri, Sep 15, 2017 at 11:49:17AM +0100, Edward Napierala wrote: > > > 2017-09-15 11:30 GMT+01:00 Konstantin Belousov : > > > > > > > On Fri, Sep 15, 2017 at 11:08:31AM +0100, Edward Napierala wrote: > > > > > 2017-09-15 10:20 GMT+01:00 Konstantin Belousov > >: > > > > > > I believe that the current autofs does not allow a process to get > > into > > > > > > this situation at all. > > > > > > > > > > > > > > > > It does. For example: > > > > > > > > > > [trasz@v2:~]% cd /media/md0 > > > > > [trasz@v2:/media/md0]% mount > > > > > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > > > > > devfs on /dev (devfs, local, multilabel) > > > > > map -hosts on /net (autofs) > > > > > map -media on /media (autofs) > > > > > [trasz@v2:/media/md0]% ls > > > > > [trasz@v2:/media/md0]% mount > > > > > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > > > > > devfs on /dev (devfs, local, multilabel) > > > > > map -hosts on /net (autofs) > > > > > map -media on /media (autofs) > > > > > /dev/md0 on /media/md0 (ufs, local, noatime, nosuid, automounted) > > > > > > > > > > Getting rid of mountcheckdirs() in the unmount path should be fine, I > > > > think. > > > > How the example proves that mountcheckdirs() can be removed ? > > > > > > > > > In the unmount path? It doesn't; I just don't think autofs would be > > > affected, > > > since having a current working directory in a mountpoint prevents the > > > unmount > > > (unless it's forced). > > > > > > In the mount path? See below. > > > > > > > > > > How can > > > > we see which directory content was printed by ls, the covered or > > mounted ? > > > > > > > > > > Well, it would be pretty useless if you got the covered directory. > > This is exactly what is being discussed. > > > > > But ok, here's > > > a better example, one that actually shows that: > > > > > > [trasz@v2:/media/md0]% mount > > > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > > > devfs on /dev (devfs, local, multilabel) > > > map -hosts on /net (autofs) > > > map -media on /media (autofs) > > > [trasz@v2:/media/md0]% ls -al > > > total 9 > > > drwxr-xr-x 3 root wheel 512 Sep 10 10:16 . > > > drwxr-xr-x 3 root wheel 512 Sep 8 11:42 .. > > > drwxrwxr-x 2 root operator 512 Sep 10 10:16 .snap > > > [trasz@v2:/media/md0]% mount > > > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > > > devfs on /dev (devfs, local, multilabel) > > > map -hosts on /net (autofs) > > > map -media on /media (autofs) > > > /dev/md0 on /media/md0 (ufs, local, noatime, nosuid, automounted) > > > > > > Notice the .snap/ directory; it's an empty UFS filesystem. > > And the .snap is from the /dev/md0 volume, right ? > > > > Yes. > > > > If yes, then this demonstration indeed proves that autofs behaves as > > intended on mount. > > > > Exactly. And, from what I understand, that would break if mountcheckdirs() > got removed from the mount path. Even for autofs ? Don't autofs postpone lookup() completion until the mount occurs ? From owner-freebsd-fs@freebsd.org Fri Sep 15 11:57:56 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AC065E10F4E; Fri, 15 Sep 2017 11:57:56 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citapm.icyb.net.ua (citapm.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id CE7C464151; Fri, 15 Sep 2017 11:57:54 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citapm.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id OAA20952; Fri, 15 Sep 2017 14:57:52 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1dspFo-000185-Dr; Fri, 15 Sep 2017 14:57:52 +0300 Subject: Re: mount / unmount and mountcheckdirs() To: Edward Napierala , Konstantin Belousov Cc: Kirk McKusick , freebsd-fs , "freebsd-arch@freebsd.org" References: <134c7c6e-f4f1-ef38-cc50-0e56c27c9fb8@FreeBSD.org> <201709150314.v8F3Ea6B085072@chez.mckusick.com> <20170915092001.GK78693@kib.kiev.ua> From: Andriy Gapon Message-ID: Date: Fri, 15 Sep 2017 14:56:55 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Sep 2017 11:57:56 -0000 On 15/09/2017 13:08, Edward Napierala wrote: > It does.  For example: > > [trasz@v2:~]% cd /media/md0 > [trasz@v2:/media/md0]% mount > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > devfs on /dev (devfs, local, multilabel) > map -hosts on /net (autofs) > map -media on /media (autofs) Interesting, I thought that /dev/md0 would get mounted as soon as /media/md0 is looked up. But maybe that would be sub-optimal for some common scenarios... FWIW, ZFS snapshots get auto-mounted under .zfs as soon as there is a lookup. > [trasz@v2:/media/md0]% ls > [trasz@v2:/media/md0]% mount > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > devfs on /dev (devfs, local, multilabel) > map -hosts on /net (autofs) > map -media on /media (autofs) > /dev/md0 on /media/md0 (ufs, local, noatime, nosuid, automounted) > -- Andriy Gapon From owner-freebsd-fs@freebsd.org Fri Sep 15 11:59:01 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5BA8CE11125; Fri, 15 Sep 2017 11:59:01 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citapm.icyb.net.ua (citapm.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 725D3643EC; Fri, 15 Sep 2017 11:59:00 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citapm.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id OAA20960; Fri, 15 Sep 2017 14:58:58 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1dspGs-000189-PW; Fri, 15 Sep 2017 14:58:58 +0300 Subject: Re: mount / unmount and mountcheckdirs() To: Kirk McKusick Cc: Konstantin Belousov , freebsd-arch@FreeBSD.org, freebsd-fs References: <201709150314.v8F3Ea6B085072@chez.mckusick.com> From: Andriy Gapon Message-ID: <8a8e6eb6-2a80-de22-4944-38820c327a76@FreeBSD.org> Date: Fri, 15 Sep 2017 14:58:23 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <201709150314.v8F3Ea6B085072@chez.mckusick.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Sep 2017 11:59:01 -0000 On 15/09/2017 06:14, Kirk McKusick wrote: > I had to dig back through some *really* old emails to find out what > triggered the addition of mountcheckdirs(). The problem that it was > specifically solving was that as part of the startup script a minimal > root directory was replaced by the real root directory. The shell > running the startup script needed to be moved to the new mounted-on > root so that the rest of the script would not fail. > > That disaster of a hack has been replaced with the much more functional > code that deals with setting up the root and the devfs filesystem on > /dev. So the need for which it was designed no longer exists. But I > still believe that it is the correct thing to do. For example, if you > are using automount code and chdir into your home directory triggering > an auto-mount, you should just be in your home directory after the > mount rather than having to do cd ../$USER to get there. Kirk, thank you very much both for the historical reference and for the auto-mount example. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Fri Sep 15 14:00:07 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9A11EE183DF for ; Fri, 15 Sep 2017 14:00:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 88BC56920D for ; Fri, 15 Sep 2017 14:00:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v8FE05N9018189 for ; Fri, 15 Sep 2017 14:00:07 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 222288] g_bio leak after zfs ABD commit Date: Fri, 15 Sep 2017 14:00:05 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: avg@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Sep 2017 14:00:07 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D222288 --- Comment #2 from Andriy Gapon --- Created attachment 186415 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D186415&action= =3Dedit work in progress This is a patch that I am testing right now. So far so good. --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Fri Sep 15 15:19:17 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 888AEE1CA31; Fri, 15 Sep 2017 15:19:17 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1C1DB6B9B8; Fri, 15 Sep 2017 15:19:17 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id v8FFJFVt065087; Fri, 15 Sep 2017 08:19:15 -0700 (PDT) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id v8FFJFHK065086; Fri, 15 Sep 2017 08:19:15 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201709151519.v8FFJFHK065086@pdx.rh.CN85.dnsmgr.net> Subject: Re: mount / unmount and mountcheckdirs() In-Reply-To: To: Edward Napierala Date: Fri, 15 Sep 2017 08:19:15 -0700 (PDT) CC: Konstantin Belousov , Kirk McKusick , freebsd-fs , Andriy Gapon , "freebsd-arch@freebsd.org" X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Sep 2017 15:19:17 -0000 > 2017-09-15 10:20 GMT+01:00 Konstantin Belousov : > > > On Thu, Sep 14, 2017 at 08:14:36PM -0700, Kirk McKusick wrote: > > > > To: Kirk McKusick > > > > Subject: Re: mount / unmount and mountcheckdirs() > > > > Cc: freebsd-arch@FreeBSD.org, freebsd-fs > > > > From: Andriy Gapon > > > > Date: Thu, 14 Sep 2017 15:45:07 +0300 > > > > > > > > On 22/05/2016 09:40, Kirk McKusick wrote: > > > >> I added the checkdirs functionality in the mount direction only > > > >> (I actually did it in 4.4BSD-Lite and it got swept in with commit > > > >> 22521). The reason is that when a directory that is not empty is > > > >> mounted on, the expectation is that the entries in that directory > > > >> should no longer be present; rather they should be replaced by the > > > >> entries in the newly mounted directory. Thus all processes sitting > > > >> in the mounted on directory should see the newly mounted directory > > > >> as if they had come to it using a lookup after the mount had been > > > >> done. If a process had proceeded through the mounted on directory > > > >> into one of its other entries, then they are left alone until such > > > >> time as they chdir back into the mount point directory through ".." > > > >> at which time they will be passed up to the mounted directory using > > > >> the same mechanism that would put them there if they traversed into > > > >> the mount point from above it in the tree. I believe this is the > > > >> correct behavior, is not a security threat, and should be left alone. > > > > > > > > I almost dropped a ball on this issue, but I am now picking it up > > again. > > > > At the moment I am moving forward with the dounmount change as it > > seems to be > > > > non-contentious and rather simple to do and test. > > > > > > > > Regarding the mount part, I am not sure that I completely agree with > > you. > > > > Even if mountcheckdirs() does not cause any problems in the mount > > path, I still > > > > fail to see its usefulness. Specifically, I still do not see any > > significant > > > > difference between the covered directory and any directory below it. > > So, if we > > > > leave the lower directories alone, while bother with the covered > > directory... > > > > > > > > The covered directory: > > > > - absolute paths work correctly > > > > - relative paths with enough ".." (one) can access the actual namespace > > > > - other relative paths operate on the shadowed sub-tree of the original > > > > filesystem > > > > > > > > > > > > The lower directories: > > > > - absolute paths work correctly > > > > - relative paths with enough ".." (> 1) can access the actual namespace > > > > - other relative paths operate on the shadowed sub-tree of the original > > > > filesystem > > > > > > > > The only difference I can think of is that the root of the mounted > > filesystem > > > > cannot be reached with just ".."-s from the covered directory. But is > > this > > > > difference of any significance? > > > > > > > > Mateusz also raised some interesting points. > > > > > > > > On the other hand, it seems that illumos and probably Solaris has > > > > interesting parallels to the FreeBSD behavior. It does not allow > > > > to mount over a directory that is a current directory for any process > > > > ("Device busy"), but does not object against processes in directories > > > > below the mount point. > > > > > > > > So, probably it's just I who misses something about that scenario :-) > > > > > > > >> I was not aware that the functionality had been added at unmount > > > >> time, and I do not believe that it should have been done. Normally > > > >> an unmount will not succeed if any vnodes are busy (for example, if > > > >> any directory in the filesystem is a current directory). The only > > > >> way that it can succeed in such a case is if a forcible unmount is > > > >> done. The forcible unmount will effectively do a revoke(2) on all > > > >> current directory vnodes in the unmounted filesystem. Further attempts > > > >> to access them will fail with "." not found errors. The only way to > > > >> get a valid current directory is to chdir to an absolute pathname. > > > >> Gratuitously fixing this if you happen to be in the former root of > > > >> the filesystem is wrong. And as you note can lead to unintensionally > > > >> giving an escape path from a prison. So I concur with your removing > > > >> this added functionality. > > > > -- > > > > Andriy Gapon > > > > > > I had to dig back through some *really* old emails to find out what > > > triggered the addition of mountcheckdirs(). The problem that it was > > > specifically solving was that as part of the startup script a minimal > > > root directory was replaced by the real root directory. The shell > > > running the startup script needed to be moved to the new mounted-on > > > root so that the rest of the script would not fail. > > If the mountcheckdirs() code not going away, please add your spelunking > > results as a comment before the function. This theme is recurring, and > > it would be highly beneficial to not loose the non-trivial reasoning > > behind the code existence. > > > > > > > > That disaster of a hack has been replaced with the much more functional > > > code that deals with setting up the root and the devfs filesystem on > > > /dev. So the need for which it was designed no longer exists. But I > > > still believe that it is the correct thing to do. For example, if you > > > are using automount code and chdir into your home directory triggering > > > an auto-mount, you should just be in your home directory after the > > > mount rather than having to do cd ../$USER to get there. > > I believe that the current autofs does not allow a process to get into > > this situation at all. > > > > It does. For example: > > [trasz@v2:~]% cd /media/md0 This process should be blocked before the cd is completed and resumed once the automouter has done its job. Even per the autofs man page that is what should be happening: DESCRIPTION The autofs driver is the kernel component of the automounter infrastructure. Its job is to pass mount requests to the automountd(8) daemon, and pause the processes trying to access the automounted ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ filesystem until the mount is completed. It is mounted by the ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ automount(8). > [trasz@v2:/media/md0]% mount > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > devfs on /dev (devfs, local, multilabel) > map -hosts on /net (autofs) > map -media on /media (autofs) > [trasz@v2:/media/md0]% ls This looks like a fail of the autofs code to block the process in cd before it changes the cwd. > [trasz@v2:/media/md0]% mount > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > devfs on /dev (devfs, local, multilabel) > map -hosts on /net (autofs) > map -media on /media (autofs) > /dev/md0 on /media/md0 (ufs, local, noatime, nosuid, automounted) > > Getting rid of mountcheckdirs() in the unmount path should be fine, I think. -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-fs@freebsd.org Fri Sep 15 15:35:03 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EE8A7E1DCF5; Fri, 15 Sep 2017 15:35:03 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9D7796C659; Fri, 15 Sep 2017 15:35:03 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id v8FFAP2N065054; Fri, 15 Sep 2017 08:10:25 -0700 (PDT) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id v8FFAPcg065053; Fri, 15 Sep 2017 08:10:25 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201709151510.v8FFAPcg065053@pdx.rh.CN85.dnsmgr.net> Subject: Re: mount / unmount and mountcheckdirs() In-Reply-To: <20170915092001.GK78693@kib.kiev.ua> To: Konstantin Belousov Date: Fri, 15 Sep 2017 08:10:25 -0700 (PDT) CC: Kirk McKusick , freebsd-fs , Andriy Gapon , freebsd-arch@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Sep 2017 15:35:04 -0000 > On Thu, Sep 14, 2017 at 08:14:36PM -0700, Kirk McKusick wrote: > > > To: Kirk McKusick > > > Subject: Re: mount / unmount and mountcheckdirs() > > > Cc: freebsd-arch@FreeBSD.org, freebsd-fs > > > From: Andriy Gapon > > > Date: Thu, 14 Sep 2017 15:45:07 +0300 > > > > > > On 22/05/2016 09:40, Kirk McKusick wrote: > > >> I added the checkdirs functionality in the mount direction only > > >> (I actually did it in 4.4BSD-Lite and it got swept in with commit > > >> 22521). The reason is that when a directory that is not empty is > > >> mounted on, the expectation is that the entries in that directory > > >> should no longer be present; rather they should be replaced by the > > >> entries in the newly mounted directory. Thus all processes sitting > > >> in the mounted on directory should see the newly mounted directory > > >> as if they had come to it using a lookup after the mount had been > > >> done. If a process had proceeded through the mounted on directory > > >> into one of its other entries, then they are left alone until such > > >> time as they chdir back into the mount point directory through ".." > > >> at which time they will be passed up to the mounted directory using > > >> the same mechanism that would put them there if they traversed into > > >> the mount point from above it in the tree. I believe this is the > > >> correct behavior, is not a security threat, and should be left alone. > > > > > > I almost dropped a ball on this issue, but I am now picking it up again. > > > At the moment I am moving forward with the dounmount change as it seems to be > > > non-contentious and rather simple to do and test. > > > > > > Regarding the mount part, I am not sure that I completely agree with you. > > > Even if mountcheckdirs() does not cause any problems in the mount path, I still > > > fail to see its usefulness. Specifically, I still do not see any significant > > > difference between the covered directory and any directory below it. So, if we > > > leave the lower directories alone, while bother with the covered directory... > > > > > > The covered directory: > > > - absolute paths work correctly > > > - relative paths with enough ".." (one) can access the actual namespace > > > - other relative paths operate on the shadowed sub-tree of the original > > > filesystem > > > > > > > > > The lower directories: > > > - absolute paths work correctly > > > - relative paths with enough ".." (> 1) can access the actual namespace > > > - other relative paths operate on the shadowed sub-tree of the original > > > filesystem > > > > > > The only difference I can think of is that the root of the mounted filesystem > > > cannot be reached with just ".."-s from the covered directory. But is this > > > difference of any significance? > > > > > > Mateusz also raised some interesting points. > > > > > > On the other hand, it seems that illumos and probably Solaris has > > > interesting parallels to the FreeBSD behavior. It does not allow > > > to mount over a directory that is a current directory for any process > > > ("Device busy"), but does not object against processes in directories > > > below the mount point. > > > > > > So, probably it's just I who misses something about that scenario :-) > > > > > >> I was not aware that the functionality had been added at unmount > > >> time, and I do not believe that it should have been done. Normally > > >> an unmount will not succeed if any vnodes are busy (for example, if > > >> any directory in the filesystem is a current directory). The only > > >> way that it can succeed in such a case is if a forcible unmount is > > >> done. The forcible unmount will effectively do a revoke(2) on all > > >> current directory vnodes in the unmounted filesystem. Further attempts > > >> to access them will fail with "." not found errors. The only way to > > >> get a valid current directory is to chdir to an absolute pathname. > > >> Gratuitously fixing this if you happen to be in the former root of > > >> the filesystem is wrong. And as you note can lead to unintensionally > > >> giving an escape path from a prison. So I concur with your removing > > >> this added functionality. > > > -- > > > Andriy Gapon > > > > I had to dig back through some *really* old emails to find out what > > triggered the addition of mountcheckdirs(). The problem that it was > > specifically solving was that as part of the startup script a minimal > > root directory was replaced by the real root directory. The shell > > running the startup script needed to be moved to the new mounted-on > > root so that the rest of the script would not fail. > If the mountcheckdirs() code not going away, please add your spelunking > results as a comment before the function. This theme is recurring, and > it would be highly beneficial to not loose the non-trivial reasoning > behind the code existence. > > > > > That disaster of a hack has been replaced with the much more functional > > code that deals with setting up the root and the devfs filesystem on > > /dev. So the need for which it was designed no longer exists. But I > > still believe that it is the correct thing to do. For example, if you > > are using automount code and chdir into your home directory triggering > > an auto-mount, you should just be in your home directory after the > > mount rather than having to do cd ../$USER to get there. > I believe that the current autofs does not allow a process to get into > this situation at all. I believe the ancient automountd did not allow a process to get into this situation either, the process got blocked before the completion of the cd that triggers the automount, and continued after the (auto)mount completed. > > In fact, the behavior implemented by mountcheckdirs() is surprising as > well. For instance, I did expected that the system would operate as if > mountcheckdirs() does not exist, and it caused me some head-scratching > when I see it first time. > _______________________________________________ > freebsd-arch@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-fs@freebsd.org Fri Sep 15 16:02:05 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B37B8E1F52A; Fri, 15 Sep 2017 16:02:05 +0000 (UTC) (envelope-from etnapierala@gmail.com) Received: from mail-ua0-x229.google.com (mail-ua0-x229.google.com [IPv6:2607:f8b0:400c:c08::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 67B346D643; Fri, 15 Sep 2017 16:02:05 +0000 (UTC) (envelope-from etnapierala@gmail.com) Received: by mail-ua0-x229.google.com with SMTP id c27so1476635uah.2; Fri, 15 Sep 2017 09:02:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=aTqGj5uk1GG7tfE+vU6w9obFTF6Vdz5wtzR1inaQaVY=; b=vgo6jRb5lRk1OnaEsc2HMv3ESvQlsN0/VAwAFwXf1mQF2/aRnlZ3OmlISE3MdzStv9 6ecG6KzvzH8hIv68FGTeD8bmYPpTg9ElFp61BaIDQpTThxoZxbYEPoID37w9GXpXVu7f s+llTtQ24FpZQ6lT030x5CzYfqpVo2nnQ2t7fMMmUgd/VEs5uxkbQlrma6564u+F+K0U XIyu1Jrt1p6DaM7sAxa/G1pNB1R5H/zbbXWdMLmmX9Rbm0rI7GcHxZLgrGowsSSrzxc3 O3loVa4oG6xnXd4jKBrGv4T4Qt9B1btcafBajlmK0n9dlECRQ/mB/fHtoFwhsfUnbj6k m6dQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=aTqGj5uk1GG7tfE+vU6w9obFTF6Vdz5wtzR1inaQaVY=; b=eu+3xhS50dzGwzPryvgQdSO9vc/nONUd6N29mMFTHkyD+GaWfkkJS3sNiIT5nkh5cT d7RS++gaVwZRhg/N8kwfoBfl4dVIxA+ANmO1Wd8rqXaMSd1WZ0Bpqf3fs+xtdcki7xd3 imQwM7OtWDMqLpqRWgTauhg2b+cJuBmbJ/60dCS4wh0SYJa0QXYEfEf0a3I4jxA7vSiR plUv3R4E3pPXfqsLekJJeTYjq4x37i+S1x9cTu1AGLFLcIdWVJtbo6eMTvy2RB8U9ytP QO93QHrFdbMuq4XxwaFPkxLI3PO5PvcBFvpIOaDwFk8u0ejkQd1+BgTh/6tdbaj1NqHv br7Q== X-Gm-Message-State: AHPjjUiEBQxY8ilnCc35oYiIZ7+XoC9KSANlt0AFKKWppJgSAl7leSYd fuVCR7nPtH2/X6AFJbgFR0OROR1QO1eDCQonfhI= X-Google-Smtp-Source: ADKCNb7RmgZhGgKJq+HyxU7oDSIZ0S6E1vdo7E1BCrBeM64NEjco6mS1JxJNzvam1JIxl5DqhVGWwI/1eonwk5Rww40= X-Received: by 10.176.92.111 with SMTP id a47mr22014144uag.111.1505491324300; Fri, 15 Sep 2017 09:02:04 -0700 (PDT) MIME-Version: 1.0 Sender: etnapierala@gmail.com Received: by 10.159.51.232 with HTTP; Fri, 15 Sep 2017 09:02:03 -0700 (PDT) In-Reply-To: References: <134c7c6e-f4f1-ef38-cc50-0e56c27c9fb8@FreeBSD.org> <201709150314.v8F3Ea6B085072@chez.mckusick.com> <20170915092001.GK78693@kib.kiev.ua> From: Edward Napierala Date: Fri, 15 Sep 2017 17:02:03 +0100 X-Google-Sender-Auth: YQoXymOHcUVy7BkjINTxt8_Zdv0 Message-ID: Subject: Re: mount / unmount and mountcheckdirs() To: Andriy Gapon Cc: Konstantin Belousov , Kirk McKusick , freebsd-fs , "freebsd-arch@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Sep 2017 16:02:05 -0000 2017-09-15 12:56 GMT+01:00 Andriy Gapon : > On 15/09/2017 13:08, Edward Napierala wrote: > > It does. For example: > > > > [trasz@v2:~]% cd /media/md0 > > [trasz@v2:/media/md0]% mount > > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > > devfs on /dev (devfs, local, multilabel) > > map -hosts on /net (autofs) > > map -media on /media (autofs) > > Interesting, I thought that /dev/md0 would get mounted as soon as > /media/md0 is > looked up. But maybe that would be sub-optimal for some common > scenarios... > FWIW, ZFS snapshots get auto-mounted under .zfs as soon as there is a > lookup. You really don't want it to work that way. If you have a directory with thousands of mountpoints inside, you don't want to mount all of them every time you do "ls" in that directory. Same problem with mounting on "cd". That's why autofs triggers in three cases: in VOP_READDIR(9), VOP_GETATTR(9), and VOP_LOOKUP(9) with the vnode as a parent dir. From owner-freebsd-fs@freebsd.org Fri Sep 15 16:06:11 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E53A3E1F972; Fri, 15 Sep 2017 16:06:11 +0000 (UTC) (envelope-from etnapierala@gmail.com) Received: from mail-vk0-x232.google.com (mail-vk0-x232.google.com [IPv6:2607:f8b0:400c:c05::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 97E2D6D958; Fri, 15 Sep 2017 16:06:11 +0000 (UTC) (envelope-from etnapierala@gmail.com) Received: by mail-vk0-x232.google.com with SMTP id z187so1243413vkd.13; Fri, 15 Sep 2017 09:06:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=t+d19FruIl6nVb34JPaYdPppoSC01L1YhVjmk3RJQMY=; b=FF0lVHJRCb1Kw6zki8WJ5nGbHOZ/oKmnoLXdHSoEThD1bvEEq6P22ljBlNyHnk9jAY ZR0CF8NKdmwHNNB/hej6hcxyrIm+7J2gi9z//k4/RPp5l+SFx9jxDwx2vCwu68zJR27C JkzZ5ZqHhK23cvUKW3uNPXAeNsOvF2GbITOW9xE88kAQwUxleEN0bW38sCES4FMVJRty EM71jMtzNy83UM0HlpvbNOb2cznfP1P2YxtofXHepzAv+TD/lPp+5NAuvTVA1L8JiKu6 LmAKyClmXvWQoco5MKJVdDuGMIEkPYetitU4keeNS0b28rRyMI0Yt1mONcGozjA3Rh1h lzNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=t+d19FruIl6nVb34JPaYdPppoSC01L1YhVjmk3RJQMY=; b=d28UWVLA/XHBC8aj+zYDAsZ7Owq1t9vM7C7j4ON+muBwY3UZId4pOwuOmJgXy9XywR Dkqav7F1n6dgnyU03MTLwW3pDWw8cGi98j8fm9GIata1713BeV0w38dfTbBSJ1vA0qim pqGqCXnPZQRnwjGE9uOgPkpkgZxr7P77cPA9Ea2f9QmTF+vz4F0fd4VgATimcJ2ZaAXE IrK3MvIoSGFhTLY2tE0Uoo5M8YNDHoBCTIIBa1Ogp0e0yh5svT4TBhApwYZkh2u4EsI3 7vE5U2j2TrhVHVdc64adl43avwZYLeQOEz9AD/oRvBbLVAX8yWC9q5onruXBos29v4UY SoyA== X-Gm-Message-State: AHPjjUgBnA8eUdjhhwWVIpQFavXOhBZjMDUpCB1/sUY8yCy7bQHA3qiq xafQtX1C9NZB1P5mAKgeBiS0mkgKLIL5VxOO38I= X-Google-Smtp-Source: AOwi7QDwvvBDFnW6Qpoq4ZO6sbhzScbi/yn3lwPPjOk3fQcug7RaUchHdqsoXLEghhIwY5R97ksiEtIb3Ubh+9nRXu8= X-Received: by 10.31.10.19 with SMTP id 19mr19326393vkk.3.1505491570639; Fri, 15 Sep 2017 09:06:10 -0700 (PDT) MIME-Version: 1.0 Sender: etnapierala@gmail.com Received: by 10.159.51.232 with HTTP; Fri, 15 Sep 2017 09:06:10 -0700 (PDT) In-Reply-To: <20170915114533.GS78693@kib.kiev.ua> References: <134c7c6e-f4f1-ef38-cc50-0e56c27c9fb8@FreeBSD.org> <201709150314.v8F3Ea6B085072@chez.mckusick.com> <20170915092001.GK78693@kib.kiev.ua> <20170915103037.GM78693@kib.kiev.ua> <20170915110033.GO78693@kib.kiev.ua> <20170915114533.GS78693@kib.kiev.ua> From: Edward Napierala Date: Fri, 15 Sep 2017 17:06:10 +0100 X-Google-Sender-Auth: qCK8B-Up8hPQ1H4SxiHOdnDAhy0 Message-ID: Subject: Re: mount / unmount and mountcheckdirs() To: Konstantin Belousov Cc: Kirk McKusick , freebsd-fs , Andriy Gapon , "freebsd-arch@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Sep 2017 16:06:12 -0000 2017-09-15 12:45 GMT+01:00 Konstantin Belousov : > On Fri, Sep 15, 2017 at 12:31:05PM +0100, Edward Napierala wrote: > > 2017-09-15 12:00 GMT+01:00 Konstantin Belousov : > > > > > On Fri, Sep 15, 2017 at 11:49:17AM +0100, Edward Napierala wrote: > > > > 2017-09-15 11:30 GMT+01:00 Konstantin Belousov >: > > > > > > > > > On Fri, Sep 15, 2017 at 11:08:31AM +0100, Edward Napierala wrote: > > > > > > 2017-09-15 10:20 GMT+01:00 Konstantin Belousov < > kostikbel@gmail.com > > > >: > > > > > > > I believe that the current autofs does not allow a process to > get > > > into > > > > > > > this situation at all. > > > > > > > > > > > > > > > > > > > It does. For example: > > > > > > > > > > > > [trasz@v2:~]% cd /media/md0 > > > > > > [trasz@v2:/media/md0]% mount > > > > > > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > > > > > > devfs on /dev (devfs, local, multilabel) > > > > > > map -hosts on /net (autofs) > > > > > > map -media on /media (autofs) > > > > > > [trasz@v2:/media/md0]% ls > > > > > > [trasz@v2:/media/md0]% mount > > > > > > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > > > > > > devfs on /dev (devfs, local, multilabel) > > > > > > map -hosts on /net (autofs) > > > > > > map -media on /media (autofs) > > > > > > /dev/md0 on /media/md0 (ufs, local, noatime, nosuid, automounted) > > > > > > > > > > > > Getting rid of mountcheckdirs() in the unmount path should be > fine, I > > > > > think. > > > > > How the example proves that mountcheckdirs() can be removed ? > > > > > > > > > > > > In the unmount path? It doesn't; I just don't think autofs would be > > > > affected, > > > > since having a current working directory in a mountpoint prevents the > > > > unmount > > > > (unless it's forced). > > > > > > > > In the mount path? See below. > > > > > > > > > > > > > How can > > > > > we see which directory content was printed by ls, the covered or > > > mounted ? > > > > > > > > > > > > > Well, it would be pretty useless if you got the covered directory. > > > This is exactly what is being discussed. > > > > > > > But ok, here's > > > > a better example, one that actually shows that: > > > > > > > > [trasz@v2:/media/md0]% mount > > > > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > > > > devfs on /dev (devfs, local, multilabel) > > > > map -hosts on /net (autofs) > > > > map -media on /media (autofs) > > > > [trasz@v2:/media/md0]% ls -al > > > > total 9 > > > > drwxr-xr-x 3 root wheel 512 Sep 10 10:16 . > > > > drwxr-xr-x 3 root wheel 512 Sep 8 11:42 .. > > > > drwxrwxr-x 2 root operator 512 Sep 10 10:16 .snap > > > > [trasz@v2:/media/md0]% mount > > > > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > > > > devfs on /dev (devfs, local, multilabel) > > > > map -hosts on /net (autofs) > > > > map -media on /media (autofs) > > > > /dev/md0 on /media/md0 (ufs, local, noatime, nosuid, automounted) > > > > > > > > Notice the .snap/ directory; it's an empty UFS filesystem. > > > And the .snap is from the /dev/md0 volume, right ? > > > > > > > Yes. > > > > > > > If yes, then this demonstration indeed proves that autofs behaves as > > > intended on mount. > > > > > > > Exactly. And, from what I understand, that would break if > mountcheckdirs() > > got removed from the mount path. > > Even for autofs ? Don't autofs postpone lookup() completion until the > mount occurs ? > It does, and that case might work - I'm not sure, but it might. But what happens afterwards, after the syscall that triggered the mount completes? You would still have a shell with cwd in that dir, now covered by another filesystem. Autofs doesn't do anything to "lift" the process to the newly mounted filesystem root. From owner-freebsd-fs@freebsd.org Fri Sep 15 16:28:02 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B3923E20D75; Fri, 15 Sep 2017 16:28:02 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5B5DE6E387; Fri, 15 Sep 2017 16:28:02 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id v8FGRrrV045173 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 15 Sep 2017 19:27:53 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua v8FGRrrV045173 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id v8FGRqJq045172; Fri, 15 Sep 2017 19:27:52 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 15 Sep 2017 19:27:52 +0300 From: Konstantin Belousov To: Edward Napierala Cc: Kirk McKusick , freebsd-fs , Andriy Gapon , "freebsd-arch@freebsd.org" Subject: Re: mount / unmount and mountcheckdirs() Message-ID: <20170915162752.GX78693@kib.kiev.ua> References: <134c7c6e-f4f1-ef38-cc50-0e56c27c9fb8@FreeBSD.org> <201709150314.v8F3Ea6B085072@chez.mckusick.com> <20170915092001.GK78693@kib.kiev.ua> <20170915103037.GM78693@kib.kiev.ua> <20170915110033.GO78693@kib.kiev.ua> <20170915114533.GS78693@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.0 (2017-09-02) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Sep 2017 16:28:02 -0000 On Fri, Sep 15, 2017 at 05:06:10PM +0100, Edward Napierala wrote: > 2017-09-15 12:45 GMT+01:00 Konstantin Belousov : > > > On Fri, Sep 15, 2017 at 12:31:05PM +0100, Edward Napierala wrote: > > > 2017-09-15 12:00 GMT+01:00 Konstantin Belousov : > > > > > > > On Fri, Sep 15, 2017 at 11:49:17AM +0100, Edward Napierala wrote: > > > > > 2017-09-15 11:30 GMT+01:00 Konstantin Belousov > >: > > > > > > > > > > > On Fri, Sep 15, 2017 at 11:08:31AM +0100, Edward Napierala wrote: > > > > > > > 2017-09-15 10:20 GMT+01:00 Konstantin Belousov < > > kostikbel@gmail.com > > > > >: > > > > > > > > I believe that the current autofs does not allow a process to > > get > > > > into > > > > > > > > this situation at all. > > > > > > > > > > > > > > > > > > > > > > It does. For example: > > > > > > > > > > > > > > [trasz@v2:~]% cd /media/md0 > > > > > > > [trasz@v2:/media/md0]% mount > > > > > > > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > > > > > > > devfs on /dev (devfs, local, multilabel) > > > > > > > map -hosts on /net (autofs) > > > > > > > map -media on /media (autofs) > > > > > > > [trasz@v2:/media/md0]% ls > > > > > > > [trasz@v2:/media/md0]% mount > > > > > > > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > > > > > > > devfs on /dev (devfs, local, multilabel) > > > > > > > map -hosts on /net (autofs) > > > > > > > map -media on /media (autofs) > > > > > > > /dev/md0 on /media/md0 (ufs, local, noatime, nosuid, automounted) > > > > > > > > > > > > > > Getting rid of mountcheckdirs() in the unmount path should be > > fine, I > > > > > > think. > > > > > > How the example proves that mountcheckdirs() can be removed ? > > > > > > > > > > > > > > > In the unmount path? It doesn't; I just don't think autofs would be > > > > > affected, > > > > > since having a current working directory in a mountpoint prevents the > > > > > unmount > > > > > (unless it's forced). > > > > > > > > > > In the mount path? See below. > > > > > > > > > > > > > > > > How can > > > > > > we see which directory content was printed by ls, the covered or > > > > mounted ? > > > > > > > > > > > > > > > > Well, it would be pretty useless if you got the covered directory. > > > > This is exactly what is being discussed. > > > > > > > > > But ok, here's > > > > > a better example, one that actually shows that: > > > > > > > > > > [trasz@v2:/media/md0]% mount > > > > > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > > > > > devfs on /dev (devfs, local, multilabel) > > > > > map -hosts on /net (autofs) > > > > > map -media on /media (autofs) > > > > > [trasz@v2:/media/md0]% ls -al > > > > > total 9 > > > > > drwxr-xr-x 3 root wheel 512 Sep 10 10:16 . > > > > > drwxr-xr-x 3 root wheel 512 Sep 8 11:42 .. > > > > > drwxrwxr-x 2 root operator 512 Sep 10 10:16 .snap > > > > > [trasz@v2:/media/md0]% mount > > > > > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > > > > > devfs on /dev (devfs, local, multilabel) > > > > > map -hosts on /net (autofs) > > > > > map -media on /media (autofs) > > > > > /dev/md0 on /media/md0 (ufs, local, noatime, nosuid, automounted) > > > > > > > > > > Notice the .snap/ directory; it's an empty UFS filesystem. > > > > And the .snap is from the /dev/md0 volume, right ? > > > > > > > > > > Yes. > > > > > > > > > > If yes, then this demonstration indeed proves that autofs behaves as > > > > intended on mount. > > > > > > > > > > Exactly. And, from what I understand, that would break if > > mountcheckdirs() > > > got removed from the mount path. > > > > Even for autofs ? Don't autofs postpone lookup() completion until the > > mount occurs ? > > > > It does, and that case might work - I'm not sure, but it might. But what > happens afterwards, after the syscall that triggered the mount completes? > You would still have a shell with cwd in that dir, now covered by another > filesystem. Autofs doesn't do anything to "lift" the process to the newly > mounted filesystem root. The change of the current directory should have already triggered the action, because it was preceeded by lookup. At least this is my understanding of the flow. From owner-freebsd-fs@freebsd.org Fri Sep 15 16:37:08 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AD01CE21541; Fri, 15 Sep 2017 16:37:08 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citapm.icyb.net.ua (citapm.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 6B4836E84A; Fri, 15 Sep 2017 16:37:07 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citapm.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id TAA21439; Fri, 15 Sep 2017 19:37:00 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1dstbv-0001KC-SP; Fri, 15 Sep 2017 19:36:59 +0300 Subject: Re: mount / unmount and mountcheckdirs() To: Edward Napierala Cc: Konstantin Belousov , Kirk McKusick , freebsd-fs , "freebsd-arch@freebsd.org" References: <134c7c6e-f4f1-ef38-cc50-0e56c27c9fb8@FreeBSD.org> <201709150314.v8F3Ea6B085072@chez.mckusick.com> <20170915092001.GK78693@kib.kiev.ua> From: Andriy Gapon Message-ID: <59255d6c-07fc-19ab-0a8a-914ada69680a@FreeBSD.org> Date: Fri, 15 Sep 2017 19:36:04 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Sep 2017 16:37:08 -0000 On 15/09/2017 19:02, Edward Napierala wrote: > 2017-09-15 12:56 GMT+01:00 Andriy Gapon >: > > On 15/09/2017 13:08, Edward Napierala wrote: > > It does.  For example: > > > > [trasz@v2:~]% cd /media/md0 > > [trasz@v2:/media/md0]% mount > > /dev/ada0s1a on / (ufs, local, noatime, journaled soft-updates) > > devfs on /dev (devfs, local, multilabel) > > map -hosts on /net (autofs) > > map -media on /media (autofs) > > Interesting, I thought that /dev/md0 would get mounted as soon as /media/md0 is > looked up.  But maybe that would be sub-optimal for some common scenarios... > FWIW, ZFS snapshots get auto-mounted under .zfs as soon as there is a lookup. > > > You really don't want it to work that way.  If you have a directory with thousands > of mountpoints inside, you don't want to mount all of them every time you do "ls" > in that directory. Well, "ls" wouldn't trigger the mounting as it's just VOP_READDIR in the parent vnode, "ls -l" (or any other options that require file attributes) is a different story :-) > Same problem with mounting on "cd".  That's why autofs triggers > in three cases: in VOP_READDIR(9), VOP_GETATTR(9), and VOP_LOOKUP(9) > with the vnode as a parent dir. Well, "cd" is also a different story, you can't cd to hundreds of directories at once. Of course, the filesystem doesn't really know if it's "cd" or something else that triggered VOP_LOOKUP. Although VOP_ACCESS could be used as a hint, it is a rather weak hint. In any case, I prefer the mount-on-lookup approach as it is "more seamless". -- Andriy Gapon From owner-freebsd-fs@freebsd.org Fri Sep 15 16:41:01 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EE6CBE219FD; Fri, 15 Sep 2017 16:41:01 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citapm.icyb.net.ua (citapm.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id A22376EB4B; Fri, 15 Sep 2017 16:41:00 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citapm.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id TAA21446; Fri, 15 Sep 2017 19:40:59 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1dstfn-0001KN-8a; Fri, 15 Sep 2017 19:40:59 +0300 Subject: Re: mount / unmount and mountcheckdirs() To: Konstantin Belousov , Edward Napierala Cc: Kirk McKusick , freebsd-fs , "freebsd-arch@freebsd.org" References: <134c7c6e-f4f1-ef38-cc50-0e56c27c9fb8@FreeBSD.org> <201709150314.v8F3Ea6B085072@chez.mckusick.com> <20170915092001.GK78693@kib.kiev.ua> <20170915103037.GM78693@kib.kiev.ua> <20170915110033.GO78693@kib.kiev.ua> <20170915114533.GS78693@kib.kiev.ua> <20170915162752.GX78693@kib.kiev.ua> From: Andriy Gapon Message-ID: <317b4617-5f0b-bc9a-316f-db1d8cf661fa@FreeBSD.org> Date: Fri, 15 Sep 2017 19:40:22 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <20170915162752.GX78693@kib.kiev.ua> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Sep 2017 16:41:02 -0000 On 15/09/2017 19:27, Konstantin Belousov wrote: > On Fri, Sep 15, 2017 at 05:06:10PM +0100, Edward Napierala wrote: >> It does, and that case might work - I'm not sure, but it might. But what >> happens afterwards, after the syscall that triggered the mount completes? >> You would still have a shell with cwd in that dir, now covered by another >> filesystem. Autofs doesn't do anything to "lift" the process to the newly >> mounted filesystem root. > The change of the current directory should have already triggered the action, > because it was preceeded by lookup. At least this is my understanding of > the flow. >From the earlier example I can only conclude that autofs does NOT do a mount on lookup in a sense of *vpp being a root of the mounted filesystem. My interpretation is the mounting is done only if there is a VOP_LOOKUP (or VOP_READDIR) with a mount point as a dvp. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Fri Sep 15 18:40:06 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2932DE01D01 for ; Fri, 15 Sep 2017 18:40:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 181D6730D5 for ; Fri, 15 Sep 2017 18:40:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v8FIe59b046004 for ; Fri, 15 Sep 2017 18:40:05 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 218626] [PATCH] cuse: new error code CUSE_ERR_NO_DEVICE (ENODEV) Date: Fri, 15 Sep 2017 18:40:05 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: hselasky@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Sep 2017 18:40:06 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D218626 --- Comment #3 from Hans Petter Selasky --- Ping. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Sep 16 11:50:05 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 52356E114FB for ; Sat, 16 Sep 2017 11:50:05 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from dss.incore.de (dss.incore.de [195.145.1.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D9BCB73564 for ; Sat, 16 Sep 2017 11:50:03 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from inetmail.dmz (inetmail.dmz [10.3.0.3]) by dss.incore.de (Postfix) with ESMTP id AA19C6796C; Sat, 16 Sep 2017 13:44:46 +0200 (CEST) X-Virus-Scanned: amavisd-new at incore.de Received: from dss.incore.de ([10.3.0.3]) by inetmail.dmz (inetmail.dmz [10.3.0.3]) (amavisd-new, port 10024) with LMTP id lADscixme4u5; Sat, 16 Sep 2017 13:44:45 +0200 (CEST) Received: from mail.local.incore (fwintern.dmz [10.0.0.253]) by dss.incore.de (Postfix) with ESMTP id 26C0D67970; Sat, 16 Sep 2017 13:44:45 +0200 (CEST) Received: from bsdmhs.longwitz (unknown [192.168.99.6]) by mail.local.incore (Postfix) with ESMTP id C08D5508A1; Sat, 16 Sep 2017 13:44:44 +0200 (CEST) Message-ID: <59BD0EAC.8030206@incore.de> Date: Sat, 16 Sep 2017 13:44:44 +0200 From: Andreas Longwitz User-Agent: Thunderbird 2.0.0.19 (X11/20090113) MIME-Version: 1.0 To: Kirk McKusick CC: freebsd-fs@freebsd.org Subject: Re: fsync: giving up on dirty on ufs partitions running vfs_write_suspend() References: <201709110519.v8B5JVmf060773@chez.mckusick.com> In-Reply-To: <201709110519.v8B5JVmf060773@chez.mckusick.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Sep 2017 11:50:05 -0000 Hello Kirk, >> Second I found, that the "dirty" situation during vfs_write_suspend() >> only occurs when a big file (more than 10G on a partition of 116G) is >> removed. If vfs_write_suspend() is called immediately after "rm >> bigfile", then in vop_stdfsync() 1000 tries (maxretry) are done to wait >> for the "rm bigfile" to complete. Because a lot of bitmap writes must be >> done, the value 1000 is not sufficient on my servers. I have increased >> maxretry and in the worst case I saw 8650 tries to complete without >> "dirty". In this case the time spent in vop_stdfsync() was about 0,5 >> seconds. The following patch solves the "dirty problem" for me: >> >> --- vfs_default.c.orig 2016-10-24 12:26:57.000000000 +0200 >> +++ vfs_default.c 2017-09-08 12:49:18.059970000 +0200 >> @@ -644,7 +644,7 @@ >> struct bufobj *bo; >> struct buf *nbp; >> int error = 0; >> - int maxretry = 1000; /* large, arbitrarily chosen */ >> + int maxretry = 100000; /* large, arbitrarily chosen */ >> >> bo = &vp->v_bufobj; >> BO_LOCK(bo); > > This message has plagued me for years. It started out as a panic, > then got changed to a printf because I could not get rid of it. I > was never able to figure out why it should take more than five > iterations to finish, but obviously it takes more. The 1000 number > was picked because that just seemed insanely large and I did not > want to iterate forever. I have no problem with bumping up the > iteration count if there is some way to figure out that each iteration > is making forward progress (so we know that we are not in an infinite > loop). Can you come up with a scheme that can measure forward progress? > I would much prefer that to just making this number ever bigger. > > Kirk McKusick Ok, I understand your thoughts about the "big loop" and I agree. On the other side it is not easy to measure the progress of the dirty buffers because these buffers a created from another process at the same time we loop in vop_stdfsync(). I can explain from my tests, where I use the following loop on a gjournaled partition: while true; do cp -p bigfile bigfile.tmp rm bigfile mv bigfile.tmp bigfile done When g_journal_switcher starts vfs_write_suspend() immediately after the rm command has started to do his "rm stuff" (ufs_inactive, ffs_truncate, ffs_indirtrunc at different levels, ffs_blkfree, ...) the we must loop (that means wait) in vop_stdfsync() until the rm process has finished his work. A lot of locking overhead is needed for coordination. Returning from bufobj_wwait() we always see one left dirty buffer (very seldom two), that is not optimal. Therefore I have tried the following patch (instead of bumping maxretry): --- vfs_default.c.orig 2016-10-24 12:26:57.000000000 +0200 +++ vfs_default.c 2017-09-15 12:30:44.792274000 +0200 @@ -688,6 +688,8 @@ bremfree(bp); bawrite(bp); } + if( maxretry < 1000) + DELAY(waitns); BO_LOCK(bo); goto loop2; } with different values for waitns. If I run the testloop 5000 times on my testserver, the problem is triggered always round about 10 times. The results from several runs are given in the following table: waitns max time max loops ------------------------------- no DELAY 0,5 sec 8650 (maxres = 100000) 1000 0,2 sec 24 10000 0,8 sec 3 100000 7,2 sec 3 "time" means spent time in vop_stdfsync() measured from entry to return by a dtrace script. "loops" means the number of times "--maxretry" is executed. I am not sure if DELAY() is the best way to wait or if waiting has other drawbacks. Anyway with DELAY() it does not take more than five iterazions to finish. -- Andreas Longwitz From owner-freebsd-fs@freebsd.org Sat Sep 16 18:31:23 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1F8ADE005D3 for ; Sat, 16 Sep 2017 18:31:23 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B1AB483F1A for ; Sat, 16 Sep 2017 18:31:22 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id v8GIVHnQ094010 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 16 Sep 2017 21:31:17 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua v8GIVHnQ094010 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id v8GIVHIA094009; Sat, 16 Sep 2017 21:31:17 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 16 Sep 2017 21:31:17 +0300 From: Konstantin Belousov To: Andreas Longwitz Cc: Kirk McKusick , freebsd-fs@freebsd.org Subject: Re: fsync: giving up on dirty on ufs partitions running vfs_write_suspend() Message-ID: <20170916183117.GF78693@kib.kiev.ua> References: <201709110519.v8B5JVmf060773@chez.mckusick.com> <59BD0EAC.8030206@incore.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <59BD0EAC.8030206@incore.de> User-Agent: Mutt/1.9.0 (2017-09-02) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Sep 2017 18:31:23 -0000 On Sat, Sep 16, 2017 at 01:44:44PM +0200, Andreas Longwitz wrote: > Ok, I understand your thoughts about the "big loop" and I agree. On the > other side it is not easy to measure the progress of the dirty buffers > because these buffers a created from another process at the same time we > loop in vop_stdfsync(). I can explain from my tests, where I use the > following loop on a gjournaled partition: > > while true; do > cp -p bigfile bigfile.tmp > rm bigfile > mv bigfile.tmp bigfile > done > > When g_journal_switcher starts vfs_write_suspend() immediately after the > rm command has started to do his "rm stuff" (ufs_inactive, ffs_truncate, > ffs_indirtrunc at different levels, ffs_blkfree, ...) the we must loop > (that means wait) in vop_stdfsync() until the rm process has finished > his work. A lot of locking overhead is needed for coordination. > Returning from bufobj_wwait() we always see one left dirty buffer (very > seldom two), that is not optimal. Therefore I have tried the following > patch (instead of bumping maxretry): > > --- vfs_default.c.orig 2016-10-24 12:26:57.000000000 +0200 > +++ vfs_default.c 2017-09-15 12:30:44.792274000 +0200 > @@ -688,6 +688,8 @@ > bremfree(bp); > bawrite(bp); > } > + if( maxretry < 1000) > + DELAY(waitns); > BO_LOCK(bo); > goto loop2; > } > > with different values for waitns. If I run the testloop 5000 times on my > testserver, the problem is triggered always round about 10 times. The > results from several runs are given in the following table: > > waitns max time max loops > ------------------------------- > no DELAY 0,5 sec 8650 (maxres = 100000) > 1000 0,2 sec 24 > 10000 0,8 sec 3 > 100000 7,2 sec 3 > > "time" means spent time in vop_stdfsync() measured from entry to return > by a dtrace script. "loops" means the number of times "--maxretry" is > executed. I am not sure if DELAY() is the best way to wait or if waiting > has other drawbacks. Anyway with DELAY() it does not take more than five > iterazions to finish. This is not explicitly stated in your message, but I suppose that the vop_stdfsync() is called due to VOP_FSYNC(devvp, MNT_SUSPEND) call in ffs_sync(). Am I right ? If yes, then the solution is most likely to continue looping in the vop_stdfsync() until there is no dirty buffers or the mount point mnt_secondary_writes counter is zero. The pauses trick you tried might be still useful, e.g. after some threshold of the performed loop iterations. Some problem with this suggestion is that vop_stdfsync(devvp) needs to know that the vnode is devvp for some UFS mount. The struct cdev, acessible as v_rdev, has the pointer to struct mount. You should be accurate to not access freed or reused struct mount. From owner-freebsd-fs@freebsd.org Sat Sep 16 20:14:26 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8C920E06165 for ; Sat, 16 Sep 2017 20:14:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7B0F0354B for ; Sat, 16 Sep 2017 20:14:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v8GKEQAV014322 for ; Sat, 16 Sep 2017 20:14:26 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 222227] ZFS: Confusing error message when attempting to create zpool on too small media Date: Sat, 16 Sep 2017 20:14:26 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Sep 2017 20:14:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D222227 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Sep 16 20:16:52 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 31303E063D0 for ; Sat, 16 Sep 2017 20:16:52 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [70.36.157.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 194ED3677 for ; Sat, 16 Sep 2017 20:16:51 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (localhost [IPv6:::1]) by chez.mckusick.com (8.15.2/8.15.2) with ESMTP id v8GKKQj0033706; Sat, 16 Sep 2017 13:20:26 -0700 (PDT) (envelope-from mckusick@chez.mckusick.com) Message-Id: <201709162020.v8GKKQj0033706@chez.mckusick.com> From: Kirk McKusick To: Andreas Longwitz Subject: Re: fsync: giving up on dirty on ufs partitions running vfs_write_suspend() cc: freebsd-fs@freebsd.org In-reply-to: <59BD0EAC.8030206@incore.de> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <33704.1505593226.1@chez.mckusick.com> Date: Sat, 16 Sep 2017 13:20:26 -0700 X-Spam-Status: No, score=0.1 required=5.0 tests=MISSING_MID, UNPARSEABLE_RELAY autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on chez.mckusick.com X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Sep 2017 20:16:52 -0000 > From: Konstantin Belousov > Date: Sat, 16 Sep 2017 21:31:17 +0300 > To: Andreas Longwitz > Subject: Re: fsync: giving up on dirty on ufs partitions running > vfs_write_suspend() > Cc: Kirk McKusick , freebsd-fs@freebsd.org > > On Sat, Sep 16, 2017 at 01:44:44PM +0200, Andreas Longwitz wrote: >> Ok, I understand your thoughts about the "big loop" and I agree. On the >> other side it is not easy to measure the progress of the dirty buffers >> because these buffers a created from another process at the same time we >> loop in vop_stdfsync(). I can explain from my tests, where I use the >> following loop on a gjournaled partition: >> >> while true; do >> cp -p bigfile bigfile.tmp >> rm bigfile >> mv bigfile.tmp bigfile >> done >> >> When g_journal_switcher starts vfs_write_suspend() immediately after the >> rm command has started to do his "rm stuff" (ufs_inactive, ffs_truncate, >> ffs_indirtrunc at different levels, ffs_blkfree, ...) the we must loop >> (that means wait) in vop_stdfsync() until the rm process has finished >> his work. A lot of locking overhead is needed for coordination. >> Returning from bufobj_wwait() we always see one left dirty buffer (very >> seldom two), that is not optimal. Therefore I have tried the following >> patch (instead of bumping maxretry): >> >> --- vfs_default.c.orig 2016-10-24 12:26:57.000000000 +0200 >> +++ vfs_default.c 2017-09-15 12:30:44.792274000 +0200 >> @@ -688,6 +688,8 @@ >> bremfree(bp); >> bawrite(bp); >> } >> + if( maxretry < 1000) >> + DELAY(waitns); >> BO_LOCK(bo); >> goto loop2; >> } >> >> with different values for waitns. If I run the testloop 5000 times on my >> testserver, the problem is triggered always round about 10 times. The >> results from several runs are given in the following table: >> >> waitns max time max loops >> ------------------------------- >> no DELAY 0,5 sec 8650 (maxres = 100000) >> 1000 0,2 sec 24 >> 10000 0,8 sec 3 >> 100000 7,2 sec 3 >> >> "time" means spent time in vop_stdfsync() measured from entry to return >> by a dtrace script. "loops" means the number of times "--maxretry" is >> executed. I am not sure if DELAY() is the best way to wait or if waiting >> has other drawbacks. Anyway with DELAY() it does not take more than five >> iterazions to finish. > > This is not explicitly stated in your message, but I suppose that the > vop_stdfsync() is called due to VOP_FSYNC(devvp, MNT_SUSPEND) call in > ffs_sync(). Am I right ? > > If yes, then the solution is most likely to continue looping in the > vop_stdfsync() until there is no dirty buffers or the mount point > mnt_secondary_writes counter is zero. The pauses trick you tried might > be still useful, e.g. after some threshold of the performed loop > iterations. > > Some problem with this suggestion is that vop_stdfsync(devvp) needs to > know that the vnode is devvp for some UFS mount. The struct cdev, > acessible as v_rdev, has the pointer to struct mount. You should be > accurate to not access freed or reused struct mount. I concur with Kostik's comments. It would be helpful if you could try out his suggestions and see if the produces a better result. Once you converge on a solution, I will ensure that it gets checked in. ~Kirk