Date: Tue, 14 May 2002 17:19:44 -0700 From: Terry Lambert <tlambert2@mindspring.com> To: Richard Sharpe <rsharpe@ns.aus.com> Cc: freebsd-hackers@freebsd.org Subject: Re: File locking, closes and performance in a distributed file systemenv Message-ID: <3CE1A9A0.A15C42B9@mindspring.com> References: <Pine.LNX.4.33.0205150251310.3047-100000@ns.aus.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Richard Sharpe wrote: > I might be way off base here, but we have run into what looks like a > performance issue with locking and file closes. [ ... ] > This seems to mean that once a process locks a file, every close after > that will pay the penalty of calling the underlying vnode unlock call. In > a distributed file system, with a simple implementation, that could be an > RPC to the lock manager to implement. Yes. This is pretty much required by the POSIX locking semantics, which require that the first close remove all locks. Unfortunately, you can't know on a per process basis that there are no locks remaining on *any* vnode for a given process, so the overhead is sticky. > Now, there seems to be a few ways to migitate this: > > 1. Keep (more) state at the vnode layer that allows us to not issue a > network traversing unlock if the file was not locked. This means that any > process that has opened the file will have to issue the network traversing > unlock request once the flag is set on the vnode. > > 2. Place a flag in the struct file structure that keeps the state of any > locks on the file. This means that any processes that share the struct > (those related by fork) will need to issue unlock requests if one of them > locks the file. > > 3. Change a file descriptor table that hangs off the process structure so > that it includes state about whether or not this process has locked the > file. > > It seems that each of these reduces the performance penalty that processes > that might be sharing the file, but which have not locked the file, might > have to pay. > > Option 2 looks easy. > > Are there any comments? #3 is really unreasonable. It implies non-coelescing. I know that CIFS requires this, and so does NFSv4, so it's not an unreasonable thing do do eventually (historical behaviour can be maintained by removing all locks in the overlap region on an unlock, yielding logical coelescing). The amount of things that will need to be touched by this, though, means it's probably not worth doing now. In reality, for remote FS's, you want to assert the lock locally before transitting the network anyway, in case there is a local conflict, in which case you avoid propagating the request over the network. For union mounts of local and remote FS's, for which there is a local lock against the local FS by another process that doesn't respect the union (a legitimate thing to have happen), it's actually a requirement, since the remote system may promote or coelesce locks, and that means that there is no reverse process for a remote success followed by a local failure. This is basically a twist on #1: a) Assert the lock locally before asserting it remotely; if the assertion fails, then you have avoided a network operation which is doomed to failure (the RPC call you are trying to avoid is similar). b) When unlocking, verify that the lock exists locally before attempting to deassert it remotely. This means there there is still the same local overhead as there always was, but at least you avoid the RPC in the case where there are no outstanding locks that will be cleared by the call. I've actually wanted the VOP_ADVLOCK to be veto-based for going on 6 years now, to avoid precisely the type of problems your are now facing. If the upper layer code did local assertion on vnodes, and called the lower layer code only in the success cases, then the implementation would actually be done for you already. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3CE1A9A0.A15C42B9>