Date: Wed, 20 Sep 1995 10:11:52 -0700 (MST) From: Terry Lambert <terry@lambert.org> To: mpp@mpp.minn.net (Mike Pritchard) Cc: terry@lambert.org, nate@rocky.sri.MT.net, davidg@root.com, hackers@freefall.freebsd.org Subject: Re: Coding style ( was Re: why is this not a bug in namei?) Message-ID: <199509201711.KAA01075@phaeton.artisoft.com> In-Reply-To: <199509200623.BAA08946@mpp.minn.net> from "Mike Pritchard" at Sep 20, 95 01:23:12 am
next in thread | previous in thread | raw e-mail | index | archive | help
[ ... order edited for purposes of discussion ... ]
[ ... multithreading kernels ... ]
> XXX_LOCK();
> ...
> if (error_condition) {
> XXX_UNLOCK();
> return (EWHATEVER);
> }
> ...
> XXX_UNLOCK();
>
> Rather than:
>
> XXX_LOCK();
> have_xxx_lock = 1;
> ...
> if (error_condition) {
> error = EWHATEVER;
> goto done;
> }
> ...
> XXX_UNLOCK():
> have_xxx_lock = 0;
> done:
> if (have_xxx_lock)
> XXX_UNLOCK():
> return (error);
Why not:
XXX_LOCK();
...
if (error_condition) {
error = EWHATEVER;
goto err_with_lock_held;
}
...
err_with_lock_held:
XXX_UNLOCK():
err:
return (error);
Or better yet:
XXX_LOCK();
...
if (error_condition) {
error = EWHATEVER;
} else {
...
}
XXX_UNLOCK():
err:
return (error);
I don't understand the need for the local lock state variable in your
second example.
> I used to work for Cray Research back when they were working on
> getting their multithreading kernel working and ready for release.
> Depending on the function, single entry/exit functions could be a real
> pain the the ass. For efficiency reasons, we would lock either the
> entire array, individual elements of the array, or even individual
> variables in some cases.
Yes. This is variable granularity locking. Typically it applies to
memory mapped devices, interrupt handler instances, and global buffers.
The common mistake with this schema is that multithreading is not
multiprocessing, and the lock implementation generally does not include
computation of transitive closure over the directed graph, so you
end up with deadly-embrace deadlocks inherent in the use of the lock
hierarchy by multple processors.
The global buffer case is, in most implementations, a design flaw. The
SVR4/Solaris kernel and slab allocator have this flaw, in that they must
go to a global mutex in order to allocate memory.
The Sequent kernel bypasses this by using a zone allocator and defining
per processor zones. Only when a zone needs refilled from the global
memory pool (high water mark) or the zone usage has decreased far
enough that the zone is overutilizing resources (low water mark) is a
global mutex held to refill/drain the zone's free page pool.
This is described in some detail in a book that I recently completed a
techinical review on for Prentice Hall (earlier this year):
"UNIX Internals: the new frontier"
Uresh Vahalia
ISBN 0-13-101908-2
Prentice Hall, 1995
A good (and large) book. It actually compares various aspects of
different UNIX implementations. Like memory allocation policy, file
system implementation, paging, threading, etc.. As far as I know,
it's the only book on comparative UNIX architecture that exists.
> In some routines, the single entry/exit idea just didn't fly. Either
> you held the lock for much longer than you needed to,
I'd argue that this was an error in judgement in the programmer assigning
interface boundries... Only in the rarest of cases should a lock be
held over a function call boundry (pushing a stack frame on a SPARC is
not a speedy operation). Allocation and deallocation of the locks
themselves and system initialization are the only such cases that
spring immediately to mind.
> or you wound up having to keep track of which locks you had
> and then release them on exit.
I'd argue that this was an error in the implementation of the lock
manager. Having the lock hierarchy implied by stack unwind state is an
error in design of a hierarchical lock management system (or simply an
error, when using a non-hierarchical lock management system to imply
hierarchy). It *must* be possible to traverse the directed hierarchy
graph to compute transitive closure to implement deadlock avoidance.
> When you have 16 - 64+ processors possibly wating for a lock, you never
> wanted to keep a lock any longer than needed.
Agreed. When I was working on the file system and the system open file
table management in the SMP version of SVR4, I faced many of these same
issues. I am not a novice. Unfortunately, by the time I got to that area,
the lock model was already broken and being held in place with red tape.
Terry Lambert
terry@lambert.org
---
Any opinions in this posting are my own and not those of my present
or previous employers.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199509201711.KAA01075>