Date: Thu, 15 Mar 2001 12:33:39 -0800 (PST) From: John Baldwin <jhb@FreeBSD.org> To: Tony Griffiths <tonyg@OntheNet.com.au> Cc: alpha@FreeBSD.org, Andrew Gallatin <gallatin@cs.duke.edu> Subject: Re: Deadlocks, whee! Message-ID: <XFMail.010315123339.jhb@FreeBSD.org> In-Reply-To: <Pine.BSF.4.30.0103151055520.84004-100000@lancia.onthenet.com.au>
next in thread | previous in thread | raw e-mail | index | archive | help
On 15-Mar-01 Tony Griffiths wrote: > On Wed, 14 Mar 2001, John Baldwin wrote: > >> >> On 15-Mar-01 Andrew Gallatin wrote: >> > >> > John Baldwin writes: >> > > Hi all, >> > > >> > > I managed to deadlock my alpha yesterday with a -j 4 buildworld. >> > Previously it >> > > would die when it trapped with a raised IPL as a blockable mtx_lock() >> > > of >> > lockmgr >> > > in trap(). I'm not sure if these two things are related or not. I'll >> > > try >> > a >> > > normal world without -j X today to see if it fairs better. Just FYI >> > > for >> > those >> > > running current that heavy load may deadlock right now. :( >> > >> > The machine is really deadlocked, or just one process is wedged and >> > the buildworld stalled? >> >> Well, no messages on the console, no ddb (I have vidconsole), no pings, etc. >> So interrupts aren't getting through, or if they are their threads aren't >> running, and since I use preemption on this alpha, that is very, very >> unlikely. >> I'm assuming it is genuinely deadlocked or possibly spinning somewhere with >> a >> raised IPL. > > Looks like a "deadlock" to me! Unfortunately. :-/ > Actually, I'm surprised that the 'fine-grained' SMP project in FreeBSD has > managed to get as far as it has without implementing some form of "sanity" > checking. I worked for DEC (Digital Equipment Corp) in the Networking Group > at > the time Ultrix (BSD 4.2/4.3) was doing fine-grained SMP and we had the > following sanity checks in the locking code as an aide to maintaining our own > sanity! ;-) Actually, we do have this sanity checking. We have taken code from BSD/OS that builds dynamic order lists of mutexes on the fly (for sleep mutexes, spin mutexes have to have their order statically defined) and warns of lock order reversals, sleeping with mutexes held, etc. It has proved valuable already and will continue to do so in the future. Currently I'm working to generalize it so that it will work with our reader/writer locks to ensure that a consistent lock order is maintained between them and mutexes as well as making it possible to build the spin lock order list on the fly, though I think I will still require the spinlock list to be static just because very few spinlocks should be used and only in certain cases. We also have a kernel event tracing facility that logs events of lock acquires/releases to a circular log buffer that can be examined from the in kernel debugger or via gdb for post-mortem crash dump analysis. > 1) Logging of request/release calls > 2) Lock hierarchy (ie. take-out ordering) > 3) Spin-lock timeout (ie. panic() after 5000000 failed attempts to gain lock) We also have this in that the spin mutexes will panic if they spin for more than about 3 seconds. > 4) something else that I can't remember 'cause it was too long ago!!! > > The lock hierarchy was a BIG WIN in detecting/preventing deadlock conditions > since it forced an order in lock acquisition although it didn't stop > deadlocks > from occurring when the locks were at the same level. The spin count > exceeded > picked those up. Yep, we've already found that the original VFS code we inherited from 4.4BSD contained a few lock order reversals that could lead to potential deadlocks. Fixing those will be part of fixing VFS, which will be a non-trivial task I'm afraid. :-/ > We also found a few problems on tri/quad-cpu systems that didn't occur on > dual-cpu systems. So far my test quad box here at work hasn't hit any weird bugs that the dual testboxes I have didn't eventually run into. It does run into some things quicker however. > Of course the amount of checking was a compile-time setting so that > production > code didn't suffer too badly. Yep, same here. You can also disable all or part of its operation by setting appropriate environment variables in the kernel loader. (Well, the new witness can be entirely disabled from the loader, the old one currently in the tree can only have its spinlock checking disabled.) > > We learnt a lot of hard lessons on Ultrix, the main one being that we were > too > ambitious in trying for a VERY FINE-GRAINED locking strategy (especially in > the > networking code) than was warrented by any possible payback. Our OSF/Tru-64 > implementation was much cleaner with pretty much a single lock at each layer > of > the network code (eg. socket, tcp/ip, driver). The locking hierarchy caused > a > few problems between the socket layer and transport but we could get around > by > using reference counts on objects that needed to stick around even when there > was no 'lock' on them! Hmm. Currently we have per-process stucture locks and I think per-mbuf locks. I'm aware that we can end up with too many locks, but I don't have the experience yet to make those judgement calls. I have a feeling we will be trying it both ways for some things until we have a good basis to make these decisions on. > Hope you have more 'fun' then we did (NOT) ... Heh, we are already having _loads_ of fun. :) > Tony -- John Baldwin <jhb@FreeBSD.org> -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-alpha" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?XFMail.010315123339.jhb>