From owner-freebsd-current@FreeBSD.ORG Thu Jun 26 07:57:42 2003 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C2F8F37B404 for ; Thu, 26 Jun 2003 07:57:42 -0700 (PDT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 734CF43FA3 for ; Thu, 26 Jun 2003 07:57:41 -0700 (PDT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.9/8.12.9) with ESMTP id h5QEvGKJ064495; Thu, 26 Jun 2003 10:57:16 -0400 (EDT) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)h5QEvGE9064492; Thu, 26 Jun 2003 10:57:16 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Thu, 26 Jun 2003 10:57:16 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: Socketd In-Reply-To: <20030626151113.28866ac9.db@traceroute.dk> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: current@freebsd.org Subject: Re: FreeBSD 5.1-Release freezes X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Jun 2003 14:57:43 -0000 On Thu, 26 Jun 2003, Socketd wrote: > On Wed, 25 Jun 2003 16:44:56 -0400 (EDT) > Robert Watson wrote: > > > The code most likely to cause a memory leak in the MAC Framework is > > the label management code, since that's the only code that really does > > much in the way of memory allocaiton. Try compiling options MAC_DEBUG > > into your kernel, which causes the MAC Framework to track the number > > of labels it has allocated/free'd in a series of variables: > > > > static unsigned int nmacmbufs, nmaccreds, nmacifnets, nmacbpfdescs, > > nmacsockets, nmacmounts, nmactemp, nmacvnodes, nmacdevfsdirents, > > nmacipqs, nmacpipes, nmacprocs; > > After running a few commands, ssh logins and loading mac_seeotheruids we > now have: <...> > Loading mac_seeotheruids made vnodes increase a little, but it has (very > slowly) been increasing and as you can see is now at 524 (don't know if > this means anything). You can compare the number of labeled vnodes with the number of actual vnodes by looking at the vfs.numvnodes sysctl value -- ideally they should be the same. The fact that the problem went away with the addition of the debugging code is a little concerning. Here's what I'd like you to try, if you can: set up a serial console to another machine. Compile the kernel with options DDB, but without options MAC_DEBUG. Do whatever it is you do that triggers the hang. On the serial console, send a serial break and see if you can get into DDB. If you can, send the output of the following commands: ps show lockedvnods I may then ask you to run the trace command on various processes/threads, and use "show locks" to show which threads hold which locks (assuming you have WITNESS in the kernel, which also changes the timing). On the whole, enabling MAC doesn't substantially change the locking, but there are a couple of edge cases where we grab locks that were not previously grabbed. For example, we now grab vnode locks when polling a vnode so we can do a label check on the vnode; this generates WITNESS warnings as we hold the file descriptor lock over the call to VOP_POLL() (ouch). If we wend up concluding this is a locking problem, it might be that I send you some patches to modify the locking and see if we can get rid of the problem. thanks! Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Network Associates Laboratories