From owner-freebsd-current@FreeBSD.ORG Wed Aug 25 03:21:10 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DCD4D16A4CE for ; Wed, 25 Aug 2004 03:21:10 +0000 (GMT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6365B43D5D for ; Wed, 25 Aug 2004 03:21:10 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.11/8.12.11) with ESMTP id i7P3Iur6019622; Tue, 24 Aug 2004 23:18:56 -0400 (EDT) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i7P3ItDk019619; Tue, 24 Aug 2004 23:18:55 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Tue, 24 Aug 2004 23:18:55 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: Christian Brueffer In-Reply-To: <20040824203210.GA30220@unixpages.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: current@freebsd.org Subject: Re: panic: nfssvc_nfsd(): debug.mpsafenet=1 && Giant X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Aug 2004 03:21:11 -0000 On Tue, 24 Aug 2004, Christian Brueffer wrote: > I'm getting the following panic on 6-CURRENT as well as 5.3-BETA1 with > sources from today. It's easily reproducible on both, with > debug.mpsafenet=1. Hmm. Looks like something has acquired and failed to release Giant in the NFS server. It could be we're leaking Giant in an error case that didn't turn up in previous testing, but for some reason is more common in your environment. Unfortunately, as Giant can be acquired recursively, the "last acquired" information presented by WITNESS isn't useful to us. There are a couple of ways we could approach debugging this. I think the easiest might be the following: - Recompile your kernel with the following options: options KTR options KTR_COMPILE=(KTR_LOCK|KTR_PROC) options KTR_ENTRIES=16384 - At run-time, before triggering the crash, use sysctl to set the following settings: sysctl debug.ktr.cpumask=0xff sysctl debug.ktr.mask=`sysctl -n debug.ktr.compile` - When the crash occurs, use "show ktr" to list recent lock operations. Using your serial console, copy and paste a few pages of locking operations into an e-mail, probably until you hit the next context switch (mi_switch: new thread). You'll notice that the entries are sorted by event id, which generally results in reverse-chronological order with most recent event earliest. We'd like to look at all acquisitions of Giant and releases of Giant since that context switch. There should be one or more acquires than releases in the event stream, and we'd like to figure out which is the one not associated with a matching unlock. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Principal Research Scientist, McAfee Research > > Crashdumps and debugging kernels available for both. > > panic: nfssvc_nfsd(): debug.mpsafenet=1 && Giant > cpuid = 0 > KDB: enter: panic > [thread 100102] > Stopped at kdb_enter+0x2b: nop > db> tr > kdb_enter(c06e619b) at kdb_enter+0x2b > panic(c06fbcc1,c0515fd4,0,2,2) at panic+0x125 > nfssvc_nfsd(c1c37420,c06e5488,121,c202b22c,c202b1c0) at > nfssvc_nfsd+0x77e > nfssvc(c1c37420,d8908d14,2,1,292) at nfssvc+0x1ac > syscall(2f,2f,2f,bfbfecd8,0) at syscall+0x22b > Xint0x80_syscall() at Xint0x80_syscall+0x1f > --- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x280d02ef, esp = > 0xbfbfe92c, ebp = 0xbfbfe948 --- > db> show locks > exclusive sleep mutex nfsd_mtx r = 0 (0xc078dee0) locked @ > /usr/home/build/src/sys/nfsserver/nfs_syscalls.c: > 510 > exclusive sleep mutex Giant r = 1 (0xc074c100) locked @ > /usr/home/build/src/sys/nfsserver/nfs_serv.c:2693 > db> > > > - Christian > > -- > Christian Brueffer chris@unixpages.org brueffer@FreeBSD.org > GPG Key: http://people.freebsd.org/~brueffer/brueffer.key.asc > GPG Fingerprint: A5C8 2099 19FF AACA F41B B29B 6C76 178C A0ED 982D >