From owner-freebsd-stable@FreeBSD.ORG Tue Mar 19 17:35:32 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 387D6C8C for ; Tue, 19 Mar 2013 17:35:32 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from qmta12.emeryville.ca.mail.comcast.net (qmta12.emeryville.ca.mail.comcast.net [IPv6:2001:558:fe2d:44:76:96:27:227]) by mx1.freebsd.org (Postfix) with ESMTP id 115458CC for ; Tue, 19 Mar 2013 17:35:32 +0000 (UTC) Received: from omta06.emeryville.ca.mail.comcast.net ([76.96.30.51]) by qmta12.emeryville.ca.mail.comcast.net with comcast id DSMu1l00216AWCUACVbXGg; Tue, 19 Mar 2013 17:35:31 +0000 Received: from koitsu.strangled.net ([67.180.84.87]) by omta06.emeryville.ca.mail.comcast.net with comcast id DVbW1l00e1t3BNj8SVbXLs; Tue, 19 Mar 2013 17:35:31 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id BB5C373A1C; Tue, 19 Mar 2013 10:35:30 -0700 (PDT) Date: Tue, 19 Mar 2013 10:35:30 -0700 From: Jeremy Chadwick To: Michael Landin Hostbaek Subject: Re: Core Dump / panic sleeping thread Message-ID: <20130319173530.GA72669@icarus.home.lan> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=q20121106; t=1363714531; bh=bJCQlSjjypg16V2QmQl1L0p/kMIBdUeYcgocJfVlRwQ=; h=Received:Received:Received:Date:From:To:Subject:Message-ID: MIME-Version:Content-Type; b=EeR9aoefWc4aV4gmyrg41a1pIbs7fDhweTib75OcMYFx2bXHYaGelacd5Q/v9FNuh dgqSuuvsAd0WoGInA4UX1vYK0FcmP4zt7Qv6ZDyRlmAy0CCxFRLN2MIwjymWYATD5e ITyDmDWdzYiWxr3sFzGz4otwh+qqwx9gAuQ6zd9yOXSpeZO2eTpFD9e0cw7vtZqE+6 +GW0h5ZW2NNv9oT9AJx9gMYPyp0V3tyHNDSxWz4Ou2gtapoHAksnyO438bQoTASY1u 0DWnB2q3OwQ6lT6c5u2cyRzHiC5Y5RAZROjyH4MPrcNiU8sTXwOvBEBXmthoW2YEV+ HjTAQ27cvthtQ== Cc: Rick Macklem , freebsd-stable@freebsd.org, John Baldwin X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Mar 2013 17:35:32 -0000 On Tue, Mar 19, 2013 at 06:18:06PM +0100, Michael Landin Hostbaek wrote: > Hi, > > I am running a FreeBSD 9.1-REL system with GENERIC kernel: > FreeBSD xxxxx 9.1-RELEASE FreeBSD 9.1-RELEASE #0: Fri Jan 4 12:28:48 CET 2013 root@xxxxx:/usr/obj/usr/src/sys/GENERIC amd64 > > > It is crashing a couple of times per week, without any real pattern. There are no hints in the syslog, and I only have the core debug to work from... > > It is a webserver, using a NFS mounted docroot (if it might help) - here's the backtrace: > > > This GDB was configured as "amd64-marcel-freebsd"... > > Unread portion of the kernel message buffer: > Sleeping thread (tid 100256, pid 85641) owns a non-sleepable lock > KDB: stack backtrace of thread 100256: > #0 0xffffffff808f2d46 at mi_switch+0x186 > #1 0xffffffff8092bb52 at sleepq_wait+0x42 > #2 0xffffffff808f34d6 at _sleep+0x376 > #3 0xffffffff80b4f3ae at vm_object_page_remove+0x2ce > #4 0xffffffff80b5ac7d at vnode_pager_setsize+0x17d > #5 0xffffffff8082102c at nfscl_loadattrcache+0x2cc > #6 0xffffffff80818d37 at nfs_getattr+0x287 > #7 0xffffffff8098f1c0 at vn_stat+0xb0 > #8 0xffffffff809869d9 at kern_statat_vnhook+0xf9 > #9 0xffffffff80986b55 at kern_statat+0x15 > #10 0xffffffff80986c1a at sys_lstat+0x2a > #11 0xffffffff80bd7ae6 at amd64_syscall+0x546 > #12 0xffffffff80bc3447 at Xfast_syscall+0xf7 > panic: sleeping thread > cpuid = 0 > KDB: stack backtrace: > #0 0xffffffff809208a6 at kdb_backtrace+0x66 > #1 0xffffffff808ea8be at panic+0x1ce > #2 0xffffffff8092ed22 at propagate_priority+0x1d2 > #3 0xffffffff8092fa4e at turnstile_wait+0x1be > #4 0xffffffff808d8d48 at _mtx_lock_sleep+0xd8 > #5 0xffffffff80820fa4 at nfscl_loadattrcache+0x244 > #6 0xffffffff8081758c at ncl_readrpc+0xac > #7 0xffffffff80824c45 at ncl_getpages+0x485 > #8 0xffffffff80b5aa0c at vnode_pager_getpages+0x9c > #9 0xffffffff80b3fc93 at vm_fault_hold+0x673 > #10 0xffffffff80b41cc3 at vm_fault+0x73 > #11 0xffffffff80bd84b4 at trap_pfault+0x124 > #12 0xffffffff80bd8c6c at trap+0x49c > #13 0xffffffff80bc315f at calltrap+0x8 > Uptime: 8d0h54m10s > Dumping 2381 out of 24547 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% > > Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from /boot/kernel/geom_mirror.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/geom_mirror.ko > Reading symbols from /boot/kernel/geom_stripe.ko...Reading symbols from /boot/kernel/geom_stripe.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/geom_stripe.ko > Reading symbols from /boot/kernel/if_em.ko...Reading symbols from /boot/kernel/if_em.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/if_em.ko > Reading symbols from /boot/kernel/linprocfs.ko...Reading symbols from /boot/kernel/linprocfs.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/linprocfs.ko > Reading symbols from /boot/kernel/linux.ko...Reading symbols from /boot/kernel/linux.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/linux.ko > #0 doadump (textdump=Variable "textdump" is not available. > ) at pcpu.h:224 > 224 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) bt > #0 doadump (textdump=Variable "textdump" is not available. > ) at pcpu.h:224 > #1 0xffffffff808ea3a1 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:448 > #2 0xffffffff808ea897 in panic (fmt=0x1
) at /usr/src/sys/kern/kern_shutdown.c:636 > #3 0xffffffff8092ed22 in propagate_priority (td=Variable "td" is not available. > ) at /usr/src/sys/kern/subr_turnstile.c:227 > #4 0xffffffff8092fa4e in turnstile_wait (ts=Variable "ts" is not available. > ) at /usr/src/sys/kern/subr_turnstile.c:743 > #5 0xffffffff808d8d48 in _mtx_lock_sleep (m=0xfffffe044a3c8238, tid=18446741888664231936, opts=Variable "opts" is not available. > ) > at /usr/src/sys/kern/kern_mutex.c:471 > #6 0xffffffff80820fa4 in nfscl_loadattrcache (vpp=Variable "vpp" is not available. > ) at /usr/src/sys/fs/nfsclient/nfs_clport.c:379 > #7 0xffffffff8081758c in ncl_readrpc (vp=0xfffffe044a6cd780, uiop=0xffffff86962fc650, cred=Variable "cred" is not available. > ) > at /usr/src/sys/fs/nfsclient/nfs_clvnops.c:1369 > #8 0xffffffff80824c45 in ncl_getpages (ap=0xffffff86962fc6f0) at /usr/src/sys/fs/nfsclient/nfs_clbio.c:171 > #9 0xffffffff80b5aa0c in vnode_pager_getpages (object=0xfffffe016aa16570, m=0xffffff86962fc770, count=Variable "count" is not available. > ) > at vnode_if.h:1154 > #10 0xffffffff80b3fc93 in vm_fault_hold (map=0xfffffe007f7e3188, vaddr=34366988288, fault_type=1 '\001', fault_flags=Variable "fault_flags" is not available. > ) > at vm_pager.h:128 > #11 0xffffffff80b41cc3 in vm_fault (map=0xfffffe007f7e3188, vaddr=34366988288, fault_type=Variable "fault_type" is not available. > ) > at /usr/src/sys/vm/vm_fault.c:229 > #12 0xffffffff80bd84b4 in trap_pfault (frame=0xffffff86962fcc40, usermode=1) at /usr/src/sys/amd64/amd64/trap.c:740 > #13 0xffffffff80bd8c6c in trap (frame=0xffffff86962fcc40) at /usr/src/sys/amd64/amd64/trap.c:358 > #14 0xffffffff80bc315f in calltrap () at /usr/src/sys/amd64/amd64/exception.S:228 > #15 0x0000000802091386 in ?? () > Previous frame inner to this frame (corrupt stack?) > (kgdb) > > > > Dump header from device /dev/mirror/gm0s1b > Architecture: amd64 > Architecture Version: 2 > Dump Length: 2496667648B (2381 MB) > Blocksize: 512 > Dumptime: Mon Mar 18 19:35:00 2013 > Hostname: xxxxxxxxx > Magic: FreeBSD Kernel Dump > Version String: FreeBSD 9.1-RELEASE #0: Fri Jan 4 12:28:48 CET 2013 > root@xxxxx:/usr/obj/usr/src/sys/GENERIC > Panic String: sleeping thread > Dump Parity: 826144189 > Bounds: 5 > Dump Status: good > > > > Any ideas? The kernel panic is happening in NFS-related code. Rick Macklem (and/or John Baldwin) should be able to help with this; I've CC'd both here. You're going to need to provide the following details: 1. Contents of /etc/rc.conf 2. Contents of /etc/sysctl.conf (if modified) 3. Contents of /etc/fstab 4. ifconfig -a 5. OS used by the NFS server, and all configuration details pertaining to that system You may also be asked to upgrade to 9.1-STABLE, as there may be fixes for whatever this is in base/stable/9 that are not in -RELEASE, but this is speculative on my part. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |