Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 19 Mar 2013 10:35:30 -0700
From:      Jeremy Chadwick <jdc@koitsu.org>
To:        Michael Landin Hostbaek <mich@freebsd.org>
Cc:        Rick Macklem <rmacklem@uoguelph.ca>, freebsd-stable@freebsd.org, John Baldwin <jhb@freebsd.org>
Subject:   Re: Core Dump / panic sleeping thread
Message-ID:  <20130319173530.GA72669@icarus.home.lan>
In-Reply-To: <A6D4A877-2DBA-4E9D-996D-2E34834BF60E@freebsd.org>
References:  <A6D4A877-2DBA-4E9D-996D-2E34834BF60E@freebsd.org>

next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Mar 19, 2013 at 06:18:06PM +0100, Michael Landin Hostbaek wrote:
> Hi, 
> 
> I am running a FreeBSD 9.1-REL system with GENERIC kernel:
> FreeBSD xxxxx 9.1-RELEASE FreeBSD 9.1-RELEASE #0: Fri Jan  4 12:28:48 CET 2013     root@xxxxx:/usr/obj/usr/src/sys/GENERIC  amd64
> 
> 
> It is crashing a couple of times per week, without any real pattern. There are no hints in the syslog, and I only have the core debug to work from...  
> 
> It is a webserver, using a NFS mounted docroot (if it might help) - here's the backtrace:
> 
> <snip>
> This GDB was configured as "amd64-marcel-freebsd"...
> 
> Unread portion of the kernel message buffer:
> Sleeping thread (tid 100256, pid 85641) owns a non-sleepable lock
> KDB: stack backtrace of thread 100256:
> #0 0xffffffff808f2d46 at mi_switch+0x186
> #1 0xffffffff8092bb52 at sleepq_wait+0x42
> #2 0xffffffff808f34d6 at _sleep+0x376
> #3 0xffffffff80b4f3ae at vm_object_page_remove+0x2ce
> #4 0xffffffff80b5ac7d at vnode_pager_setsize+0x17d
> #5 0xffffffff8082102c at nfscl_loadattrcache+0x2cc
> #6 0xffffffff80818d37 at nfs_getattr+0x287
> #7 0xffffffff8098f1c0 at vn_stat+0xb0
> #8 0xffffffff809869d9 at kern_statat_vnhook+0xf9
> #9 0xffffffff80986b55 at kern_statat+0x15
> #10 0xffffffff80986c1a at sys_lstat+0x2a
> #11 0xffffffff80bd7ae6 at amd64_syscall+0x546
> #12 0xffffffff80bc3447 at Xfast_syscall+0xf7
> panic: sleeping thread
> cpuid = 0
> KDB: stack backtrace:
> #0 0xffffffff809208a6 at kdb_backtrace+0x66
> #1 0xffffffff808ea8be at panic+0x1ce
> #2 0xffffffff8092ed22 at propagate_priority+0x1d2
> #3 0xffffffff8092fa4e at turnstile_wait+0x1be
> #4 0xffffffff808d8d48 at _mtx_lock_sleep+0xd8
> #5 0xffffffff80820fa4 at nfscl_loadattrcache+0x244
> #6 0xffffffff8081758c at ncl_readrpc+0xac
> #7 0xffffffff80824c45 at ncl_getpages+0x485
> #8 0xffffffff80b5aa0c at vnode_pager_getpages+0x9c
> #9 0xffffffff80b3fc93 at vm_fault_hold+0x673
> #10 0xffffffff80b41cc3 at vm_fault+0x73
> #11 0xffffffff80bd84b4 at trap_pfault+0x124
> #12 0xffffffff80bd8c6c at trap+0x49c
> #13 0xffffffff80bc315f at calltrap+0x8
> Uptime: 8d0h54m10s
> Dumping 2381 out of 24547 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
> 
> Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from /boot/kernel/geom_mirror.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/geom_mirror.ko
> Reading symbols from /boot/kernel/geom_stripe.ko...Reading symbols from /boot/kernel/geom_stripe.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/geom_stripe.ko
> Reading symbols from /boot/kernel/if_em.ko...Reading symbols from /boot/kernel/if_em.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/if_em.ko
> Reading symbols from /boot/kernel/linprocfs.ko...Reading symbols from /boot/kernel/linprocfs.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/linprocfs.ko
> Reading symbols from /boot/kernel/linux.ko...Reading symbols from /boot/kernel/linux.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/linux.ko
> #0  doadump (textdump=Variable "textdump" is not available.
> ) at pcpu.h:224
> 224	pcpu.h: No such file or directory.
> 	in pcpu.h
> (kgdb) bt
> #0  doadump (textdump=Variable "textdump" is not available.
> ) at pcpu.h:224
> #1  0xffffffff808ea3a1 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:448
> #2  0xffffffff808ea897 in panic (fmt=0x1 <Address 0x1 out of bounds>) at /usr/src/sys/kern/kern_shutdown.c:636
> #3  0xffffffff8092ed22 in propagate_priority (td=Variable "td" is not available.
> ) at /usr/src/sys/kern/subr_turnstile.c:227
> #4  0xffffffff8092fa4e in turnstile_wait (ts=Variable "ts" is not available.
> ) at /usr/src/sys/kern/subr_turnstile.c:743
> #5  0xffffffff808d8d48 in _mtx_lock_sleep (m=0xfffffe044a3c8238, tid=18446741888664231936, opts=Variable "opts" is not available.
> )
>     at /usr/src/sys/kern/kern_mutex.c:471
> #6  0xffffffff80820fa4 in nfscl_loadattrcache (vpp=Variable "vpp" is not available.
> ) at /usr/src/sys/fs/nfsclient/nfs_clport.c:379
> #7  0xffffffff8081758c in ncl_readrpc (vp=0xfffffe044a6cd780, uiop=0xffffff86962fc650, cred=Variable "cred" is not available.
> )
>     at /usr/src/sys/fs/nfsclient/nfs_clvnops.c:1369
> #8  0xffffffff80824c45 in ncl_getpages (ap=0xffffff86962fc6f0) at /usr/src/sys/fs/nfsclient/nfs_clbio.c:171
> #9  0xffffffff80b5aa0c in vnode_pager_getpages (object=0xfffffe016aa16570, m=0xffffff86962fc770, count=Variable "count" is not available.
> )
>     at vnode_if.h:1154
> #10 0xffffffff80b3fc93 in vm_fault_hold (map=0xfffffe007f7e3188, vaddr=34366988288, fault_type=1 '\001', fault_flags=Variable "fault_flags" is not available.
> )
>     at vm_pager.h:128
> #11 0xffffffff80b41cc3 in vm_fault (map=0xfffffe007f7e3188, vaddr=34366988288, fault_type=Variable "fault_type" is not available.
> )
>     at /usr/src/sys/vm/vm_fault.c:229
> #12 0xffffffff80bd84b4 in trap_pfault (frame=0xffffff86962fcc40, usermode=1) at /usr/src/sys/amd64/amd64/trap.c:740
> #13 0xffffffff80bd8c6c in trap (frame=0xffffff86962fcc40) at /usr/src/sys/amd64/amd64/trap.c:358
> #14 0xffffffff80bc315f in calltrap () at /usr/src/sys/amd64/amd64/exception.S:228
> #15 0x0000000802091386 in ?? ()
> Previous frame inner to this frame (corrupt stack?)
> (kgdb) 
> </snip>
> 
> 
> Dump header from device /dev/mirror/gm0s1b
>   Architecture: amd64
>   Architecture Version: 2
>   Dump Length: 2496667648B (2381 MB)
>   Blocksize: 512
>   Dumptime: Mon Mar 18 19:35:00 2013
>   Hostname: xxxxxxxxx
>   Magic: FreeBSD Kernel Dump
>   Version String: FreeBSD 9.1-RELEASE #0: Fri Jan  4 12:28:48 CET 2013
>     root@xxxxx:/usr/obj/usr/src/sys/GENERIC
>   Panic String: sleeping thread
>   Dump Parity: 826144189
>   Bounds: 5
>   Dump Status: good
> 
> 
> 
> Any ideas? 

The kernel panic is happening in NFS-related code.  Rick Macklem (and/or
John Baldwin) should be able to help with this; I've CC'd both here.

You're going to need to provide the following details:

1. Contents of /etc/rc.conf
2. Contents of /etc/sysctl.conf (if modified)
3. Contents of /etc/fstab
4. ifconfig -a
5. OS used by the NFS server, and all configuration details pertaining
to that system

You may also be asked to upgrade to 9.1-STABLE, as there may be fixes
for whatever this is in base/stable/9 that are not in -RELEASE, but this
is speculative on my part.

-- 
| Jeremy Chadwick                                   jdc@koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Mountain View, CA, US                                            |
| Making life hard for others since 1977.             PGP 4BD6C0CB |



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130319173530.GA72669>