Date: Mon, 28 Jan 2008 17:59:16 -0800 From: Jos Backus <jos@catnook.com> To: Robert Watson <rwatson@FreeBSD.org> Cc: freebsd-current@freebsd.org Subject: Re: panic: Lock so_rcv_sx not exclusively locked Message-ID: <20080129015916.GA5413@lizzy.catnook.local> In-Reply-To: <20080128204730.I56811@fledge.watson.org> References: <20080128180840.GA9582@lizzy.catnook.local> <20080128204730.I56811@fledge.watson.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--BOKacYhQ+x31HxR3
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
On Mon, Jan 28, 2008 at 08:49:41PM +0000, Robert Watson wrote:
> On Mon, 28 Jan 2008, Jos Backus wrote:
>
>> Recent -current kernels can reliably be made to panic by killing
>> npviewer.bin. Right before killing, npviewer.bin shows up like this in
>> top:
>>
>> 6288 jos 1 45 5 62844K 36788K so_rcv 0:08 0.00% npviewer.bin
>
> Is this a multithreaded app? If so, could you look and see if you can find
> the other threads in npviewer.bin in the kernel (info thread, etc) and get
> stack traces for those threads as well?
It's a multithreaded Linux app, installed through ports; see the attached
file. Please let me know if you'd like me to gather more information.
Thanks!
--
Jos Backus
jos at catnook.com
--BOKacYhQ+x31HxR3
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="npviewer.bin-panic.txt"
lizzy:~# pkg_which /usr/local/lib/nspluginwrapper/i386/linux/npviewer.bin
nspluginwrapper-0.9.91.5
lizzy:~% ldd /usr/local/lib/nspluginwrapper/i386/linux/npviewer.bin
/usr/local/lib/nspluginwrapper/i386/linux/npviewer.bin:
libgtk-x11-2.0.so.0 => /usr/lib/libgtk-x11-2.0.so.0 (0x2807e000)
libgdk-x11-2.0.so.0 => /usr/lib/libgdk-x11-2.0.so.0 (0x28376000)
libgobject-2.0.so.0 => /usr/lib/libgobject-2.0.so.0 (0x283fa000)
libdl.so.2 => /lib/libdl.so.2 (0x28436000)
libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0x2843a000)
libX11.so.6 => /usr/X11R6/lib/libX11.so.6 (0x284c3000)
libXt.so.6 => /usr/X11R6/lib/libXt.so.6 (0x28596000)
libpthread.so.0 => /lib/obsolete/linuxthreads/libpthread.so.0 (0x285ec000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2863f000)
libc.so.6 => /lib/obsolete/linuxthreads/libc.so.6 (0x28649000)
libgdk_pixbuf-2.0.so.0 => /usr/lib/libgdk_pixbuf-2.0.so.0 (0x28768000)
libpangoxft-1.0.so.0 => /usr/lib/libpangoxft-1.0.so.0 (0x2877e000)
libpangox-1.0.so.0 => /usr/lib/libpangox-1.0.so.0 (0x28785000)
libpango-1.0.so.0 => /usr/lib/libpango-1.0.so.0 (0x28791000)
libatk-1.0.so.0 => /usr/lib/libatk-1.0.so.0 (0x287cc000)
libgmodule-2.0.so.0 => /usr/lib/libgmodule-2.0.so.0 (0x287e3000)
libm.so.6 => /lib/obsolete/linuxthreads/libm.so.6 (0x287e7000)
libXrandr.so.2 => /usr/X11R6/lib/libXrandr.so.2 (0x2880d000)
libXi.so.6 => /usr/X11R6/lib/libXi.so.6 (0x28812000)
libXinerama.so.1 => /usr/X11R6/lib/libXinerama.so.1 (0x2881a000)
libXft.so.2 => /usr/X11R6/lib/libXft.so.2 (0x2881d000)
libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0x28830000)
libfontconfig.so.1 => /usr/lib/libfontconfig.so.1 (0x28898000)
libXfixes.so.3 => /usr/X11R6/lib/libXfixes.so.3 (0x288c2000)
libXcursor.so.1 => /usr/X11R6/lib/libXcursor.so.1 (0x288c7000)
libXrender.so.1 => /usr/X11R6/lib/libXrender.so.1 (0x288d2000)
libXext.so.6 => /usr/X11R6/lib/libXext.so.6 (0x288da000)
/lib/ld-linux.so.2 (0x2805f000)
libSM.so.6 => /usr/X11R6/lib/libSM.so.6 (0x288e9000)
libICE.so.6 => /usr/X11R6/lib/libICE.so.6 (0x288f2000)
libpangoft2-1.0.so.0 => /usr/lib/libpangoft2-1.0.so.0 (0x2890c000)
libexpat.so.0 => /usr/lib/libexpat.so.0 (0x28934000)
libz.so.1 => /usr/lib/libz.so.1 (0x28953000)
lizzy:~%
191 Thread 100197 (PID=3595: npviewer.bin) sched_switch (td=0xc5092440, newtd=Variable "newtd" is not available.
190 Thread 100201 (PID=3594: npviewer.bin) sched_switch (td=0xc5092aa0, newtd=Variable "newtd" is not available.
189 Thread 100200 (PID=3592: npviewer.bin) sched_switch (td=0xc5092880, newtd=Variable "newtd" is not available.
188 Thread 100183 (PID=3591: npviewer.bin) sched_switch (td=0xc5028440, newtd=Variable "newtd" is not available.
187 Thread 100163 (PID=3588: npviewer.bin) sched_switch (td=0xc4faf880, newtd=Variable "newtd" is not available.
* 186 Thread 100134 (PID=3561: npviewer.bin) doadump () at pcpu.h:195
(kgdb) thread
[Current thread is 186 (Thread 100134)]
(kgdb) bt
#0 doadump () at pcpu.h:195
#1 0xc05435ef in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:417
#2 0xc0543867 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:571
#3 0xc054a427 in _sx_assert (sx=0xc4fb89a0, what=4, file=0xc06d49ca "/usr/src/sys/kern/uipc_sockbuf.c", line=157) at /usr/src/sys/kern/kern_sx.c:931
#4 0xc054a8da in _sx_xunlock (sx=0xc4fb89a0, file=0xc06d49ca "/usr/src/sys/kern/uipc_sockbuf.c", line=157) at /usr/src/sys/kern/kern_sx.c:316
#5 0xc0595444 in sbunlock (sb=0xc4fb8968) at /usr/src/sys/kern/uipc_sockbuf.c:157
#6 0xc0598245 in sorflush (so=0xc4fb8918) at /usr/src/sys/kern/uipc_socket.c:1902
#7 0xc05982df in soshutdown (so=0xc4fb8918, how=2) at /usr/src/sys/kern/uipc_socket.c:1866
#8 0xc059ce86 in shutdown (td=0xc4b31880, uap=0xe7219c58) at /usr/src/sys/kern/uipc_syscalls.c:1248
#9 0xc07ffa80 in ?? ()
#10 0xc4b31880 in ?? ()
#11 0xe7219c58 in ?? ()
#12 0x00000008 in ?? ()
#13 0x00000000 in ?? ()
#14 0x00000000 in ?? ()
#15 0xc103d800 in ?? ()
#16 0x00000c1e in ?? ()
#17 0x00000000 in ?? ()
#18 0xc4e25488 in ?? ()
#19 0xc5007e9c in ?? ()
#20 0x00000000 in ?? ()
#21 0x00000000 in ?? ()
#22 0xe7219c72 in ?? ()
#23 0x00000004 in ?? ()
#24 0x80000000 in ?? ()
#25 0xe7219c8c in ?? ()
#26 0xc06a1a44 in __qdivrem (uq=Unhandled dwarf expression opcode 0x93
) at /usr/src/sys/libkern/qdivrem.c:186
#27 0xc0697b93 in syscall (frame=0xe7219d38) at /usr/src/sys/i386/i386/trap.c:1034
#28 0xc06818b0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:203
#29 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) thread 187
[Switching to thread 187 (Thread 100163)]
#0 sched_switch (td=0xc4faf880, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1905
1905 cpuid = PCPU_GET(cpuid);
(kgdb) bt
#0 sched_switch (td=0xc4faf880, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1905
#1 0xc054b6d3 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2 0xc05720a0 in sleepq_switch (wchan=0xc46caaa8) at /usr/src/sys/kern/subr_sleepqueue.c:478
#3 0xc05722ee in sleepq_catch_signals (wchan=0xc46caaa8) at /usr/src/sys/kern/subr_sleepqueue.c:414
#4 0xc0572a77 in sleepq_timedwait_sig (wchan=0xc46caaa8) at /usr/src/sys/kern/subr_sleepqueue.c:612
#5 0xc050ffc8 in _cv_timedwait_sig (cvp=0xc46caaa8, lock=0xc46caa90, timo=2001) at /usr/src/sys/kern/kern_condvar.c:369
#6 0xc05787ca in seltdwait (td=Variable "td" is not available.
) at /usr/src/sys/kern/sys_generic.c:1416
#7 0xc0578d7a in poll (td=0xc4faf880, uap=0xe728ccfc) at /usr/src/sys/kern/sys_generic.c:1060
#8 0xc0697b93 in syscall (frame=0xe728cd38) at /usr/src/sys/i386/i386/trap.c:1034
#9 0xc06818b0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:203
#10 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) thread 188
[Switching to thread 188 (Thread 100183)]
#0 sched_switch (td=0xc5028440, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1905
1905 cpuid = PCPU_GET(cpuid);
(kgdb) bt
#0 sched_switch (td=0xc5028440, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1905
#1 0xc054b6d3 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2 0xc05720a0 in sleepq_switch (wchan=0xc073b664) at /usr/src/sys/kern/subr_sleepqueue.c:478
#3 0xc05722ee in sleepq_catch_signals (wchan=0xc073b664) at /usr/src/sys/kern/subr_sleepqueue.c:414
#4 0xc0572a77 in sleepq_timedwait_sig (wchan=0xc073b664) at /usr/src/sys/kern/subr_sleepqueue.c:612
#5 0xc054bb91 in _sleep (ident=0xc073b664, lock=0x0, priority=348, wmesg=0xc06ce3dc "nanslp", timo=21) at /usr/src/sys/kern/kern_synch.c:222
#6 0xc0552891 in kern_nanosleep (td=0xc5028440, rqt=0xe72ddc5c, rmt=0x0) at /usr/src/sys/kern/kern_time.c:379
#7 0xc0802dd3 in ?? ()
#8 0xc5028440 in ?? ()
#9 0xe72ddc5c in ?? ()
#10 0x00000000 in ?? ()
#11 0xe72ddcfc in ?? ()
#12 0xbefffa28 in ?? ()
#13 0x00000000 in ?? ()
#14 0x01312d00 in ?? ()
#15 0xc5028440 in ?? ()
#16 0x0020dd38 in ?? ()
#17 0x00000000 in ?? ()
#18 0x01312d00 in ?? ()
#19 0xbefff9f0 in ?? ()
#20 0xc5028440 in ?? ()
#21 0xc4fa0804 in ?? ()
#22 0xe72ddd2c in ?? ()
#23 0xc0697b93 in syscall (frame=0xc5028440) at /usr/src/sys/i386/i386/trap.c:1034
Previous frame identical to this frame (corrupt stack?)
(kgdb) thread 189
[Switching to thread 189 (Thread 100200)]
#0 sched_switch (td=0xc5092880, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1905
1905 cpuid = PCPU_GET(cpuid);
(kgdb) bt
#0 sched_switch (td=0xc5092880, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1905
#1 0xc054b6d3 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2 0xc05720a0 in sleepq_switch (wchan=0xc4fb89d0) at /usr/src/sys/kern/subr_sleepqueue.c:478
#3 0xc05722ee in sleepq_catch_signals (wchan=0xc4fb89d0) at /usr/src/sys/kern/subr_sleepqueue.c:414
#4 0xc0572b84 in sleepq_wait_sig (wchan=0xc4fb89d0) at /usr/src/sys/kern/subr_sleepqueue.c:575
#5 0xc054bbb9 in _sleep (ident=0xc4fb89d0, lock=0xc4fb8988, priority=344, wmesg=0xc06d4a16 "sbwait", timo=0) at /usr/src/sys/kern/kern_synch.c:226
#6 0xc0595556 in sbwait (sb=0xc4fb8968) at /usr/src/sys/kern/uipc_sockbuf.c:131
#7 0xc059a69f in soreceive_generic (so=0xc4fb8918, psa=0xe7313b34, uio=0xe7313b40, mp0=0x0, controlp=0x0, flagsp=0xe7313bc4)
at /usr/src/sys/kern/uipc_socket.c:1485
#8 0xc059670d in soreceive (so=0xc4fb8918, psa=0xe7313b34, uio=0xe7313b40, mp0=0x0, controlp=0x0, flagsp=0xe7313bc4)
at /usr/src/sys/kern/uipc_socket.c:1853
#9 0xc059bdfd in kern_recvit (td=0xc5092880, s=63, mp=0xe7313bac, fromseg=UIO_USERSPACE, controlp=0x0) at /usr/src/sys/kern/uipc_syscalls.c:968
#10 0xc059bfe1 in recvit (td=Variable "td" is not available.
) at /usr/src/sys/kern/uipc_syscalls.c:1075
#11 0xc059c156 in recvfrom (td=0xc5092880, uap=0xe7313c28) at /usr/src/sys/kern/uipc_syscalls.c:1119
#12 0xc07ff8ad in ?? ()
#13 0xc5092880 in ?? ()
#14 0xe7313c28 in ?? ()
#15 0x00000010 in ?? ()
#16 0x00000000 in ?? ()
#17 0x00000000 in ?? ()
#18 0xc5055680 in ?? ()
#19 0x0000001e in ?? ()
#20 0x00000000 in ?? ()
#21 0xc4e25488 in ?? ()
#22 0xc730b2a8 in ?? ()
#23 0x00000000 in ?? ()
#24 0x00000000 in ?? ()
#25 0x00000000 in ?? ()
#26 0xba7b9cd6 in ?? ()
#27 0xe7313c14 in ?? ()
#28 0xc068cd18 in spinlock_enter () at /usr/src/sys/i386/i386/machdep.c:2380
#29 0xc0697b93 in syscall (frame=0xe7313d38) at /usr/src/sys/i386/i386/trap.c:1034
#30 0xc06818b0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:203
#31 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) thread 190
[Switching to thread 190 (Thread 100201)]
#0 sched_switch (td=0xc5092aa0, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1905
1905 cpuid = PCPU_GET(cpuid);
(kgdb) bt
#0 sched_switch (td=0xc5092aa0, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1905
#1 0xc054b6d3 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2 0xc05720a0 in sleepq_switch (wchan=0xc073b664) at /usr/src/sys/kern/subr_sleepqueue.c:478
#3 0xc05722ee in sleepq_catch_signals (wchan=0xc073b664) at /usr/src/sys/kern/subr_sleepqueue.c:414
#4 0xc0572a77 in sleepq_timedwait_sig (wchan=0xc073b664) at /usr/src/sys/kern/subr_sleepqueue.c:612
#5 0xc054bb91 in _sleep (ident=0xc073b664, lock=0x0, priority=348, wmesg=0xc06ce3dc "nanslp", timo=21) at /usr/src/sys/kern/kern_synch.c:222
#6 0xc0552891 in kern_nanosleep (td=0xc5092aa0, rqt=0xe7316c5c, rmt=0x0) at /usr/src/sys/kern/kern_time.c:379
#7 0xc0802dd3 in ?? ()
#8 0xc5092aa0 in ?? ()
#9 0xe7316c5c in ?? ()
#10 0x00000000 in ?? ()
#11 0xe7316cfc in ?? ()
#12 0xbe9ffa28 in ?? ()
#13 0x00000000 in ?? ()
#14 0x01312d00 in ?? ()
#15 0xc5092aa0 in ?? ()
#16 0x40242aa0 in ?? ()
#17 0x00000000 in ?? ()
#18 0x01312d00 in ?? ()
#19 0xbe9ff9f0 in ?? ()
#20 0xc5092aa0 in ?? ()
#21 0xc505b2ac in ?? ()
#22 0xe7316d2c in ?? ()
#23 0xc0697b93 in syscall (frame=0xc5092aa0) at /usr/src/sys/i386/i386/trap.c:1034
Previous frame identical to this frame (corrupt stack?)
(kgdb) thread 191
[Switching to thread 191 (Thread 100197)]
#0 sched_switch (td=0xc5092440, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1905
1905 cpuid = PCPU_GET(cpuid);
(kgdb) bt
#0 sched_switch (td=0xc5092440, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1905
#1 0xc054b6d3 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2 0xc05720a0 in sleepq_switch (wchan=0xc44bd84c) at /usr/src/sys/kern/subr_sleepqueue.c:478
#3 0xc05722ee in sleepq_catch_signals (wchan=0xc44bd84c) at /usr/src/sys/kern/subr_sleepqueue.c:414
#4 0xc0572b84 in sleepq_wait_sig (wchan=0xc44bd84c) at /usr/src/sys/kern/subr_sleepqueue.c:575
#5 0xc054bbb9 in _sleep (ident=0xc44bd84c, lock=0xc44bd804, priority=344, wmesg=0xc06d4a16 "sbwait", timo=0) at /usr/src/sys/kern/kern_synch.c:226
#6 0xc0595556 in sbwait (sb=0xc44bd7e4) at /usr/src/sys/kern/uipc_sockbuf.c:131
#7 0xc059a69f in soreceive_generic (so=0xc44bd794, psa=0xe730ab34, uio=0xe730ab40, mp0=0x0, controlp=0x0, flagsp=0xe730abc4)
at /usr/src/sys/kern/uipc_socket.c:1485
#8 0xc059670d in soreceive (so=0xc44bd794, psa=0xe730ab34, uio=0xe730ab40, mp0=0x0, controlp=0x0, flagsp=0xe730abc4)
at /usr/src/sys/kern/uipc_socket.c:1853
#9 0xc059bdfd in kern_recvit (td=0xc5092440, s=65, mp=0xe730abac, fromseg=UIO_USERSPACE, controlp=0x0) at /usr/src/sys/kern/uipc_syscalls.c:968
#10 0xc059bfe1 in recvit (td=Variable "td" is not available.
) at /usr/src/sys/kern/uipc_syscalls.c:1075
#11 0xc059c156 in recvfrom (td=0xc5092440, uap=0xe730ac28) at /usr/src/sys/kern/uipc_syscalls.c:1119
#12 0xc07ff8ad in ?? ()
#13 0xc5092440 in ?? ()
#14 0xe730ac28 in ?? ()
#15 0x00000010 in ?? ()
#16 0x00000000 in ?? ()
#17 0x00000000 in ?? ()
#18 0xc5055680 in ?? ()
#19 0x00000034 in ?? ()
#20 0x00000000 in ?? ()
#21 0xc4e25488 in ?? ()
#22 0xc730b2a8 in ?? ()
#23 0x00000000 in ?? ()
#24 0x00000000 in ?? ()
#25 0x00000000 in ?? ()
#26 0xdd313180 in ?? ()
#27 0xe730ac14 in ?? ()
#28 0xc068cd18 in spinlock_enter () at /usr/src/sys/i386/i386/machdep.c:2380
#29 0xc0697b93 in syscall (frame=0xe730ad38) at /usr/src/sys/i386/i386/trap.c:1034
#30 0xc06818b0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:203
#31 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) q
--BOKacYhQ+x31HxR3--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20080129015916.GA5413>