From owner-freebsd-bugs@FreeBSD.ORG Mon Feb 13 09:00:29 2012 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 63F05106564A for ; Mon, 13 Feb 2012 09:00:29 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 4033A8FC26 for ; Mon, 13 Feb 2012 09:00:29 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q1D90Ton060516 for ; Mon, 13 Feb 2012 09:00:29 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q1D90Ttj060515; Mon, 13 Feb 2012 09:00:29 GMT (envelope-from gnats) Resent-Date: Mon, 13 Feb 2012 09:00:29 GMT Resent-Message-Id: <201202130900.q1D90Ttj060515@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Pavel Polyakov Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6D359106564A for ; Mon, 13 Feb 2012 08:52:11 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id 5C4618FC12 for ; Mon, 13 Feb 2012 08:52:11 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q1D8qBX2092472 for ; Mon, 13 Feb 2012 08:52:11 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.4/8.14.4/Submit) id q1D8qB5U092471; Mon, 13 Feb 2012 08:52:11 GMT (envelope-from nobody) Message-Id: <201202130852.q1D8qB5U092471@red.freebsd.org> Date: Mon, 13 Feb 2012 08:52:11 GMT From: Pavel Polyakov To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/165087: lock violation in unionfs X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Feb 2012 09:00:29 -0000 >Number: 165087 >Category: kern >Synopsis: lock violation in unionfs >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Mon Feb 13 09:00:28 UTC 2012 >Closed-Date: >Last-Modified: >Originator: Pavel Polyakov >Release: FreeBSD 9.0-STABLE r230270 >Organization: >Environment: FreeBSD 9.0-STABLE #0 r230270M: Wed Feb 1 00:31:59 EET 2012 >Description: Dump header from device /dev/label/dump0 Architecture: amd64 Architecture Version: 2 Dump Length: 743739392B (709 MB) Blocksize: 512 Dumptime: Sun Feb 12 15:22:16 2012 Hostname: cel.home Magic: FreeBSD Kernel Dump Version String: FreeBSD 9.0-STABLE #0 r230270M: Wed Feb 1 00:31:59 EET 2012 root@cel.home:/usr/obj/usr/src/sys/PDC90 Panic String: Dump Parity: 2051880815 Bounds: 75 Dump Status: good Unread portion of the kernel message buffer: KDB: stack backtrace: db_trace_self_wrapper() at 0xffffffff802da48a = db_trace_self_wrapper+0x2a kdb_backtrace() at 0xffffffff805cc157 = kdb_backtrace+0x37 vfs_badlock() at 0xffffffff8062ee75 = vfs_badlock+0x95 assert_vop_elocked() at 0xffffffff80632ff9 = assert_vop_elocked+0x69 insmntque1() at 0xffffffff8063337a = insmntque1+0x4a unionfs_nodeget() at 0xffffffff8143469c = unionfs_nodeget+0x10c unionfs_domount() at 0xffffffff81436e2f = unionfs_domount+0x4af vfs_donmount() at 0xffffffff80629784 = vfs_donmount+0x884 sys_nmount() at 0xffffffff8062a542 = sys_nmount+0x82 amd64_syscall() at 0xffffffff80831b1b = amd64_syscall+0x27b Xfast_syscall() at 0xffffffff8081cef7 = Xfast_syscall+0xf7 --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x8008a638c, rsp = 0x7fffffffcb48, rbp = 0x7fffffffcb70 --- insmntque: mp-safe fs and non-locked vp: 0xfffffe01d96704f0 is not exclusive locked but should be KDB: enter: lock violation Dumping 709 out of 8175 MB:..3%..12%..21%..32%..41%..52%..61%..73%..82%..91% .. Reading symbols from /boot/kernel/unionfs.ko...Reading symbols from /boot/kernel/unionfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/unionfs.ko #0 doadump (textdump=1818177984) at /usr/src/sys/kern/kern_shutdown.c:260 260 if (textdump && textdump_pending) { (kgdb) #0 doadump (textdump=1818177984) at /usr/src/sys/kern/kern_shutdown.c:260 #1 0xffffffff802d7cdc in db_fncall (dummy1=Variable "dummy1" is not available. ) at /usr/src/sys/ddb/db_command.c:573 #2 0xffffffff802d8011 in db_command (last_cmdp=0xffffffff80c337c0, cmd_table=Variable "cmd_table" is not available. ) at /usr/src/sys/ddb/db_command.c:449 #3 0xffffffff802d8260 in db_command_loop () at /usr/src/sys/ddb/db_command.c:502 #4 0xffffffff802da3c9 in db_trap (type=Variable "type" is not available. ) at /usr/src/sys/ddb/db_main.c:229 #5 0xffffffff805cbf91 in kdb_trap (type=3, code=0, tf=0xffffff816c5f33f0) at /usr/src/sys/kern/subr_kdb.c:620 #6 0xffffffff808328fa in trap (frame=0xffffff816c5f33f0) at /usr/src/sys/amd64/amd64/trap.c:591 #7 0xffffffff8081cc0f in calltrap () at /usr/src/sys/amd64/amd64/exception.S:228 #8 0xffffffff805cbd3b in kdb_enter (why=0xffffffff8092b343 "vfslock", msg=0x80
) at cpufunc.h:63 #9 0xffffffff80632ff9 in assert_vop_elocked (vp=0xfffffe01d96704f0, str=0xffffffff8092c460 "insmntque: mp-safe fs and non-locked vp") at /usr/src/sys/kern/vfs_subr.c:3848 #10 0xffffffff8063337a in insmntque1 (vp=0xfffffe01d96704f0, mp=0xfffffe0120200730, dtr=0xffffffff806359f0 , dtr_arg=0x0) at /usr/src/sys/kern/vfs_subr.c:1124 #11 0xffffffff8143469c in unionfs_nodeget (mp=0xfffffe0120200730, uppervp=0xfffffe001e3919e0, lowervp=0xfffffe016974d768, dvp=0x0, vpp=0xfffffe00156a4b10, cnp=0x0, td=0xfffffe001e251460) at /usr/src/sys/modules/unionfs/../../fs/unionfs/union_subr.c:264 #12 0xffffffff81436e2f in unionfs_domount (mp=0xfffffe0120200730) at /usr/src/sys/modules/unionfs/../../fs/unionfs/union_vfsops.c:289 #13 0xffffffff80629784 in vfs_donmount (td=Variable "td" is not available. ) at /usr/src/sys/kern/vfs_mount.c:800 #14 0xffffffff8062a542 in sys_nmount (td=0xfffffe001e251460, uap=0xffffff816c5f3bc0) at /usr/src/sys/kern/vfs_mount.c:410 #15 0xffffffff80831b1b in amd64_syscall (td=0xfffffe001e251460, traced=0) at subr_syscall.c:131 #16 0xffffffff8081cef7 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:387 #17 0x00000008008a638c in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) db> run lockinfo db:0:lockinfo> show locks exclusive sleep mutex Giant (Giant) r = 0 (0xffffffff80c59ea0) locked @ /usr/src/sys/dev/usb/usb_transfer.c:3165 db:0:locks> show alllocks Process 9335 (mount_unionfs) thread 0xfffffe001e251460 (100436) exclusive sleep mutex Giant (Giant) r = 0 (0xffffffff80c59ea0) locked @ /usr/src/sys/dev/usb/usb_transfer.c:3165 Process 2392 (perl5.8.9) thread 0xfffffe001e26d8c0 (100444) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xfffffe001eb33648) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 2391 (perl5.8.9) thread 0xfffffe001e83b8c0 (100476) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xfffffe001eb27b98) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 2389 (perl5.8.9) thread 0xfffffe001e26e000 (100443) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xfffffe001eb330f8) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 2388 (perl5.8.9) thread 0xfffffe001e47a460 (100492) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xfffffe001e10e8f0) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 2387 (perl5.8.9) thread 0xfffffe001e8268c0 (100470) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xfffffe001eb270f8) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 2382 (perl5.8.9) thread 0xfffffe001e56c460 (100483) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xfffffe001eb338f0) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 2380 (perl5.8.9) thread 0xfffffe00156488c0 (100090) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xfffffe001eb33e40) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 db:0:alllocks> show lockedvnods Locked vnodes db> scripts lockinfo=show locks; show alllocks; show lockedvnods kdb.enter.panic=textdump set; capture on; run lockinfo; show pcpu; bt; ps; alltrace; capture off; call doadump; reset kdb.enter.witness=run lockinfo db> bt Tracing pid 9335 tid 100436 td 0xfffffe001e251460 kdb_enter() at 0xffffffff805cbd3b = kdb_enter+0x3b assert_vop_elocked() at 0xffffffff80632ff9 = assert_vop_elocked+0x69 insmntque1() at 0xffffffff8063337a = insmntque1+0x4a unionfs_nodeget() at 0xffffffff8143469c = unionfs_nodeget+0x10c unionfs_domount() at 0xffffffff81436e2f = unionfs_domount+0x4af vfs_donmount() at 0xffffffff80629784 = vfs_donmount+0x884 sys_nmount() at 0xffffffff8062a542 = sys_nmount+0x82 amd64_syscall() at 0xffffffff80831b1b = amd64_syscall+0x27b Xfast_syscall() at 0xffffffff8081cef7 = Xfast_syscall+0xf7 --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x8008a638c, rsp = 0x7fffffffcb48, rbp = 0x7fffffffcb70 --- db> capture off USER CMD PID FD MOUNT INUM MODE SZ|DV R/W root mount_unionfs 9335 root / 2 drwxr-xr-x 1536 r root mount_unionfs 9335 wd / 26654 drwxr-xr-x 2048 r root mount_unionfs 9335 text / 52659 -r-xr-xr-x 9888 r root mount_unionfs 9335 ctty /dev 288 crw--w---- pts/9 rw root mount_unionfs 9335 0 /dev 288 crw--w---- pts/9 rw root mount_unionfs 9335 1 /dev 288 crw--w---- pts/9 rw root mount_unionfs 9335 2 /dev 288 crw--w---- pts/9 rw root mount 9334 root / 2 drwxr-xr-x 1536 r >How-To-Repeat: mount -t unionfs -o noatime /usr /mnt >Fix: >Release-Note: >Audit-Trail: >Unformatted: