From owner-freebsd-fs@FreeBSD.ORG Tue Apr 10 08:12:37 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0BA6A16A402; Tue, 10 Apr 2007 08:12:37 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from smtp2.yandex.ru (smtp2.yandex.ru [213.180.200.18]) by mx1.freebsd.org (Postfix) with ESMTP id B369513C4B8; Tue, 10 Apr 2007 08:12:35 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from mail.kirov.so-cdu.ru ([77.72.136.145]:1289 "EHLO [127.0.0.1]" smtp-auth: "bu7cher" TLS-CIPHER: "DHE-RSA-AES256-SHA keybits 256/256 version TLSv1/SSLv3" TLS-PEER-CN1: ) by mail.yandex.ru with ESMTP id S3375841AbXDJIMZ (ORCPT + 2 others); Tue, 10 Apr 2007 12:12:25 +0400 X-Comment: RFC 2476 MSA function at smtp2.yandex.ru logged sender identity as: bu7cher Message-ID: <461B46E5.5070509@yandex.ru> Date: Tue, 10 Apr 2007 12:12:21 +0400 From: "Andrey V. Elsukov" User-Agent: Mozilla Thunderbird 1.5 (FreeBSD/20051231) MIME-Version: 1.0 To: Kris Kennaway References: <20070406025700.GB98545@garage.freebsd.pl> <461B1DDC.8050009@yandex.ru> <20070410061006.GA42711@xor.obsecurity.org> <6eb82e0704092335h31df5be5qd7cee8f7234b1539@mail.gmail.com> <20070410063817.GA43061@xor.obsecurity.org> <461B36D5.3020307@yandex.ru> <20070410070628.GA43365@xor.obsecurity.org> In-Reply-To: <20070410070628.GA43365@xor.obsecurity.org> Content-Type: multipart/mixed; boundary="------------080400060005060508070803" Cc: freebsd-fs@freebsd.org, Rong-en Fan , zfs-discuss@opensolaris.org, freebsd-current@freebsd.org, Pawel Jakub Dawidek Subject: Re: ZFS committed to the FreeBSD base. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Apr 2007 08:12:37 -0000 This is a multi-part message in MIME format. --------------080400060005060508070803 Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit Kris Kennaway wrote: > \o/ > > You might need to recompile with DEBUG_LOCKS and DEBUG_VFS_LOCKS and > do 'show lockedvnods', but maybe this is trivially reproducible. I've rollbacked and destroyed this snapshot and now don't have this problem. But i have several LOR. -- WBR, Andrey V. Elsukov --------------080400060005060508070803 Content-Type: text/plain; name="zfs-report.txt" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="zfs-report.txt" [btr-nb butcher]# zfs list NAME USED AVAIL REFER MOUNTPOINT media 431M 28,4G 19K /media media/disk3 431M 28,4G 21K /media/disk3 media/disk3/src 431M 28,4G 428M /media/disk3/src media/disk3/src@test 2,87M - 430M - [btr-nb butcher]# zfs rollback media/disk3/src@test lock order reversal: 1st 0xc2be9154 zfs:&db->db_mtx (zfs:&db->db_mtx) @ /usr/src/sys/modules/zfs/../ ../contrib/opensolaris/uts/common/fs/zfs/dnode.c:318 2nd 0xc2c94b20 zfs:&zp->z_lock (zfs:&zp->z_lock) @ /usr/src/sys/modules/zfs/../ ../contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:73 KDB: stack backtrace: db_trace_self_wrapper(c06f8954) at db_trace_self_wrapper+0x25 kdb_backtrace(0,ffffffff,c0786648,c07864b8,c073cbac,...) at kdb_backtrace+0x29 witness_checkorder(c2c94b20,9,c294b949,49) at witness_checkorder+0x586 _sx_xlock(c2c94b20,c294b949,49,c056dc15,c07c5498,...) at _sx_xlock+0x3e znode_pageout_func(c2be9118,c2c94b10,c2be9118,d3ca8aa4,c28f5ad1,...) at znode_pa geout_func+0x1c dbuf_evict_user(c2be9230,c2be9118,c2cb4250,c2c53cb0,d3ca8ab4,...) at dbuf_evict_ user+0x31 dbuf_clear(c2be9118,c2cb42e8,d3ca8ad8,c290283a,c2be9118,...) at dbuf_clear+0x1d dbuf_evict(c2be9118,c2be9154,c2947a33,13e,3,...) at dbuf_evict+0xd dnode_destroy(c2be9230,c2c921d0,c2be92bc,d3ca8b00,c28f4a7d,...) at dnode_destroy +0xbe dnode_buf_pageout(c2be9230,c2cb6300,c2be9230,d3ca8b18,c28f5ad1,...) at dnode_buf _pageout+0x33 dbuf_evict_user(0,c2be9230,c2be926c,c2c92350,d3ca8bcc,...) at dbuf_evict_user+0x 31 dbuf_clear(c2be9230,c2c92338,0,1,9,...) at dbuf_clear+0x1d dnode_evict_dbufs(c2c921d0,1,c2cb4250,c2947717,151,...) at dnode_evict_dbufs+0x7 8 dmu_objset_evict_dbufs(c281e610,1,c2a6d048,c2c7407c,c2a6d070,...) at dmu_objset_ evict_dbufs+0xe5 zfs_umount(c2c74000,0,c2daa870) at zfs_umount+0x1b1 dounmount(c2c74000,0,c2daa870,c0568bf2,c2daa870,...) at dounmount+0x3c1 unmount(c2daa870,d3ca8d00) at unmount+0x231 syscall(d3ca8d38) at syscall+0x26a Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (22, FreeBSD ELF32, unmount), eip = 0x2811044f, esp = 0xbfbfd23c, eb p = 0xbfbfd268 --- KDB: enter: witness_checkorder [thread pid 1122 tid 100181 ] Stopped at kdb_enter+0x2b: nop db> db> db> cont lock order reversal: 1st 0xc30ca520 zfs:&dr->dt.di.dr_mtx (zfs:&dr->dt.di.dr_mtx) @ /usr/src/sys/mod ules/zfs/../../contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1865 2nd 0xc2be9bb8 zfs:&db->db_mtx (zfs:&db->db_mtx) @ /usr/src/sys/modules/zfs/../ ../contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1836 KDB: stack backtrace: db_trace_self_wrapper(c06f8954) at db_trace_self_wrapper+0x25 kdb_backtrace(0,ffffffff,c0786350,c0786648,c073cbac,...) at kdb_backtrace+0x29 witness_checkorder(c2be9bb8,9,c294715e,72c) at witness_checkorder+0x586 _sx_xlock(c2be9bb8,c294715e,72c,728,c07101b6,...) at _sx_xlock+0x3e dbuf_sync_list(c30ca538,c2ebc700,c06f7bd3,91,c294880b,...) at dbuf_sync_list+0x5 e dbuf_sync_list(c2c92b80,c2ebc700,d3c48b84,c28fe6b1,c2bacc00,...) at dbuf_sync_li st+0xde dnode_sync(c2c92ae0,c2ebc700,720,0,c2b1d400,...) at dnode_sync+0x3a8 dmu_objset_sync(c2c3de00,c2bacc00,c2ebc700,c281294c,c28bf600,...) at dmu_objset_ sync+0xf6 dsl_pool_sync(c2812800,720,0,c2868000,720,...) at dsl_pool_sync+0x6d spa_sync(c2868000,720,0,c28128ac,c294aa21,...) at spa_sync+0x33f txg_sync_thread(c2812800,d3c48d38) at txg_sync_thread+0x183 fork_exit(c291a75c,c2812800,d3c48d38) at fork_exit+0xac fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xd3c48d70, ebp = 0 --- KDB: enter: witness_checkorder [thread pid 295 tid 100149 ] Stopped at kdb_enter+0x2b: nop db> cont [btr-nb butcher]# [btr-nb butcher]# zfs list NAME USED AVAIL REFER MOUNTPOINT media 431M 28,4G 19K /media media/disk3 431M 28,4G 21K /media/disk3 media/disk3/src 430M 28,4G 430M /media/disk3/src media/disk3/src@test 0 - 430M - [btr-nb butcher]# zfs destroy media/disk3/src@test lock order reversal: 1st 0xc2c3e818 zfs:&ds->ds_deadlist.bpl_lock (zfs:&ds->ds_deadlist.bpl_lock) @ /usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/fs/zfs/bplist.c:15 4 2nd 0xc2be63a0 zfs:&dn->dn_struct_rwlock (zfs:&dn->dn_struct_rwlock) @ /usr/src /sys/modules/zfs/../../contrib/opensolaris/uts/common/fs/zfs/dnode.c:571 KDB: stack backtrace: db_trace_self_wrapper(c06f8954) at db_trace_self_wrapper+0x25 kdb_backtrace(0,ffffffff,c0786508,c0786710,c073cbac,...) at kdb_backtrace+0x29 witness_checkorder(c2be63a0,1,c2947a33,23b,c2be63a0,...) at witness_checkorder+0 x586 _sx_slock(c2be63a0,c2947a33,23b,91,c2a42000,...) at _sx_slock+0x3e dnode_hold_impl(c28bf600,22,0,1,c29456cd,...) at dnode_hold_impl+0x5b dnode_hold(c28bf600,22,0,c29456cd,d3c48a10,...) at dnode_hold+0x19 dmu_bonus_hold(c28bf618,22,0,c2c3e818,c2c3e848,...) at dmu_bonus_hold+0x20 bplist_hold(c2c3e818,c294700d,9a,0,0,...) at bplist_hold+0x27 bplist_iterate(c2c3e818,d3c48ac8,d3c48ad8,c1056788,728,...) at bplist_iterate+0x 23 dsl_dataset_destroy_sync(c2ff3200,c2945a94,c2eeb600,c29846f0,c28129a4,...) at ds l_dataset_destroy_sync+0x1f5 dsl_sync_task_group_sync(c2eead00,c2eeb600,c281294c,c28bf600,c2eeb600,...) at ds l_sync_task_group_sync+0x110 dsl_pool_sync(c2812800,7a0,0,c2868000,7a0,...) at dsl_pool_sync+0xb9 spa_sync(c2868000,7a0,0,c28128ac,c294aa21,...) at spa_sync+0x33f txg_sync_thread(c2812800,d3c48d38) at txg_sync_thread+0x183 fork_exit(c291a75c,c2812800,d3c48d38) at fork_exit+0xac fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xd3c48d70, ebp = 0 --- KDB: enter: witness_checkorder [thread pid 295 tid 100149 ] Stopped at kdb_enter+0x2b: nop db> cont [btr-nb butcher]# [btr-nb butcher]# zfs snapshot media/disk3@test acquiring duplicate lock of same type: "sleepq chain" 1st sleepq chain @ /usr/src/sys/kern/subr_sleepqueue.c:229 2nd sleepq chain @ /usr/src/sys/kern/subr_sleepqueue.c:229 KDB: stack backtrace: db_trace_self_wrapper(c06f8954) at db_trace_self_wrapper+0x25 kdb_backtrace(c077f434,0,c0788c68,c0788c68,c073b578,...) at kdb_backtrace+0x29 witness_checkorder(c077d784,9,c06f93a3,e5) at witness_checkorder+0x586 _mtx_lock_spin_flags(c077d784,0,c06f93a3,e5,d3c15c34,...) at _mtx_lock_spin_flag s+0x81 sleepq_lock(c29700ec,c29700ec,d3c15c58,c0547fd8,c29700ec,...) at sleepq_lock+0x2 e _sx_xunlock_hard(c29700ec,c2a291b0,c06f674e,98) at _sx_xunlock_hard+0x7e _sx_xunlock(c29700ec,c06f674e,98,c2a291b0,d3c15ca0,...) at _sx_xunlock+0x80 unlock_sx(c29700ec,c0566c7a,c073cbac,0,c06f93a3,...) at unlock_sx+0x3d _cv_wait(c297011c,c29700ec,c2970104,c2970124,c2970148,...) at _cv_wait+0x161 taskq_thread(c29700cc,d3c15d38) at taskq_thread+0x1de fork_exit(c28e7740,c29700cc,d3c15d38) at fork_exit+0xac fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xd3c15d70, ebp = 0 --- KDB: enter: witness_checkorder [thread pid 243 tid 100134 ] Stopped at kdb_enter+0x2b: nop db> cont [btr-nb butcher]# --------------080400060005060508070803--