From owner-freebsd-stable@FreeBSD.ORG Fri Sep 27 08:59:26 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id CCD77BB5 for ; Fri, 27 Sep 2013 08:59:26 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from cpsmtpb-ews04.kpnxchange.com (cpsmtpb-ews04.kpnxchange.com [213.75.39.7]) by mx1.freebsd.org (Postfix) with ESMTP id 26F1E2C99 for ; Fri, 27 Sep 2013 08:59:25 +0000 (UTC) Received: from cpsps-ews03.kpnxchange.com ([10.94.84.170]) by cpsmtpb-ews04.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514); Fri, 27 Sep 2013 10:58:14 +0200 Received: from CPSMTPM-TLF102.kpnxchange.com ([195.121.3.5]) by cpsps-ews03.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514); Fri, 27 Sep 2013 10:58:14 +0200 Received: from sjakie.klop.ws ([212.182.167.131]) by CPSMTPM-TLF102.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514); Fri, 27 Sep 2013 10:58:14 +0200 Received: from 212-182-167-131.ip.telfort.nl (localhost [127.0.0.1]) by sjakie.klop.ws (Postfix) with ESMTP id 33CED822 for ; Fri, 27 Sep 2013 10:58:14 +0200 (CEST) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes To: freebsd-stable@freebsd.org Subject: Re: lock order reversal in 10-alpha2 References: <52450239.7010100@ShaneWare.Biz> Date: Fri, 27 Sep 2013 10:58:14 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Ronald Klop" Message-ID: In-Reply-To: <52450239.7010100@ShaneWare.Biz> User-Agent: Opera Mail/12.16 (FreeBSD) X-OriginalArrivalTime: 27 Sep 2013 08:58:14.0466 (UTC) FILETIME=[B34B3220:01CEBB5F] X-RcptDomain: freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Sep 2013 08:59:26 -0000 On Fri, 27 Sep 2013 05:57:45 +0200, Shane Ambler wrote: > After booting from a 10-alpha2 disk I am seeing "lock order reversal" > messages show up from time to time. Current logs have 35 entries. FreeBSD 10-ALPHA is still being build with kernel option WITNESS on. This gives more diagnostics of internal state of the kernel. A lot of these LORs (Lock Order Reverals) should be fixed some day, but are harmless for the continues working of the machine. If you compile FreeBSD 9 with WITNESS on, you will some LORs also. Ronald. > > The machine normally is running 9.1 from zfs root and I have setup a > separate disk (eSATA case connected through backplane port to onboard > SATA port) that I have installed 10-alpha amd64 onto a ufs partition to > test port building with. I started by building 10 alpha1 and installing > onto the new disk. I have since done svn up (last revision is 255868) > then rebuilt and installed kernel and world while running 10 and still > see these messages. > > I mentioned the existing 9.1 on zfs which I am not importing while > running 10 from ufs as I noticed zfs mentioned in one of the entries. > > Initially I built with an empty src.conf but the last build I used the > following - > > WITH_BSD_GREP=yes > WITH_CLANG_EXTRAS=yes > WITH_CTF=yes > WITHOUT_LIB32=yes > WITH_LLDB=yes > > Hardware is ASUS P8H61-M LE/USB3 corei5 8GB RAM nvidia GT520 > > I can provide full copy of log/messages or dmesg if required. > > A few samples -- > > messages:Sep 26 02:01:27 leader kernel: lock order reversal: > messages-Sep 26 02:01:27 leader kernel: 1st 0xfffffe01eebd07f8 bufwait > (bufwait) @ /usr/src/sys/kern/vfs_bio.c:3059 > messages-Sep 26 02:01:27 leader kernel: 2nd 0xfffff800122f8200 dirhash > (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:284 > messages-Sep 26 02:01:27 leader kernel: KDB: stack backtrace: > messages-Sep 26 02:01:27 leader kernel: db_trace_self_wrapper() at > db_trace_self_wrapper+0x2b/frame 0xfffffe0238a8f270 > messages-Sep 26 02:01:27 leader kernel: kdb_backtrace() at > kdb_backtrace+0x39/frame 0xfffffe0238a8f320 > messages-Sep 26 02:01:27 leader kernel: witness_checkorder() at > witness_checkorder+0xd23/frame 0xfffffe0238a8f3b0 > messages-Sep 26 02:01:27 leader kernel: _sx_xlock() at > _sx_xlock+0x75/frame 0xfffffe0238a8f3f0 > messages-Sep 26 02:01:27 leader kernel: ufsdirhash_add() at > ufsdirhash_add+0x3b/frame 0xfffffe0238a8f430 > messages-Sep 26 02:01:27 leader kernel: ufs_direnter() at > ufs_direnter+0x688/frame 0xfffffe0238a8f4f0 > messages-Sep 26 02:01:27 leader kernel: ufs_makeinode() at > ufs_makeinode+0x573/frame 0xfffffe0238a8f6b0 > messages-Sep 26 02:01:27 leader kernel: VOP_CREATE_APV() at > VOP_CREATE_APV+0xea/frame 0xfffffe0238a8f6e0 > messages-Sep 26 02:01:27 leader kernel: vn_open_cred() at > vn_open_cred+0x300/frame 0xfffffe0238a8f830 > messages-Sep 26 02:01:27 leader kernel: kern_openat() at > kern_openat+0x261/frame 0xfffffe0238a8f9a0 > messages-Sep 26 02:01:27 leader kernel: amd64_syscall() at > amd64_syscall+0x265/frame 0xfffffe0238a8fab0 > messages-Sep 26 02:01:27 leader kernel: Xfast_syscall() at > Xfast_syscall+0xfb/frame 0xfffffe0238a8fab0 > messages-Sep 26 02:01:27 leader kernel: --- syscall (5, FreeBSD ELF64, > sys_open), rip = 0x80185baca, rsp = 0x7fffffffd168, rbp = 0x7fffffffd1a0 > --- > > messages.0:Sep 23 10:08:11 leader kernel: lock order reversal: > messages.0-Sep 23 10:08:11 leader kernel: 1st 0xfffff801ba2e65f0 ufs > (ufs) @ /usr/src/sys/kern/vfs_syscalls.c:3435 > messages.0-Sep 23 10:08:11 leader kernel: 2nd 0xfffffe01ef93c1c0 bufwait > (bufwait) @ /usr/src/sys/ufs/ffs/ffs_vnops.c:262 > messages.0-Sep 23 10:08:11 leader kernel: 3rd 0xfffff801ba2e6240 ufs > (ufs) @ /usr/src/sys/kern/vfs_subr.c:2099 > messages.0-Sep 23 10:08:11 leader kernel: KDB: stack backtrace: > messages.0-Sep 23 10:08:11 leader kernel: db_trace_self_wrapper() at > db_trace_self_wrapper+0x2b/frame 0xfffffe02397c2300 > messages.0-Sep 23 10:08:11 leader kernel: kdb_backtrace() at > kdb_backtrace+0x39/frame 0xfffffe02397c23b0 > messages.0-Sep 23 10:08:11 leader kernel: witness_checkorder() at > witness_checkorder+0xd23/frame 0xfffffe02397c2440 > messages.0-Sep 23 10:08:11 leader kernel: __lockmgr_args() at > __lockmgr_args+0x6f2/frame 0xfffffe02397c2570 > messages.0-Sep 23 10:08:11 leader kernel: ffs_lock() at > ffs_lock+0x84/frame 0xfffffe02397c25c0 > messages.0-Sep 23 10:08:11 leader kernel: VOP_LOCK1_APV() at > VOP_LOCK1_APV+0xf5/frame 0xfffffe02397c25f0 > messages.0-Sep 23 10:08:11 leader kernel: _vn_lock() at > _vn_lock+0xab/frame 0xfffffe02397c2660 > messages.0-Sep 23 10:08:11 leader kernel: vget() at vget+0x70/frame > 0xfffffe02397c26b0 > messages.0-Sep 23 10:08:11 leader kernel: vfs_hash_get() at > vfs_hash_get+0xf5/frame 0xfffffe02397c2700 > messages.0-Sep 23 10:08:11 leader kernel: ffs_vgetf() at > ffs_vgetf+0x41/frame 0xfffffe02397c2790 > messages.0-Sep 23 10:08:11 leader kernel: softdep_sync_buf() at > softdep_sync_buf+0x8fa/frame 0xfffffe02397c2840 > messages.0-Sep 23 10:08:11 leader kernel: ffs_syncvnode() at > ffs_syncvnode+0x258/frame 0xfffffe02397c28c0 > messages.0-Sep 23 10:08:11 leader kernel: ffs_fsync() at > ffs_fsync+0x20/frame 0xfffffe02397c28f0 > messages.0-Sep 23 10:08:11 leader kernel: VOP_FSYNC_APV() at > VOP_FSYNC_APV+0xf0/frame 0xfffffe02397c2920 > messages.0-Sep 23 10:08:11 leader kernel: sys_fsync() at > sys_fsync+0x156/frame 0xfffffe02397c29a0 > messages.0-Sep 23 10:08:11 leader kernel: amd64_syscall() at > amd64_syscall+0x265/frame 0xfffffe02397c2ab0 > messages.0-Sep 23 10:08:11 leader kernel: Xfast_syscall() at > Xfast_syscall+0xfb/frame 0xfffffe02397c2ab0 > messages.0-Sep 23 10:08:11 leader kernel: --- syscall (95, FreeBSD > ELF64, sys_fsync), rip = 0x8029a41fa, rsp = 0x7fffffffcf28, rbp = > 0x7fffffffcf40 --- > > messages.0:Sep 23 10:08:11 leader kernel: lock order reversal: > messages.0-Sep 23 10:08:11 leader kernel: 1st 0xfffff801ba2e65f0 ufs > (ufs) @ /usr/src/sys/kern/vfs_syscalls.c:3435 > messages.0-Sep 23 10:08:11 leader kernel: 2nd 0xfffffe01ef93c1c0 bufwait > (bufwait) @ /usr/src/sys/ufs/ffs/ffs_vnops.c:262 > messages.0-Sep 23 10:08:11 leader kernel: 3rd 0xfffff801ba2e6240 ufs > (ufs) @ /usr/src/sys/kern/vfs_subr.c:2099 > messages.0-Sep 23 10:08:11 leader kernel: KDB: stack backtrace: > messages.0-Sep 23 10:08:11 leader kernel: db_trace_self_wrapper() at > db_trace_self_wrapper+0x2b/frame 0xfffffe02397c2300 > messages.0-Sep 23 10:08:11 leader kernel: kdb_backtrace() at > kdb_backtrace+0x39/frame 0xfffffe02397c23b0 > messages.0-Sep 23 10:08:11 leader kernel: witness_checkorder() at > witness_checkorder+0xd23/frame 0xfffffe02397c2440 > messages.0-Sep 23 10:08:11 leader kernel: __lockmgr_args() at > __lockmgr_args+0x6f2/frame 0xfffffe02397c2570 > messages.0-Sep 23 10:08:11 leader kernel: ffs_lock() at > ffs_lock+0x84/frame 0xfffffe02397c25c0 > messages.0-Sep 23 10:08:11 leader kernel: VOP_LOCK1_APV() at > VOP_LOCK1_APV+0xf5/frame 0xfffffe02397c25f0 > messages.0-Sep 23 10:08:11 leader kernel: _vn_lock() at > _vn_lock+0xab/frame 0xfffffe02397c2660 > messages.0-Sep 23 10:08:11 leader kernel: vget() at vget+0x70/frame > 0xfffffe02397c26b0 > messages.0-Sep 23 10:08:11 leader kernel: vfs_hash_get() at > vfs_hash_get+0xf5/frame 0xfffffe02397c2700 > messages.0-Sep 23 10:08:11 leader kernel: ffs_vgetf() at > ffs_vgetf+0x41/frame 0xfffffe02397c2790 > messages.0-Sep 23 10:08:11 leader kernel: softdep_sync_buf() at > softdep_sync_buf+0x8fa/frame 0xfffffe02397c2840 > messages.0-Sep 23 10:08:11 leader kernel: ffs_syncvnode() at > ffs_syncvnode+0x258/frame 0xfffffe02397c28c0 > messages.0-Sep 23 10:08:11 leader kernel: ffs_fsync() at > ffs_fsync+0x20/frame 0xfffffe02397c28f0 > messages.0-Sep 23 10:08:11 leader kernel: VOP_FSYNC_APV() at > VOP_FSYNC_APV+0xf0/frame 0xfffffe02397c2920 > messages.0-Sep 23 10:08:11 leader kernel: sys_fsync() at > sys_fsync+0x156/frame 0xfffffe02397c29a0 > messages.0-Sep 23 10:08:11 leader kernel: amd64_syscall() at > amd64_syscall+0x265/frame 0xfffffe02397c2ab0 > messages.0-Sep 23 10:08:11 leader kernel: Xfast_syscall() at > Xfast_syscall+0xfb/frame 0xfffffe02397c2ab0 > messages.0-Sep 23 10:08:11 leader kernel: --- syscall (95, FreeBSD > ELF64, sys_fsync), rip = 0x8029a41fa, rsp = 0x7fffffffcf28, rbp = > 0x7fffffffcf40 --- > > messages.0:Sep 23 10:36:02 leader kernel: lock order reversal: > messages.0-Sep 23 10:36:02 leader kernel: 1st 0xfffff801ba9be240 zfs > (zfs) @ /usr/src/sys/kern/vfs_mount.c:1237 > messages.0-Sep 23 10:36:02 leader kernel: 2nd 0xfffff801babab7c8 syncer > (syncer) @ /usr/src/sys/kern/vfs_subr.c:2210 > messages.0-Sep 23 10:36:02 leader kernel: KDB: stack backtrace: > messages.0-Sep 23 10:36:02 leader kernel: db_trace_self_wrapper() at > db_trace_self_wrapper+0x2b/frame 0xfffffe02397ef460 > messages.0-Sep 23 10:36:02 leader kernel: kdb_backtrace() at > kdb_backtrace+0x39/frame 0xfffffe02397ef510 > messages.0-Sep 23 10:36:02 leader kernel: witness_checkorder() at > witness_checkorder+0xd23/frame 0xfffffe02397ef5a0 > messages.0-Sep 23 10:36:02 leader kernel: __lockmgr_args() at > __lockmgr_args+0x6f2/frame 0xfffffe02397ef6d0 > messages.0-Sep 23 10:36:02 leader kernel: vop_stdlock() at > vop_stdlock+0x3c/frame 0xfffffe02397ef6f0 > messages.0-Sep 23 10:36:02 leader kernel: VOP_LOCK1_APV() at > VOP_LOCK1_APV+0xf5/frame 0xfffffe02397ef720 > messages.0-Sep 23 10:36:02 leader kernel: _vn_lock() at > _vn_lock+0xab/frame 0xfffffe02397ef790 > messages.0-Sep 23 10:36:02 leader kernel: vputx() at vputx+0x208/frame > 0xfffffe02397ef7f0 > messages.0-Sep 23 10:36:02 leader kernel: dounmount() at > dounmount+0x327/frame 0xfffffe02397ef870 > messages.0-Sep 23 10:36:02 leader kernel: sys_unmount() at > sys_unmount+0x356/frame 0xfffffe02397ef9a0 > messages.0-Sep 23 10:36:02 leader kernel: amd64_syscall() at > amd64_syscall+0x265/frame 0xfffffe02397efab0 > messages.0-Sep 23 10:36:02 leader kernel: Xfast_syscall() at > Xfast_syscall+0xfb/frame 0xfffffe02397efab0 > messages.0-Sep 23 10:36:02 leader kernel: --- syscall (22, FreeBSD > ELF64, sys_unmount), rip = 0x80191f24a, rsp = 0x7fffffffc3d8, rbp = > 0x7fffffffc860 --- > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"