Date: Sat, 18 Dec 2010 01:18:51 GMT From: Pete Wright <pete@nomadlogic.org> To: freebsd-gnats-submit@FreeBSD.org Subject: i386/153260: lor in vfs_bio and ufs_dirhash on EC2 AMI Message-ID: <201012180118.oBI1IpbW018745@red.freebsd.org> Resent-Message-ID: <201012180120.oBI1K6am086599@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 153260 >Category: i386 >Synopsis: lor in vfs_bio and ufs_dirhash on EC2 AMI >Confidential: no >Severity: non-critical >Priority: low >Responsible: freebsd-i386 >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sat Dec 18 01:20:06 UTC 2010 >Closed-Date: >Last-Modified: >Originator: Pete Wright >Release: 9.0-CURRENT >Organization: >Environment: FreeBSD ip-10-195-201-120 9.0-CURRENT FreeBSD 9.0-CURRENT #68: Sun Dec 12 03:52:29 UTC 2010 root@chch.daemonology.net:/usr/obj/i386.i386/usr/src/sys/XEN i386 >Description: Have been testing out the EC2 AMI made by Colin and have noticed this shortly after boot on systems I have brought online: Dec 18 00:44:17 ip-10-195-201-120 kernel: lock order reversal: Dec 18 00:44:17 ip-10-195-201-120 kernel: 1st 0xcef7ed00 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2659 Dec 18 00:44:17 ip-10-195-201-120 kernel: 2nd 0xc2b05800 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:284 Dec 18 00:44:17 ip-10-195-201-120 kernel: KDB: stack backtrace: Dec 18 00:44:17 ip-10-195-201-120 kernel: X_db_sym_numargs(c037be31,c052a7b8,c2b77840,c2e7fe7c,f9f60208,...) at X_db_sym_numargs+0x146 Dec 18 00:44:17 ip-10-195-201-120 kernel: kdb_backtrace(c0114f4b,c037ef3f,c28df878,c28e2388,d94d18a4,...) at kdb_backtrace+0x2a Dec 18 00:44:17 ip-10-195-201-120 kernel: witness_display_spinlock(c037ef3f,c2b05800,c039b943,c28e2388,c039b5c8,...) at witness_display_spinlock+0x75 Dec 18 00:44:17 ip-10-195-201-120 kernel: witness_checkorder(c2b05800,9,c039b5c8,11c,0,...) at witness_checkorder+0x839 Dec 18 00:44:17 ip-10-195-201-120 kernel: _sx_xlock(c2b05800,0,c039b5c8,11c,c2e449f8,...) at _sx_xlock+0x85 Dec 18 00:44:17 ip-10-195-201-120 kernel: ufsdirhash_enduseful(cef7eca0,d02a0800,200,d02a081c,d94d1974,...) at ufsdirhash_enduseful+0x2f5 Dec 18 00:44:17 ip-10-195-201-120 kernel: ufsdirhash_add(c2e449f8,d94d19fc,81c,d94d1960,d94d1964,...) at ufsdirhash_add+0x13 Dec 18 00:44:17 ip-10-195-201-120 kernel: ufs_direnter(c2e7fdd0,c2ed7220,d94d19fc,d94d1be0,cef866d0,...) at ufs_direnter+0x6f9 Dec 18 00:44:17 ip-10-195-201-120 kernel: ufs_readdir(d94d1c08,d94d1c1c,0,d94d1bb4,d94d1b4c,...) at ufs_readdir+0x288f Dec 18 00:44:17 ip-10-195-201-120 kernel: VOP_MKDIR_APV(c03da9a0,d94d1c08,68,1,0,...) at VOP_MKDIR_APV+0xa5 Dec 18 00:44:17 ip-10-195-201-120 kernel: kern_mkdirat(c2b77840,ffffff9c,8051420,0,1c0,...) at kern_mkdirat+0x211 Dec 18 00:44:17 ip-10-195-201-120 kernel: kern_mkdir(c2b77840,8051420,0,1c0,d94d1c8c,...) at kern_mkdir+0x2e Dec 18 00:44:17 ip-10-195-201-120 kernel: mkdir(c2b77840,d94d1cfc,d94d1d38,c037d866,0,...) at mkdir+0x29 Dec 18 00:44:17 ip-10-195-201-120 kernel: syscallenter(c2b77840,d94d1cf4,d94d1cf4,0,c295f000,...) at syscallenter+0x263 Dec 18 00:44:17 ip-10-195-201-120 kernel: syscall(d94d1d38) at syscall+0x34 Dec 18 00:44:17 ip-10-195-201-120 kernel: Xint0x80_syscall() at Xint0x80_syscall+0x22 Dec 18 00:44:17 ip-10-195-201-120 kernel: --- syscall (136, FreeBSD ELF32, mkdir), eip = 0x28189313, esp = 0xbf7fe90c, ebp = 0xbf7fe998 --- Dec 18 00:50:54 ip-10-195-201-120 kernel: lock order reversal: Dec 18 00:50:54 ip-10-195-201-120 kernel: 1st 0xc2d50e28 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2117 Dec 18 00:50:54 ip-10-195-201-120 kernel: 2nd 0xcef7d270 bufwait (bufwait) @ /usr/src/sys/ufs/ffs/ffs_softdep.c:11317 Dec 18 00:50:54 ip-10-195-201-120 kernel: 3rd 0xc2ffe7c8 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2117 Dec 18 00:50:54 ip-10-195-201-120 kernel: KDB: stack backtrace: Dec 18 00:50:54 ip-10-195-201-120 kernel: X_db_sym_numargs(c037be31,c039d81c,1,c28dc340,f9f60208,...) at X_db_sym_numargs+0x146 Dec 18 00:50:54 ip-10-195-201-120 kernel: kdb_backtrace(c0114f4b,c037ef58,c28df878,c28e2320,d94e0534,...) at kdb_backtrace+0x2a Dec 18 00:50:54 ip-10-195-201-120 kernel: witness_display_spinlock(c037ef58,c2ffe7c8,c03714f2,c28e2320,c0386645,...) at witness_display_spinlock+0x75 Dec 18 00:50:54 ip-10-195-201-120 kernel: witness_checkorder(c2ffe7c8,9,c0386645,845,0,...) at witness_checkorder+0x839 Dec 18 00:50:54 ip-10-195-201-120 kernel: __lockmgr_args(c2ffe7c8,80100,c2ffe7e8,0,0,...) at __lockmgr_args+0x804 Dec 18 00:50:54 ip-10-195-201-120 kernel: ffs_syncvnode(d94e065c,c01261eb,c0385998,80100,c2ffe770,...) at ffs_syncvnode+0x62a Dec 18 00:50:54 ip-10-195-201-120 kernel: VOP_LOCK1_APV(c03da9a0,d94e065c,c2d28370,c03eb520,c2ffe770,...) at VOP_LOCK1_APV+0xb5 Dec 18 00:50:54 ip-10-195-201-120 kernel: _vn_lock(c2ffe770,80100,c0386645,845,4,...) at _vn_lock+0x5e Dec 18 00:50:54 ip-10-195-201-120 kernel: vget(c2ffe770,80100,c2d282c0,50,0,...) at vget+0xb9 Dec 18 00:50:54 ip-10-195-201-120 kernel: vfs_hash_get(c2b28ca8,84527,80000,c2d282c0,d94e07b0,...) at vfs_hash_get+0xe6 Dec 18 00:50:54 ip-10-195-201-120 kernel: ffs_vgetf(c2b28ca8,84527,80000,d94e07b0,1,...) at ffs_vgetf+0x49 Dec 18 00:50:54 ip-10-195-201-120 kernel: softdep_sync_metadata(c2d50dd0,0,c039b18c,144,0,...) at softdep_sync_metadata+0xca2 Dec 18 00:50:54 ip-10-195-201-120 kernel: ffs_syncvnode(c2d50dd0,1,0,0,c2d50e7c,...) at ffs_syncvnode+0x3e2 Dec 18 00:50:54 ip-10-195-201-120 kernel: ffs_syncvnode(d94e08ac,c2d519f8,d94e08ac,d94e08ac,d94e08d0,...) at ffs_syncvnode+0x4d7 Dec 18 00:50:54 ip-10-195-201-120 kernel: VOP_FSYNC_APV(c03da9a0,d94e08ac,30000,0,84d6a,...) at VOP_FSYNC_APV+0xa5 Dec 18 00:50:54 ip-10-195-201-120 kernel: ufs_direnter(c2d50dd0,c3267220,d94e092c,d94e0bb4,0,...) at ufs_direnter+0x304 Dec 18 00:50:54 ip-10-195-201-120 kernel: ufs_itimes(d94e0bb4,0,d94e0b10,d94e0a74,c0359b85,...) at ufs_itimes+0x13e7 Dec 18 00:50:54 ip-10-195-201-120 kernel: ufs_itimes(d94e0b10,d94e0b28,0,0,d94e0b88,...) at ufs_itimes+0x16f0 Dec 18 00:50:54 ip-10-195-201-120 kernel: VOP_CREATE_APV(c03da9a0,d94e0b10,2,c01261eb,0,...) at VOP_CREATE_APV+0xa5 Dec 18 00:50:54 ip-10-195-201-120 kernel: vn_open_cred(d94e0b88,d94e0c3c,1a4,0,c2cf0480,...) at vn_open_cred+0x1d3 Dec 18 00:50:54 ip-10-195-201-120 kernel: vn_open(d94e0b88,d94e0c3c,1a4,c2b43540,0,...) at vn_open+0x3b Dec 18 00:50:54 ip-10-195-201-120 kernel: kern_openat(c2d282c0,ffffff9c,284661f0,0,a02,...) at kern_openat+0x128 Dec 18 00:50:54 ip-10-195-201-120 kernel: kern_open(c2d282c0,284661f0,0,a01,1a4,...) at kern_open+0x35 Dec 18 00:50:54 ip-10-195-201-120 kernel: open(c2d282c0,d94e0cfc,d94e0d38,c037d866,0,...) at open+0x30 Dec 18 00:50:54 ip-10-195-201-120 kernel: syscallenter(c2d282c0,d94e0cf4,d94e0cf4,2,c03794f7,...) at syscallenter+0x263 Dec 18 00:50:54 ip-10-195-201-120 kernel: syscall(d94e0d38) at syscall+0x34 Dec 18 00:50:54 ip-10-195-201-120 kernel: Xint0x80_syscall() at Xint0x80_syscall+0x22 Dec 18 00:50:54 ip-10-195-201-120 kernel: --- syscall (5, FreeBSD ELF32, open), eip = 0x2836a353, esp = 0xbf7fe7dc, ebp = 0xbf7fe808 --- System seems to run fine after this though. For example, I left one instance running overnight with no more errors. I have also run portsnap and built a couple ports for testing and saw no more errors or obvious file corruptions. >How-To-Repeat: Boot EC2 AMI built by Colin. AMI ID: ami-c01aeca9 Wait roughly two min's after /sbin/init is started. >Fix: >Release-Note: >Audit-Trail: >Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201012180118.oBI1IpbW018745>