From owner-freebsd-performance@FreeBSD.ORG Thu May 17 17:47:44 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9178616A403; Thu, 17 May 2007 17:47:44 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay01.kiev.sovam.com (relay01.kiev.sovam.com [62.64.120.200]) by mx1.freebsd.org (Postfix) with ESMTP id 22D3313C455; Thu, 17 May 2007 17:47:44 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.227] (helo=fw.zoral.com.ua) by relay01.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.60) (envelope-from ) id 1Hok4k-0000Sr-F3; Thu, 17 May 2007 20:47:43 +0300 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by fw.zoral.com.ua (8.13.4/8.13.4) with ESMTP id l4HHlanA087265 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 17 May 2007 20:47:36 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.1/8.14.1) with ESMTP id l4HHlZpq044397; Thu, 17 May 2007 20:47:35 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.1/8.14.1/Submit) id l4HHlZSt044396; Thu, 17 May 2007 20:47:35 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 17 May 2007 20:47:35 +0300 From: Kostik Belousov To: Andrew Edwards Message-ID: <20070517174735.GB41395@deviant.kiev.zoral.com.ua> References: <20070517170100.GA41395@deviant.kiev.zoral.com.ua> <5230D3C40B842D4F9FB3CD368021BEF72F090C@exchange-2.sandvine.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="9zSXsLTf0vkW971A" Content-Disposition: inline In-Reply-To: <5230D3C40B842D4F9FB3CD368021BEF72F090C@exchange-2.sandvine.com> User-Agent: Mutt/1.4.2.2i X-Virus-Scanned: ClamAV version 0.88.7, clamav-milter version 0.88.7 on fw.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-0.1 required=5.0 tests=ALL_TRUSTED,SPF_NEUTRAL autolearn=failed version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on fw.zoral.com.ua X-Scanner-Signature: 811d4bb63238b2e39a625e79cff8aaca X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Info: Profiles 1053 [May 16 2007] X-SpamTest-Info: helo_type=3 X-SpamTest-Info: {received from trusted relay: not dialup} X-SpamTest-Method: none X-SpamTest-Method: Local Lists X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0255], KAS30/Release X-Mailman-Approved-At: Thu, 17 May 2007 23:02:57 +0000 Cc: freebsd-fs@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Ufs dead-locks on freebsd 6.2 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 May 2007 17:47:44 -0000 --9zSXsLTf0vkW971A Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, May 17, 2007 at 01:03:37PM -0400, Andrew Edwards wrote: > Here it is. >=20 > db> show vnode 0xccd47984 > vnode 0xccd47984: tag ufs, type VDIR > usecount 5135, writecount 0, refcount 5137 mountedhere 0 > flags (VV_ROOT) > v_object 0xcd02518c ref 0 pages 1 > #0 0xc0593f0d at lockmgr+0x4ed > #1 0xc06b8e0e at ffs_lock+0x76 > #2 0xc0739787 at VOP_LOCK_APV+0x87 > #3 0xc0601c28 at vn_lock+0xac > #4 0xc05ee832 at lookup+0xde > #5 0xc05ee4b2 at namei+0x39a > #6 0xc05e2ab0 at unp_connect+0xf0 > #7 0xc05e1a6a at uipc_connect+0x66 > #8 0xc05d9992 at soconnect+0x4e > #9 0xc05dec60 at kern_connect+0x74 > #10 0xc05debdf at connect+0x2f > #11 0xc0723e2b at syscall+0x25b > #12 0xc070ee0f at Xint0x80_syscall+0x1f >=20 > ino 2, on dev amrd0s1a=20 It seems to be the sort of things that cannot happen. VOP_LOCK() returned 0, but vnode was not really locked. Although claiming that kernel code cannot have such bug is too optimistic, I would first make sure that: 1. You checked the memory of the machine. 2. Your kernel is built from pristine sources. >=20 > > -----Original Message----- > > From: Kostik Belousov [mailto:kostikbel@gmail.com]=20 > > Sent: Thursday, May 17, 2007 1:01 PM > > To: Andrew Edwards > > Cc: freebsd-performance@freebsd.org; freebsd-fs@freebsd.org > > Subject: Re: Ufs dead-locks on freebsd 6.2 > >=20 > > On Thu, May 17, 2007 at 11:44:15AM -0400, Andrew Edwards wrote: > > > I've upgraded to 6-stable, added the kernel options as per=20 > > the kernel=20 > > > handbook. After about 5 hours of they system in a deadlock=20 > > it panic'd. > > > Here's the backtrace, and show pcpu, show allpcpu, show locks, show= =20 > > > alllocks, show lockedvnods and alltrace. > > >=20 > > > I will have the system down for approx another 15-20mins if there's= =20 > > > anything else someone would like while I'm in the debugger. > > >=20 > > >=20 > > > db> bt > > > Tracing pid 46784 tid 105112 td 0xd44a8000 > > > kdb_enter(c0785f13) at kdb_enter+0x2b > > > vfs_badlock(c0785f2c,c0786051,ccd47984) at vfs_badlock+0x47 > > > assert_vop_locked(ccd47984,c0786051) at assert_vop_locked+0x4a > > > vop_lock_post(f9f709dc,0,1002,ccd47984,f9f709f8,...) at=20 > > > vop_lock_post+0x2a > > > VOP_LOCK_APV(c07dc2e0,f9f709dc) at VOP_LOCK_APV+0xa0 > > > vn_lock(ccd47984,1002,d44a8000) at vn_lock+0xac > > > lookup(f9f70c08) at lookup+0xde > > > namei(f9f70c08) at namei+0x39a > > > unp_connect(d44b2de8,d44dc380,d44a8000,d44b2de8,25,...) at=20 > > > unp_connect+0xf0 > > > uipc_connect(d44b2de8,d44dc380,d44a8000) at uipc_connect+0x66 > > > soconnect(d44b2de8,d44dc380,d44a8000) at soconnect+0x4e > > > kern_connect(d44a8000,7,d44dc380,d44dc380,0,...) at=20 > > kern_connect+0x74 > > > connect(d44a8000,f9f70d04) at connect+0x2f > > > syscall(3b,805003b,bfbf003b,bfbfd920,bfbfd922,...) at syscall+0x25b > > > Xint0x80_syscall() at Xint0x80_syscall+0x1f > >=20 > > Could you, please, do the "show vnode 0xccd47984" from ddb=20 > > prompt or "p/x *(struct vnode *)0xccd47984" from kgdb using=20 > > dump for this panic ? > >=20 > > Side note: it seems that on HEAD, vnode_if.awk does not=20 > > generate call to vop_lock_{pre,post} due to mismatch in the=20 > > name of vop and pre/post names. > >=20 > >=20 > >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" --9zSXsLTf0vkW971A Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGTJU2C3+MBN1Mb4gRAi/sAKC7GVHcLs8Lq1rUKs/VL2JIk01atACfemRU vELK27Wc9MkcDj6Mn2mJRHQ= =h+0C -----END PGP SIGNATURE----- --9zSXsLTf0vkW971A--