From owner-freebsd-questions@FreeBSD.ORG Thu Mar 15 10:07:53 2007 Return-Path: X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5E74D16A400 for ; Thu, 15 Mar 2007 10:07:53 +0000 (UTC) (envelope-from ml.diespammer@netfence.it) Received: from parrot.aev.net (parrot.aev.net [212.31.247.179]) by mx1.freebsd.org (Postfix) with ESMTP id D23F913C44B for ; Thu, 15 Mar 2007 10:07:52 +0000 (UTC) (envelope-from ml.diespammer@netfence.it) Received: from soth.ventu ([151.77.253.161]) (authenticated bits=128) by parrot.aev.net (8.14.0/8.13.8) with ESMTP id l2FAEZqX008868 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Thu, 15 Mar 2007 11:14:41 +0100 (CET) (envelope-from ml.diespammer@netfence.it) Received: from [10.1.2.18] (alamar.ventu [10.1.2.18]) by soth.ventu (8.14.0/8.13.8) with ESMTP id l2FA7pTf081699 for ; Thu, 15 Mar 2007 11:07:51 +0100 (CET) (envelope-from ml.diespammer@netfence.it) Message-ID: <45F91AE9.1090006@netfence.it> Date: Thu, 15 Mar 2007 11:07:37 +0100 From: Andrea Venturoli User-Agent: Thunderbird 1.5.0.10 (X11/20070306) MIME-Version: 1.0 To: freebsd-questions@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.61 on 212.31.247.179 Subject: Server hanged on VFS lock problem X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-questions@freebsd.org List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Mar 2007 10:07:53 -0000 Hello. Yesterday a 6.2/amd64 SMP server of mine entered DDB after a problem with locks. Since I was not there, I instructed one user to type "panic" and I got a dump. Here's what I get: > xxxxx# kgdb kernel.debug /var/crash/vmcore.0 > [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "amd64-marcel-freebsd". > > Unread portion of the kernel message buffer: > KDB: stack backtrace: > vfs_badlock() at vfs_badlock+0x95 > assert_vop_elocked() at assert_vop_elocked+0x7d > VOP_WRITE_APV() at VOP_WRITE_APV+0xee > vn_write() at vn_write+0x1ec > dofilewrite() at dofilewrite+0x87 > kern_writev() at kern_writev+0x51 > write() at write+0x4a > syscall() at syscall+0x4d1 > Xfast_syscall() at Xfast_syscall+0xa8 > --- syscall (4, FreeBSD ELF64, write), rip = 0x8017ecbac, rsp = 0x7fffffffb568, rbp = 0x8054079c0 --- > VOP_WRITE: 0xffffff00097f1ca8 is not exclusive locked but should be > KDB: enter: lock violation > Locked vnodes > > 0xffffff00097f1ca8: tag ufs, type VREG > usecount 5, writecount 2, refcount 560 mountedhere 0 > flags () > v_object 0xffffff0002f65000 ref 3 pages 2216 > lock type ufs: SHARED (count 1)#0 0xffffffff80238786 at lockmgr+0x5f6 > #1 0xffffffff8033d558 at ffs_lock+0x58 > #2 0xffffffff803b4df1 at VOP_LOCK_APV+0x81 > #3 0xffffffff802b80cb at vn_lock+0x6b > #4 0xffffffff802b92c6 at vn_write+0x156 > #5 0xffffffff80271b37 at dofilewrite+0x87 > #6 0xffffffff80271e01 at kern_writev+0x51 > #7 0xffffffff80271efa at write+0x4a > #8 0xffffffff803854a1 at syscall+0x4d1 > #9 0xffffffff80370b78 at Xfast_syscall+0xa8 > > ino 1078094, on dev mirror/gm0s1e > panic: from debugger > cpuid = 0 > Uptime: 13d23h8m15s > Dumping 1023 MB (2 chunks) > chunk 0: 1MB (151 pages) ... ok > chunk 1: 1023MB (261744 pages) 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 > > #0 doadump () at pcpu.h:172 > 172 __asm __volatile("movq %%gs:0,%0" : "=r" (td)); > (kgdb) Any hint? Is there a way I can get these dumps automatically, without entering DDB, since this is an unattended server? I have this options in my kernel configuration: options KDB options DDB options KDB_UNATTENDED options INVARIANTS options INVARIANT_SUPPORT options WITNESS options DEBUG_LOCKS options DEBUG_VFS_LOCKS options DIAGNOSTIC bye & Thanks av.