From owner-freebsd-current@FreeBSD.ORG Wed May 27 17:58:50 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8F60D10656CC for ; Wed, 27 May 2009 17:58:50 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.241]) by mx1.freebsd.org (Postfix) with ESMTP id 4B27C8FC08 for ; Wed, 27 May 2009 17:58:50 +0000 (UTC) (envelope-from artemb@gmail.com) Received: by an-out-0708.google.com with SMTP id c3so2570986ana.13 for ; Wed, 27 May 2009 10:58:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:date :x-google-sender-auth:message-id:subject:from:to:content-type :content-transfer-encoding; bh=0rEX/oi4OWkY/mRNWyGlq/6mENFMS9E2zY5ur4sM3ow=; b=Hk0M3/gCwUp7aL3y9BleazJSxnhU4lz2RPKYbXi+mvYt+g5xLfwdwyb57vDRpNGBVi EHQi4iFbmMMXMy1EpNFNvNu1ChQygIUiS9iypdoqnxmkTRNJqBaTiwXGQyYy0UkTk5Td bM4FGWnf8T3RdUDsjrn9DxvxozDmouPVRIL4c= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:content-type:content-transfer-encoding; b=Y7aa5j88W6MBAiXNLW0Nu6CSjHKaLvZ5e+MJp9O1TMAGLAQZPRDeF4oRN7RYHtys/K SKyYD45Znf3lTY3Sv+/tBuL7OxbURzfQ984xswtwY2tYyGkRbL64vIkTZR7ZLxvD2TQw XyLdPnLT4DPPfvPHs0NUbkfV/Ppd6GcZT7Awg= MIME-Version: 1.0 Sender: artemb@gmail.com Received: by 10.100.215.12 with SMTP id n12mr397359ang.154.1243447129593; Wed, 27 May 2009 10:58:49 -0700 (PDT) Date: Wed, 27 May 2009 10:58:49 -0700 X-Google-Sender-Auth: 779a1d541afea2b5 Message-ID: From: Artem Belevich To: freebsd-current@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: ZFS : panic("sleeping thread") X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 May 2009 17:58:51 -0000 Hi, While recent ZFS improvements got rid of random hangs I used to see, there's still one problem that I keep running into -- panic in ZFS under heavy load. I can reproduce it by doing a build with -j16 in a jail running i386 binaries on -CURRENT/amd64 running on a box with quad-core CPU. It takes a while to reproduce, but it usually shows up within couple of hours. Sleeping thread (tid 100606, pid 32147) owns a non-sleepable lock sched_switch() at sched_switch+0xed mi_switch() at mi_switch+0x16f sleepq_wait() at sleepq_wait+0x42 _sx_xlock_hard() at _sx_xlock_hard+0x1f0 _sx_xlock() at _sx_xlock+0x4e rrw_exit() at rrw_exit+0x1d zfs_freebsd_getattr() at zfs_freebsd_getattr+0x2be VOP_GETATTR_APV() at VOP_GETATTR_APV+0x44 filt_vfsread() at filt_vfsread+0x51 knote() at knote+0xc2 VOP_WRITE_APV() at VOP_WRITE_APV+0x11f vn_write() at vn_write+0x279 dofilewrite() at dofilewrite+0x85 kern_writev() at kern_writev+0x60 write() at write+0x54 ia32_syscall() at ia32_syscall+0x236 Xint0x80_syscall() at Xint0x80_syscall+0x85 --- syscall (4, FreeBSD ELF32, write), rip = 0x78162153, rsp = 0xffff945c, rbp = 0xffff9478 --- It appears that locking within ZFS conflicts with vnode locking. The back-trace is always the same. For now, I've applied following patch to disable the panic, but it would be good if someone familiar with VFS locking in FreeBSD could take a look. If you need any additional info, let me know. Thanks, --Artem diff -r 930d975c8103 src/sys/kern/subr_turnstile.c --- a/sys/kern/subr_turnstile.c Fri Dec 05 16:12:43 2008 -0800 +++ b/sys/kern/subr_turnstile.c Fri Dec 12 14:31:16 2008 -0800 @@ -219,7 +219,10 @@ #ifdef DDB db_trace_thread(td, -1); #endif - panic("sleeping thread"); + /* Don't propagate priority to a sleeping thread. */ + thread_unlock(td); + return; + // panic("sleeping thread"); } /*