From owner-freebsd-stable@FreeBSD.ORG Wed May 18 01:49:33 2011 Return-Path: Delivered-To: stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 07803106566B for ; Wed, 18 May 2011 01:49:33 +0000 (UTC) (envelope-from spork@bway.net) Received: from xena.bway.net (xena.bway.net [216.220.96.26]) by mx1.freebsd.org (Postfix) with ESMTP id 9384E8FC08 for ; Wed, 18 May 2011 01:49:32 +0000 (UTC) Received: (qmail 29702 invoked by uid 0); 18 May 2011 01:49:31 -0000 Received: from smtp.bway.net (216.220.96.25) by xena.bway.net with (DHE-RSA-AES256-SHA encrypted) SMTP; 18 May 2011 01:49:31 -0000 Received: (qmail 29693 invoked by uid 90); 18 May 2011 01:49:31 -0000 Received: from unknown (HELO hotlap.nat.fasttrackmonkey.com) (spork@96.57.144.66) by smtp.bway.net with (DHE-RSA-AES256-SHA encrypted) SMTP; 18 May 2011 01:49:31 -0000 Date: Tue, 17 May 2011 21:49:30 -0400 (EDT) From: Charles Sprickman X-X-Sender: spork@hotlap.nat.fasttrackmonkey.com To: Andriy Gapon In-Reply-To: <4DD26C49.4010108@FreeBSD.org> Message-ID: References: <20110517073029.GA44359@icarus.home.lan> <4DD25264.8040305@FreeBSD.org> <20110517112952.GA48610@icarus.home.lan> <4DD2624A.9080708@FreeBSD.org> <20110517122323.GA49650@icarus.home.lan> <4DD26C49.4010108@FreeBSD.org> User-Agent: Alpine 2.00 (OSX 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: stable@FreeBSD.org, Jeremy Chadwick Subject: Re: 8.1R possible zfs snapshot livelock? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 May 2011 01:49:33 -0000 On Tue, 17 May 2011, Andriy Gapon wrote: > on 17/05/2011 15:23 Jeremy Chadwick said the following: >> So for Charles' situation, the next time it happens what would be useful >> for him to provide? The best I could come up with was to induce doadump >> then reboot to get the system up/working again, and then use kgdb >> after-the-fact. > > This is one of the best things to do, if possible. > In this case all the potentially useful info would be preserved. Will do. Just have to verify I understand the "break to debugger" stuff with serial consoles and ensure it's not easy to accidentally trigger (vauge memories of Sun boxes doing funny things if someone messes with the console). We're also going to start in on upgrading to 8.2 and see what that brings. Still sounds like a good idea to be able to force a dump if things lock up like this again, regardless of what we're running. > Less drastic approach to hanged I/O debugging is to find out where > processes/threads are actually stuck. E.g. using procstat -kk. Odd you say that because we've got an old 32-bit 8.1 box that's running spamassassin and some devel stuff that looks like it's getting a little wedged: PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 6 root 4 -8 - 0K 36K tx->tx 126.0H 76.37% zfskern And I'm not sure "procstat" is meant for this, but the output is interesting: [root@h22 /home/spork]# procstat -k 6 PID TID COMM TDNAME KSTACK 6 100053 zfskern arc_reclaim_thre mi_switch sleepq_switch sleepq_timedwait _cv_timedwait arc_reclaim_thread fork_exit fork_trampoline 6 100054 zfskern l2arc_feed_threa mi_switch sleepq_switch sleepq_timedwait _cv_timedwait l2arc_feed_thread fork_exit fork_trampoline 6 100093 zfskern txg_thread_enter mi_switch sleepq_switch sleepq_wait _cv_wait txg_thread_wait txg_quiesce_thread fork_exit fork_trampoline 6 100094 zfskern txg_thread_enter mi_switch sleepq_switch sleepq_timedwait _cv_timedwait txg_thread_wait txg_sync_thread fork_exit fork_trampoline Makes me curious about the patch mm@ has for 8.2 here: http://blog.vx.sk/archives/24-Backported-patches-for-FreeBSD-82-RELEASE.html (item "c" in the list) Anyhow, thank you *all* for an interesting discussion. We do want to forge ahead with using snapshots since it's a really nice luxury, especially on boxes with lots of jails. Makes it very easy to roll things back without having to go to the tapes. Thanks, Charles > -- > Andriy Gapon > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >