From owner-freebsd-jail@freebsd.org Wed Nov 2 14:16:13 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8987DC2A1E3 for ; Wed, 2 Nov 2016 14:16:13 +0000 (UTC) (envelope-from jamie@gritton.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 2D4F71606 for ; Wed, 2 Nov 2016 14:16:13 +0000 (UTC) (envelope-from jamie@gritton.org) Received: by mailman.ysv.freebsd.org (Postfix) id 2C293C2A1E2; Wed, 2 Nov 2016 14:16:13 +0000 (UTC) Delivered-To: jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2BA3AC2A1E0 for ; Wed, 2 Nov 2016 14:16:13 +0000 (UTC) (envelope-from jamie@gritton.org) Received: from gritton.org (gritton.org [199.192.165.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "www.gritton.org", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C4B7C15FE; Wed, 2 Nov 2016 14:16:12 +0000 (UTC) (envelope-from jamie@gritton.org) Received: from gritton.org (gritton.org [199.192.165.131]) by gritton.org (8.15.2/8.15.2) with ESMTPS id uA2EG4Nq062370 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 2 Nov 2016 08:16:05 -0600 (MDT) (envelope-from jamie@gritton.org) Received: (from www@localhost) by gritton.org (8.15.2/8.15.2/Submit) id uA2EG4eL062369; Wed, 2 Nov 2016 08:16:04 -0600 (MDT) (envelope-from jamie@gritton.org) X-Authentication-Warning: gritton.org: www set sender to jamie@gritton.org using -f To: jail@freebsd.org Subject: Re: Debugging jails in dying state X-PHP-Originating-Script: 0:rcube.php MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Wed, 02 Nov 2016 08:16:04 -0600 From: James Gritton In-Reply-To: <20161102124521.i57bpmp3w3ql333h@ivaldir.etoilebsd.net> References: <20161102124521.i57bpmp3w3ql333h@ivaldir.etoilebsd.net> Message-ID: <1c2ce2d106246aa2b0d00c4c7387489c@gritton.org> X-Sender: jamie@gritton.org User-Agent: Roundcube Webmail/1.2.0 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Nov 2016 14:16:13 -0000 On 2016-11-02 06:45, Baptiste Daroussin wrote: > > Is there a way to debug/trace what a jail is doing in dying state. > > I have a couple of jails that takes very long in dying state even after > all > processes and tcp connections are dead. > > I can't find a way to figure what it is waiting for. > > Any clue? > > By very long I mean up to 20min! A dead prison has a nonzero pr_ref (and a zero pr_uref), so that's what you want to keep an eye on. The functions to change that field are prison_hold[_locked] and prison_free[_locked]. If you're actually running a kernel debugger (which I've never done outside of a crash dump), you should be able to catch a stack trace on prison_free to see who's finally letting the last reference go. It turns out that there are very few places that call these functions on anything besides prison0 (and nothing outside of kern_jail.c twiddles the field directly): moving interfaces around between vimage jails, setting up a zfs zone, and in crcopy/crfree. Its that last one that you'll need to trace, because of course creds are everywhere and anything that holds on to a jail until some future point is doing that by holding on to a cred. So the good news is that you can use whatever tools you already have in your possession to trace creds. And the bad news is that creds are everywhere :-). Aside from TCP connections, NFS seems to be a common dying jail timeout. And perhaps ZFS - I don't recall. - Jamie