From owner-freebsd-questions@freebsd.org Sun May 3 14:06:52 2020 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EDB2A2DE7C5 for ; Sun, 3 May 2020 14:06:52 +0000 (UTC) (envelope-from ipluta@wp.pl) Received: from mx3.wp.pl (mx3.wp.pl [212.77.101.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49FSTl2h39z3FdZ for ; Sun, 3 May 2020 14:06:50 +0000 (UTC) (envelope-from ipluta@wp.pl) Received: (wp-smtpd smtp.wp.pl 29924 invoked from network); 3 May 2020 16:06:48 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=wp.pl; s=1024a; t=1588514808; bh=Iv9FzNeJT4/qFUYUV/B56UX+CWDhye9aw3WYVQLny+I=; h=Subject:To:Cc:From; b=RY46o5mVeRCJvRXh/ZJBSPu1nCjpuRylvCnAL4xKMKw7imo/YLlmQk761W9Oapvt+ DEQefGamQlOiV5649Q4TjwikV/UuM/B5H5WrFUvMvaoq7JvZt36CIxhDN5BVxU2DIw P/moW4d12409tn3VorFrji1QG92AYxi12LWsXid8= Received: from aayy197.neoplus.adsl.tpnet.pl (HELO [10.0.0.81]) (ipluta@wp.pl@[83.6.136.197]) (envelope-sender ) by smtp.wp.pl (WP-SMTPD) with ECDHE-RSA-AES256-GCM-SHA384 encrypted SMTP for ; 3 May 2020 16:06:48 +0200 Subject: Re: How to get rid of an unavailable pool? To: =?UTF-8?Q?Trond_Endrest=c3=b8l?= Cc: freebsd-questions@freebsd.org References: <32264a1f-3bcf-9d74-603d-c201bffd256c@wp.pl> From: "Ireneusz Pluta/wp.pl" Message-ID: <092e9379-37cf-f839-e4e4-eeb1e8821f1a@wp.pl> Date: Sun, 3 May 2020 16:05:25 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: pl X-WP-MailID: 86b5c01729ca29dce6e0ec0cd811570c X-WP-AV: skaner antywirusowy Poczty Wirtualnej Polski X-WP-SPAM: NO 0000000 [QcO0] X-Rspamd-Queue-Id: 49FSTl2h39z3FdZ X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=wp.pl header.s=1024a header.b=RY46o5mV; dmarc=pass (policy=none) header.from=wp.pl; spf=pass (mx1.freebsd.org: domain of ipluta@wp.pl designates 212.77.101.9 as permitted sender) smtp.mailfrom=ipluta@wp.pl X-Spamd-Result: default: False [-2.00 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:212.77.96.0/19]; FREEMAIL_FROM(0.00)[wp.pl]; DKIM_TRACE(0.00)[wp.pl:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[wp.pl,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; SUBJECT_ENDS_QUESTION(1.00)[]; FREEMAIL_ENVFROM(0.00)[wp.pl]; ASN(0.00)[asn:12827, ipnet:212.77.101.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[wp.pl.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[wp.pl:s=1024a]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; IP_SCORE_FREEMAIL(0.00)[]; IP_SCORE(0.00)[ip: (-7.50), ipnet: 212.77.101.0/24(-3.95), asn: 12827(-2.85), country: PL(0.06)]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[9.101.77.212.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[9.101.77.212.rep.mailspike.net : 127.0.0.17]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 May 2020 14:06:53 -0000 W dniu 2020-05-02 o 10:03, Trond Endrestøl pisze: > On Sat, 2 May 2020 06:15+0200, Ireneusz Pluta wrote: > >> Hi group, >> >> (Sorry if this post appears twice. The first one, initially sent from another >> email account, does not seem to appear.) >> >> I have (or rather had) a pool like this: >> >> $ sudo zpool status -v t >>   pool: t >>  state: UNAVAIL >> status: One or more devices are faulted in response to IO failures. >> action: Make sure the affected devices are connected, then run 'zpool clear'. >>    see: http://illumos.org/msg/ZFS-8000-HC >>   scan: none requested >> config: >> >>         NAME                     STATE     READ WRITE CKSUM >>         t                        UNAVAIL      0     0 0 >>           mirror-0               UNAVAIL      0     0 0 >>             4304281762335857859  REMOVED      0     0 0  was /dev/da5 >>             1909766900844089131  REMOVED      0     0 0  was /dev/da10 >> >> errors: Permanent errors have been detected in the following files: >> >>         :<0x0> >>         :<0x1b> >>         t:<0x0> >> >> That was a temporary test pool. I forgot to destroy  or at least export the >> pool before pulling these da5 and da10 drives out of the drivebay of the >> server. Now it can't be exported or destroyed, the respective zpool operations >> hust hang. How to get rid now of this pool, preferably without reboot? The da5 >> and da10 are no longer available to be put back, as they have been already >> moved elsewhere, and are now part of another pool. >> >> I guess the pool got stuck at the time of running >> /etc/periodic/security/100.chksetuid, when find operation within it tried to >> traverse into the mountpoint of the pool. >> >> The system is FreeBSD 11.2. >> >> Thanks >> >> Irek > The pool might still be listed in /boot/zfs/zpool.cache. The only way > I can think of to get rid of the old pool, is to delete this file and > reboot. If you have more pools than your root pool, you should reboot > to singleuser mode, mount the root fs read-write, import the > remaining pools, and either exit the SUS shell or reboot. Trond, thank you for your advice. Yes, that state was unrecoverable without reboot. Additionally I found this little thread https://www.databaseusers.com/article/5971869/Cannot+export+%27backup%27%3A+pool+I+O+is+currently+suspended, whose last post helped me a lot with understanding what was going on under the hood, and why. So I followed the procedure carefully, taking special care of first stopping important applications and unmounting other big and valuable datasets. Forced hard reset was necessary, the reboot command just froze. However, there was one exception: I skipped deleting  /boot/zfs/zpool.cache, to avoid falling into single user mode and importing my pools manually (I felt very uncomfortable going to do that remotely, with that crappy IPMIView console redirection). The system booted cleanly with all pools imported. The UNAVAIL pool got imported too, however, it did not get mounted, so there was no chance of any I/O attempt to it. The first thing I did after login was: `zpool destroy t`, which worked cleanly. Prior to doing all that, I reproduced that state and excercised the procedure on a virtual machine. Thanks again Irek