From owner-freebsd-fs@freebsd.org Wed Feb 26 20:26:42 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 044BC243D52 for ; Wed, 26 Feb 2020 20:26:42 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [IPv6:2001:6b0:17:f0a0::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48SS4t2FMHz3Htl for ; Wed, 26 Feb 2020 20:26:37 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id 284BE4000F for ; Wed, 26 Feb 2020 21:26:33 +0100 (CET) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id 0D09C40010; Wed, 26 Feb 2020 21:26:32 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on bernadotte.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,AWL autolearn=disabled version=3.4.2 X-Spam-Score: -1.0 Received: from [192.168.1.132] (h-201-140.A785.priv.bahnhof.se [98.128.201.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 888D14000F for ; Wed, 26 Feb 2020 21:26:32 +0100 (CET) From: Peter Eriksson Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: ZFS pools in "trouble" Date: Wed, 26 Feb 2020 21:26:32 +0100 References: <71e1f22a-1261-67d9-e41d-0f326bf81469@digiware.nl> To: FreeBSD Filesystems In-Reply-To: <71e1f22a-1261-67d9-e41d-0f326bf81469@digiware.nl> Message-Id: <2A6BA1B1-35C7-42E3-BF03-7BF95B7825D8@lysator.liu.se> X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 48SS4t2FMHz3Htl X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of pen@lysator.liu.se designates 2001:6b0:17:f0a0::3 as permitted sender) smtp.mailfrom=pen@lysator.liu.se X-Spamd-Result: default: False [-4.06 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.lysator.liu.se]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[4]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; RCVD_IN_DNSWL_NONE(0.00)[3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.a.0.f.7.1.0.0.0.b.6.0.1.0.0.2.list.dnswl.org : 127.0.11.0]; MV_CASE(0.50)[]; IP_SCORE(-1.76)[ip: (-6.90), ipnet: 2001:6b0::/32(-1.04), asn: 1653(-0.83), country: EU(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:1653, ipnet:2001:6b0::/32, country:EU]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Feb 2020 20:26:42 -0000 What type of hardware are you using? Server, controllers and disks? - Peter > On 26 Feb 2020, at 18:09, Willem Jan Withagen wrote: >=20 > Hi, >=20 > I'm using my pools in perhaps a rather awkward way as underlying = storage for my ceph cluster: > 1 disk per pool, with log and cache on SSD >=20 > For one reason or another one of the servers has crashed ad does not = really want to read several of the pools: > ---- > pool: osd_2 > state: UNAVAIL > Assertion failed: (reason =3D=3D ZPOOL_STATUS_OK), file = /usr/src/cddl/contrib/opensolaris/cmd/zpool/zpool_main.c, line 5098. > Abort (core dumped) > ---- >=20 > The code there is like: > ---- > default: > /* > * The remaining errors can't actually be generated, = yet. > */ > assert(reason =3D=3D ZPOOL_STATUS_OK); >=20 > ---- > And this on already 3 disks. > Running: > FreeBSD 12.1-STABLE (GENERIC) #0 r355208M: Fri Nov 29 10:43:47 CET = 2019 >=20 > Now this is a test cluster, so no harm there in matters of data loss. > And the ceph cluster probably can rebuild everything if I do not lose = too many disk. >=20 > But the problem also lies in the fact that not all disk are recognized = by the kernel, and not all disk end up mounted. So I need to remove a = pool first to get more disks online. >=20 > Is there anything I can do the get them back online? > Or is this a lost cause? >=20 > --WjW > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"