From owner-freebsd-questions@freebsd.org Fri Apr 28 13:53:21 2017 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 082BFD54A07 for ; Fri, 28 Apr 2017 13:53:21 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from smtp.fagskolen.gjovik.no (smtp.fagskolen.gjovik.no [IPv6:2001:700:1100:1:200:ff:fe00:b]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.fagskolen.gjovik.no", Issuer "Fagskolen i Gj??vik" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id A203DF69 for ; Fri, 28 Apr 2017 13:53:20 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from mail.fig.ol.no (localhost [127.0.0.1]) by mail.fig.ol.no (8.15.2/8.15.2) with ESMTPS id v3SDqx5q065946 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 28 Apr 2017 15:52:59 +0200 (CEST) (envelope-from trond@fagskolen.gjovik.no) Received: from localhost (trond@localhost) by mail.fig.ol.no (8.15.2/8.15.2/Submit) with ESMTP id v3SDqxgw065943; Fri, 28 Apr 2017 15:52:59 +0200 (CEST) (envelope-from trond@fagskolen.gjovik.no) X-Authentication-Warning: mail.fig.ol.no: trond owned process doing -bs Date: Fri, 28 Apr 2017 15:52:59 +0200 (CEST) From: =?ISO-8859-1?Q?Trond_Endrest=F8l?= Sender: Trond.Endrestol@fagskolen.gjovik.no To: Paul Macdonald cc: freebsd-questions@freebsd.org Subject: Re: A seperate ZFS question In-Reply-To: Message-ID: References: User-Agent: Alpine 2.20 (BSF 67 2015-01-07) Organization: Fagskolen Innlandet OpenPGP: url=http://fig.ol.no/~trond/trond.key MIME-Version: 1.0 X-Spam-Status: No, score=-2.2 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail.fig.ol.no Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Apr 2017 13:53:21 -0000 On Fri, 28 Apr 2017 14:14+0100, Paul Macdonald wrote: > > I fat fingered adding a cache drive and was unable to remove the SSD cache > drive (ada2) > > I added a 2nd SSD (ada3) for safety as i didn;t want to risk the pool at all > and exporting wasn;t feasible in timescale. > > Can someone explain : > > what i've actually managed to set up here (!), > > the ramifications of forceably offlining/removing ada2 > > and if i should worry when ada2 gets full? > > capacity operations bandwidth > pool alloc free read write read write > ------------ ----- ----- ----- ----- ----- ----- > zrootnew 692G 460G 15 174 351K 2.40M > mirror 683G 237G 14 138 328K 1.43M > gpt/zfs0 - - 6 50 155K 1.44M > ada1p3 - - 7 50 176K 1.44M > ada2 9.17G 223G 0 36 22.9K 989K > cache - - - - - - > ada3 39.0G 194G 23 5 168K 264K First, you have gpt/zfs0 and ada1p3 in a mirrored configuration. Next, you have ada2 striped (RAID 0) to the first pair of mirrors. This is a very weird "RAID 1 + 0" combo. Guess what will happen when you lose ada2! Maybe you should describe what you wanted to happen in the first place. If you really wanted a three-way mirror, you have just learned the hard way about the difference between zpool add and zpool attach. The latter is for adding additional members to existing mirrors, while the former is for adding raidz{1,2,3} groups to a pool, and the add command got you into this mess. "zpool attach zrootnew ada1p3 ada2" is the right command if you wanted to create a three-way mirror in the first place. I suggest you make good backups of this pool, destroy it, recreate it properly, and restore the missing data. -- +-------------------------------+------------------------------------+ | Vennlig hilsen, | Best regards, | | Trond Endrestøl, | Trond Endrestøl, | | IT-ansvarlig, | System administrator, | | Fagskolen Innlandet, | Gjøvik Technical College, Norway, | | tlf. mob. 952 62 567, | Cellular...: +47 952 62 567, | | sentralbord 61 14 54 00. | Switchboard: +47 61 14 54 00. | +-------------------------------+------------------------------------+ From owner-freebsd-questions@freebsd.org Fri Apr 28 14:28:44 2017 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 67E82D5322B for ; Fri, 28 Apr 2017 14:28:44 +0000 (UTC) (envelope-from ultima1252@gmail.com) Received: from mail-yb0-x229.google.com (mail-yb0-x229.google.com [IPv6:2607:f8b0:4002:c09::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 25E421F89 for ; Fri, 28 Apr 2017 14:28:44 +0000 (UTC) (envelope-from ultima1252@gmail.com) Received: by mail-yb0-x229.google.com with SMTP id 8so15840855ybw.1 for ; Fri, 28 Apr 2017 07:28:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=mMrJvqJDFsI3bl1BNQB+T7PkNiH2+HEOzHuIqO4wE/k=; b=huvzfyJqN7yYCHu8XTHbKDOARqMH6X92KMAeil6R71UwGoApqsUE3GOIVYiOBg1P+J fzNR8Pytr+vlLGqzvWFiKce0+HmnY71Nb7l/hNphxXGRrR5/6IhfEQuCgoLOtDDl3RrN NRm9LoWnaJ67u6A6erAxISok1kiz5pt9PrOQhkCeMu3U/TAsYeMTWfGUibaznodeciGF rouc8QsDvEhwPeA15QwzqBMgQVzbRxj/pgVuw6lV/7Xtl5cR8GP3dZ2yrfL84GSMXqZy yxMjv1Mma6rNj305Yfnv/Yzk6R+h+K3MJgRR+AvMpygXlPd0onW5dp8NGHnoCr6nRWni JkXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=mMrJvqJDFsI3bl1BNQB+T7PkNiH2+HEOzHuIqO4wE/k=; b=KYyHNMvVcb29d7gFKIdWbeZOu8vuGIUk5F4Y3AIu6X1+00qoq/Pru2TfYsGkh/ZiCd Nr0nDhR59tn83el+khOYQZE0J4miPB5THIb33lADqGLFNH87E0YdDhwrdjusp4kWgwdv DZrnSNp/0dnunUEgqwcv7WPuRGlrR4KgOh5H4G7G2KAHULWsIMfTyziAkStDhXA9h8I0 gORzD2J4La/dD9v3eQfXEQ4LCIgwmpDxa2BMwkz1xPctEt2o6QcJ0ynN5d5xT/cj4KLN jQp2d5HmU0F8XMZeHFJPPcEq7AnCY3Uwmbw3b6HwPiDu/8TmYPvv08z7MS2GopxP4SRA Xc2w== X-Gm-Message-State: AN3rC/7z+gmIO8JBAWlWdkzIkitaKrxAQeG/VkY5ZAkcpIvxymxi+HpI +FWDSaGtchbol7WNLvrRCzVsjklzai2cKmI= X-Received: by 10.37.1.4 with SMTP id 4mr9505859ybb.91.1493389723175; Fri, 28 Apr 2017 07:28:43 -0700 (PDT) MIME-Version: 1.0 Received: by 10.129.72.144 with HTTP; Fri, 28 Apr 2017 07:28:42 -0700 (PDT) In-Reply-To: References: From: Ultima Date: Fri, 28 Apr 2017 10:28:42 -0400 Message-ID: Subject: Re: A seperate ZFS question To: Paul Macdonald Cc: FreeBSD Mailing List Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Apr 2017 14:28:44 -0000 -n is your friend. Never used zpool add without it to verify changes are as you intend. Not using it first is similar to jumping off a plane without verifying your parachute is on properly. On Fri, Apr 28, 2017 at 9:52 AM, Trond Endrest=C3=B8l < Trond.Endrestol@fagskolen.gjovik.no> wrote: > On Fri, 28 Apr 2017 14:14+0100, Paul Macdonald wrote: > > > > > I fat fingered adding a cache drive and was unable to remove the SSD > cache > > drive (ada2) > > > > I added a 2nd SSD (ada3) for safety as i didn;t want to risk the pool a= t > all > > and exporting wasn;t feasible in timescale. > > > > Can someone explain : > > > > what i've actually managed to set up here (!), > > > > the ramifications of forceably offlining/removing ada2 > > > > and if i should worry when ada2 gets full? > > > > capacity operations bandwidth > > pool alloc free read write read write > > ------------ ----- ----- ----- ----- ----- ----- > > zrootnew 692G 460G 15 174 351K 2.40M > > mirror 683G 237G 14 138 328K 1.43M > > gpt/zfs0 - - 6 50 155K 1.44M > > ada1p3 - - 7 50 176K 1.44M > > ada2 9.17G 223G 0 36 22.9K 989K > > cache - - - - - - > > ada3 39.0G 194G 23 5 168K 264K > > First, you have gpt/zfs0 and ada1p3 in a mirrored configuration. Next, > you have ada2 striped (RAID 0) to the first pair of mirrors. This is a > very weird "RAID 1 + 0" combo. Guess what will happen when you lose > ada2! > > Maybe you should describe what you wanted to happen in the first > place. > > If you really wanted a three-way mirror, you have just learned the > hard way about the difference between zpool add and zpool attach. > > The latter is for adding additional members to existing mirrors, while > the former is for adding raidz{1,2,3} groups to a pool, and the add > command got you into this mess. > > "zpool attach zrootnew ada1p3 ada2" is the right command if you wanted > to create a three-way mirror in the first place. > > I suggest you make good backups of this pool, destroy it, recreate it > properly, and restore the missing data. > > -- > +-------------------------------+------------------------------------+ > | Vennlig hilsen, | Best regards, | > | Trond Endrest=C3=B8l, | Trond Endrest=C3=B8l, = | > | IT-ansvarlig, | System administrator, | > | Fagskolen Innlandet, | Gj=C3=B8vik Technical College, Norway, = | > | tlf. mob. 952 62 567, | Cellular...: +47 952 62 567, | > | sentralbord 61 14 54 00. | Switchboard: +47 61 14 54 00. | > +-------------------------------+------------------------------------+ > _______________________________________________ > freebsd-questions@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions- > unsubscribe@freebsd.org" >