From owner-freebsd-questions@freebsd.org Fri Apr 28 15:04:46 2017 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7BD4CD5418A for ; Fri, 28 Apr 2017 15:04:46 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from smtp.fagskolen.gjovik.no (smtp.fagskolen.gjovik.no [IPv6:2001:700:1100:1:200:ff:fe00:b]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.fagskolen.gjovik.no", Issuer "Fagskolen i Gj??vik" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 056331AAB for ; Fri, 28 Apr 2017 15:04:45 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from mail.fig.ol.no (localhost [127.0.0.1]) by mail.fig.ol.no (8.15.2/8.15.2) with ESMTPS id v3SF4fah066317 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 28 Apr 2017 17:04:41 +0200 (CEST) (envelope-from trond@fagskolen.gjovik.no) Received: from localhost (trond@localhost) by mail.fig.ol.no (8.15.2/8.15.2/Submit) with ESMTP id v3SF4fT2066314; Fri, 28 Apr 2017 17:04:41 +0200 (CEST) (envelope-from trond@fagskolen.gjovik.no) X-Authentication-Warning: mail.fig.ol.no: trond owned process doing -bs Date: Fri, 28 Apr 2017 17:04:41 +0200 (CEST) From: =?ISO-8859-1?Q?Trond_Endrest=F8l?= Sender: Trond.Endrestol@fagskolen.gjovik.no To: Paul Macdonald cc: FreeBSD Mailing List Subject: Re: A seperate ZFS question In-Reply-To: Message-ID: References: User-Agent: Alpine 2.20 (BSF 67 2015-01-07) Organization: Fagskolen Innlandet OpenPGP: url=http://fig.ol.no/~trond/trond.key MIME-Version: 1.0 Content-ID: X-Spam-Status: No, score=-2.2 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail.fig.ol.no Content-Type: text/plain; CHARSET=ISO-8859-1 Content-Transfer-Encoding: 8BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Apr 2017 15:04:46 -0000 On Fri, 28 Apr 2017 10:28-0400, Ultima wrote: > -n is your friend. Never used zpool add without it to verify changes are as > you intend. Not using it first is similar to jumping off a plane without > verifying your parachute is on properly. Not to mention using -v. Also, the OP can use: zfs snap -r zrootnew@transfer zfs send -RLev zrootnew@transfer | {ssh,nc,something-something} or zfs send -RLev zrootnew@transfer > /some/file/somewhere for creating a replicated stream preserving both data and metadata. Use these commands for restoring the missing data: {ssh,nc,something-something} | zfs recv -Fduv newpool or zfs recv -Fduv newpool < /some/file/somewhere If all drives are connected to the same system, the replication can be simplified to: zfs send -RLev zrootnew@transfer | zfs recv -Fduv newpool Remember to remove the recursive @transfer snapshot afterwards. To preview what will happen: zfs destroy -nRv newpool@transfer To go ahead and make the plunge: zfs destroy -Rv newpool@transfer Why are we top posting all of a sudden? > On Fri, Apr 28, 2017 at 9:52 AM, Trond Endrestøl < > Trond.Endrestol@fagskolen.gjovik.no> wrote: > > > On Fri, 28 Apr 2017 14:14+0100, Paul Macdonald wrote: > > > > > > > > I fat fingered adding a cache drive and was unable to remove the SSD > > cache > > > drive (ada2) > > > > > > I added a 2nd SSD (ada3) for safety as i didn;t want to risk the pool at > > all > > > and exporting wasn;t feasible in timescale. > > > > > > Can someone explain : > > > > > > what i've actually managed to set up here (!), > > > > > > the ramifications of forceably offlining/removing ada2 > > > > > > and if i should worry when ada2 gets full? > > > > > > capacity operations bandwidth > > > pool alloc free read write read write > > > ------------ ----- ----- ----- ----- ----- ----- > > > zrootnew 692G 460G 15 174 351K 2.40M > > > mirror 683G 237G 14 138 328K 1.43M > > > gpt/zfs0 - - 6 50 155K 1.44M > > > ada1p3 - - 7 50 176K 1.44M > > > ada2 9.17G 223G 0 36 22.9K 989K > > > cache - - - - - - > > > ada3 39.0G 194G 23 5 168K 264K > > > > First, you have gpt/zfs0 and ada1p3 in a mirrored configuration. Next, > > you have ada2 striped (RAID 0) to the first pair of mirrors. This is a > > very weird "RAID 1 + 0" combo. Guess what will happen when you lose > > ada2! > > > > Maybe you should describe what you wanted to happen in the first > > place. > > > > If you really wanted a three-way mirror, you have just learned the > > hard way about the difference between zpool add and zpool attach. > > > > The latter is for adding additional members to existing mirrors, while > > the former is for adding raidz{1,2,3} groups to a pool, and the add > > command got you into this mess. > > > > "zpool attach zrootnew ada1p3 ada2" is the right command if you wanted > > to create a three-way mirror in the first place. > > > > I suggest you make good backups of this pool, destroy it, recreate it > > properly, and restore the missing data. -- +-------------------------------+------------------------------------+ | Vennlig hilsen, | Best regards, | | Trond Endrestøl, | Trond Endrestøl, | | IT-ansvarlig, | System administrator, | | Fagskolen Innlandet, | Gjøvik Technical College, Norway, | | tlf. mob. 952 62 567, | Cellular...: +47 952 62 567, | | sentralbord 61 14 54 00. | Switchboard: +47 61 14 54 00. | +-------------------------------+------------------------------------+ From owner-freebsd-questions@freebsd.org Fri Apr 28 15:08:46 2017 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A36CAD54450 for ; Fri, 28 Apr 2017 15:08:46 +0000 (UTC) (envelope-from steve@sohara.org) Received: from smtp3.irishbroadband.ie (smtp4.irishbroadband.ie [62.231.32.6]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6FB331EAC for ; Fri, 28 Apr 2017 15:08:45 +0000 (UTC) (envelope-from steve@sohara.org) Received: from [89.127.62.20] (helo=smtp.lan.sohara.org) by smtp3.irishbroadband.ie with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.76) (envelope-from ) id 1d47Ej-0001yC-W5 for freebsd-questions@freebsd.org; Fri, 28 Apr 2017 15:51:10 +0100 Received: from [192.168.63.1] (helo=steve.lan.sohara.org) by smtp.lan.sohara.org with smtp (Exim 4.89 (FreeBSD)) (envelope-from ) id 1d47Fj-000KEj-60 for freebsd-questions@freebsd.org; Fri, 28 Apr 2017 14:52:11 +0000 Date: Fri, 28 Apr 2017 15:51:05 +0100 From: Steve O'Hara-Smith To: freebsd-questions@freebsd.org Subject: Re: A seperate ZFS question Message-Id: <20170428155105.563a2ee5192bb8a08f76ca9d@sohara.org> In-Reply-To: References: X-Mailer: Sylpheed 3.5.1 (GTK+ 2.24.29; amd64-portbld-freebsd10.3) X-Clacks-Overhead: "GNU Terry Pratchett" Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Apr 2017 15:08:46 -0000 On Fri, 28 Apr 2017 15:52:59 +0200 (CEST) Trond Endrestøl wrote: > On Fri, 28 Apr 2017 14:14+0100, Paul Macdonald wrote: > > > > > I fat fingered adding a cache drive and was unable to remove the SSD > > cache drive (ada2) > First, you have gpt/zfs0 and ada1p3 in a mirrored configuration. Next, > you have ada2 striped (RAID 0) to the first pair of mirrors. This is a > very weird "RAID 1 + 0" combo. Guess what will happen when you lose > ada2! > > Maybe you should describe what you wanted to happen in the first > place. He did, ada2 is supposed to be a cache drive, now it's a stripe. > I suggest you make good backups of this pool, destroy it, recreate it > properly, and restore the missing data. Agreed, you can't undo adding a stripe. -- Steve O'Hara-Smith