From owner-freebsd-geom@FreeBSD.ORG Sun Feb 6 02:58:19 2005 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9F0D916A4CE for ; Sun, 6 Feb 2005 02:58:19 +0000 (GMT) Received: from FS.denninger.net (wsip-68-15-213-52.at.at.cox.net [68.15.213.52]) by mx1.FreeBSD.org (Postfix) with ESMTP id 975A743D41 for ; Sun, 6 Feb 2005 02:58:18 +0000 (GMT) (envelope-from karl@FS.denninger.net) Received: from fs.denninger.net (localhost [127.0.0.1]) by FS.denninger.net (8.13.1/8.13.1) with SMTP id j162wHCg014750 for ; Sat, 5 Feb 2005 20:58:17 -0600 (CST) (envelope-from karl@FS.denninger.net) Received: from fs.denninger.net [127.0.0.1] by Spamblock-sys; Sat Feb 5 20:58:17 2005 Received: (from karl@localhost) by FS.denninger.net (8.13.1/8.13.1/Submit) id j162wFSp014746; Sat, 5 Feb 2005 20:58:15 -0600 (CST) (envelope-from karl) Message-ID: <20050205205815.B14091@denninger.net> Date: Sat, 5 Feb 2005 20:58:15 -0600 From: Karl Denninger To: Paul Mather References: <20050205135705.A10437@denninger.net> <20050205201237.GB1666@darkness.comp.waw.pl> <20050205170415.A12620@denninger.net> <20050205230842.GD1666@darkness.comp.waw.pl> <20050205173326.B12620@denninger.net> <1107655941.4894.29.camel@zappa.Chelsea-Ct.Org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 0.93.2i In-Reply-To: <1107655941.4894.29.camel@zappa.Chelsea-Ct.Org>; from Paul Mather on Sat, Feb 05, 2005 at 09:12:21PM -0500 Organization: Karl's Sushi and Packet Smashers X-Die-Spammers: Spammers cheerfully broiled for supper and served with ketchup! cc: Pawel Jakub Dawidek cc: freebsd-geom@freebsd.org Subject: Re: Gmirror - how to do? X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2005 02:58:19 -0000 On Sat, Feb 05, 2005 at 09:12:21PM -0500, Paul Mather wrote: > On Sat, 2005-02-05 at 17:33 -0600, Karl Denninger wrote: > > On Sun, Feb 06, 2005 at 12:08:42AM +0100, Pawel Jakub Dawidek wrote: > > > On Sat, Feb 05, 2005 at 05:04:15PM -0600, Karl Denninger wrote: > > > +> > Remember not to boot the main machine with this disk inside, as it can > > > +> > be tasted before your main 'boot' mirror. Inserting this disk after > > > +> > boot, when your 'boot' mirror is configured should be safe. > > > +> > > > +> Nope, won't work. > > > +> > > > +> The mirrors potentially have different PHYSICAL slice sizes (remember > > > +> this debate a while back on this list?) and if I do this, I'm guaranteed to > > > +> screw the partition table, as the fdisk size of the slice table will be > > > +> picked up. > > > > > > Sorry, but I don't understand. > > > How can you touch slices configuration by labeling ad4s1? > > > > > > -- > > > Pawel Jakub Dawidek http://www.wheel.pl > > > pjd@FreeBSD.org http://www.FreeBSD.org > > > FreeBSD committer Am I Evil? Yes, I Am! > > > > > > Won't the gmirror system create the new mirror (on the "backup disk" using > > the full size of the slice? > > Gmirror will truncate the mirror's size to that of the smallest mirror > component at creation (or, if you try and insert a provider that is too > small for an existing mirror, it will [should] refuse to add it). So, > if you create a slice within the mirror component (/dev/mirror/...) then > it cannot, by definition, be "too big." The slice may not cover the > entire drive, but it won't be bigger than the drive. > > The only problem you can run into is if you try and use a drive that is > smaller than the smallest one used to create the initial mirror. That > will not work. > > Cheers, > > Paul. I think you're missing what I'm trying to accomplish, and where (I think) it won't work. Given the following configuration: 1. Two internal SATA drives, each of size "X". 2. An external SATA drive bay, with two disk carriers each of size "X" (or larger) 3. An initial RAID-1 boot volume that is comprised of the two internal disks. 4. FreeBSD 5-Stable The goal is that: 1. I can and it will NOT be used, by default, by the mirror, even if the system should reboot for some odd reason. This is important, because if I screw myself in some fashion on the running machine, having the backup disk scribbled on at the same time defeats the purpose of a backup. 2. I can cause the carrier disk to be recognized, and if it is, the system will bring it current. A physical command to initiate this is acceptable (indeed, required, since I do not want that process to start unexpectedly.) 3. Once the disk in the carrier IS current, I can detach it from the running configuration and return to State #1 above. I now have a "near-line" backup that can be mounted (SEPARATELY!) at any time to recover data if for some reason there is a problem (e.g. I 'fat-finger' an "rm -rf" from a directory that I really wanted) I can also rotate these disks in the carriers so that I have an offsite copy, plus a near-line copy. 4. If the original machine takes a dump entirely (e.g. the controller or OS goes insane and scribbles on both disks, there is a physical catastrophe with the system, etc) I can take the disk in either the carrier or the safe deposit box, plug it into an arbitrary machine and boot it without drama. I could also (if I had TWO disks) boot the most current one and then bring the other into the mirror, effectively recovering both the system and the redundancy all at once. An FSCK in that instance is acceptable since I don't want to quiesce the system (which I understand is necessary to unmount/etc) Critical applications can be stopped in (3) before the disk in the carrier is detached (e.g. dbms, etc) so I know THAT data is current. The problem(s) I see with the possible ways to do this are: 1. A "gmirror remove" removes the metadata on the disk and it will no longer boot, since the root filesystem isn't on a mirror anymore (the metadata is gone). 2. A "fake out" with a gmirror label leaves me vulnerable to an unexpected resync without being prompted if the system reboots for any reason, and the possibility of the system activating the root volume. The latter could be catastrophically bad, especially if gmirror then rebuilt the STALE disk back onto the other two, immediately destroying the current data set! It leaves me with the possibility that the mirror created on that disk (if its one of the "larger" ones) is too big, and if that one is booted in recovery mode that the other backup disk could not be made part of the mirror due to it being physically smaller in block count on the same slice. In other words, whether the system is truly recoverable transparently then depends on which disk is the one you boot singly. That's not good. 3. A "gmirror deactivate" followed by a "gmirror forget" is likely (not sure yet, will test) to leave me with an unbootable disk as in (1), because while the metadata is still there it is marked "do not use"; thus, the boot will fail (I think) 4. A "atacontrol detach" leaves me with the situation where a reboot finds the disk and can leave me in exactly the same situation as (2). Something I have not tried is to: 1. Mark the mirror as "no automatic rebuilds." 2. Bring the third disk current. 3. Stop the critical processes, and perform a couple of sync's. 4. Forcibly detach it using "atacontrol detach 2". This appears to also spin the disk down. 5. Clean up the idea that there is another disk out there with "gmirror forget". I should now have a "complete" mirror set with two components, and a detached disk. Since the mirror is set "no automatic rebuilds", if the system reboots it should NOT reattach the backup volume. 6. The backup volume SHOULD be bootable on its own, without drama, as the mirror, while it will be seen as degraded, should still be operational off that volume. The only problem is that there is no way to insure any buffers for that disk have been flushed, so its very similar to a "plug pull" for an unprotected machine. I think I'll try that one next, although I don't care for the semantics. Worst thing I can get is a panic for the unexpected detach, I think :-> I can always use the disk as if it was a great big tape drive, and just dump each filesystem to it, which avoids all of this tomfoolery (maybe with a minimal system on it so I can restore the dumps onto a new set of mirrors) but while that will work, it kinda evades the purpose of what I'm trying to do - come up with a way to "hot mirror" the machine in such a fashion that recovery is painless and possible even for someone who is completely untrained, being told only to stick the backup disk into a carrier on a new CPU and turn on the power. If this sort of thing is a model that the gmirror authors didn't think of, its something they might want to in the future..... -- -- Karl Denninger (karl@denninger.net) Internet Consultant & Kids Rights Activist http://www.denninger.net My home on the net - links to everything I do! http://scubaforum.org Your UNCENSORED place to talk about DIVING! http://www.spamcuda.net SPAM FREE mailboxes - FREE FOR A LIMITED TIME! http://genesis3.blogspot.com Musings Of A Sentient Mind