Date: Wed, 1 May 2019 10:28:45 -0400 From: Paul Mather <paul@gromit.dlib.vt.edu> To: Michelle Sullivan <michelle@sorbs.net> Cc: freebsd-stable <freebsd-stable@freebsd.org> Subject: Re: ZFS... Message-ID: <7DBA7907-BE8F-4944-9A71-86E5AC1B85CA@gromit.dlib.vt.edu> In-Reply-To: <CB86C16D-87D9-4D3F-9291-1E2586246E04@sorbs.net> References: <30506b3d-64fb-b327-94ae-d9da522f3a48@sorbs.net> <CAOtMX2gf3AZr1-QOX_6yYQoqE-H%2B8MjOWc=eK1tcwt5M3dCzdw@mail.gmail.com> <56833732-2945-4BD3-95A6-7AF55AB87674@sorbs.net> <3d0f6436-f3d7-6fee-ed81-a24d44223f2f@netfence.it> <17B373DA-4AFC-4D25-B776-0D0DED98B320@sorbs.net> <70fac2fe3f23f85dd442d93ffea368e1@ultra-secure.de> <70C87D93-D1F9-458E-9723-19F9777E6F12@sorbs.net> <CAGMYy3tYqvrKgk2c==WTwrH03uTN1xQifPRNxXccMsRE1spaRA@mail.gmail.com> <5ED8BADE-7B2C-4B73-93BC-70739911C5E3@sorbs.net> <d0118f7e-7cfc-8bf1-308c-823bce088039@denninger.net> <2e4941bf-999a-7f16-f4fe-1a520f2187c0@sorbs.net> <CAOtMX2gOwwZuGft2vPpR-LmTpMVRy6hM_dYy9cNiw%2Bg1kDYpXg@mail.gmail.com> <34539589-162B-4891-A68F-88F879B59650@sorbs.net> <CAOtMX2iB7xJszO8nT_KU%2BrFuSkTyiraMHddz1fVooe23bEZguA@mail.gmail.com> <576857a5-a5ab-eeb8-2391-992159d9c4f2@denninger.net> <A7928311-8F51-4C72-839C-C9C2BA62C66E@sorbs.net> <b0fa0f8e-dc45-9d66-cc48-c733cbb9645b@denninger.net> <FD9802E0-E2E4-464A-8ABD-83B0A21C08F2@sorbs.net> <bf63007@sorbs.net> <CB86C16D-87D9-4D3F-9291-1E2586246E04@sorbs.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Apr 30, 2019, at 11:17 PM, Michelle Sullivan <michelle@sorbs.net> wrote: >> I have had it happen >> several times over my IT career. If that happens to you the odds are >> that it's absolutely unrecoverable and whatever gets corrupted is >> *gone.* > > Every drive corruption I have suffered in my career I have been able to > recover, all or partial data except where the hardware itself was totally > hosed (Ie clean room options only available)... even with brtfs.. yuk.. > puck.. yuk.. oh what a mess that was... still get nightmares on that > one... but I still managed to get most of the data off... in fact I put > it onto this machine I currently have problems with.. so after the > nightmare of brtfs looks like zfs eventually nailed me. It sounds from reading this thread that FreeBSD's built-in tools for ZFS recovery were insufficient for the corruption your pool suffered. Have you looked at the digital forensics realm to see whether those tools might help you? This article claims to extend The Sleuth Kit to support pooled storage such as ZFS, and they even describe recovering the bulk of an image file from a pool that has a disk missing (Evaluation Section, "Scenario C: reconstructing an incomplete pool"): "Extending The Sleuth Kit and its underlying model for pooled storage file system forensic analysis" https://www.sciencedirect.com/science/article/pii/S1742287617301901 >> If said software has no tools to "walk" said >> data or if it's impractical to have it do so you're at severe risk of >> being hosed. > > Umm what? I’m talking about a userland (libzfs) tool (Ie doesn’t need > the pool imported) such as zfs send (which requires the pool to be > imported - hence me not calling it a userland tool) to allow a sending of > data that can be found to other places where it can be either blindly > recovered (corruption might be present) or can be used to locate > files/paths etc that are known to be good (checksums match etc).. walk > the structures, feed the data elsewhere where it can be > examined/recovered... don’t alter it.... it’s a last resort tool when you > don’t have working backups.. See above. >> BTW if you've never had a UFS volume unlink all the blocks within a file >> on an fsck and then recover them back into the free list after a crash >> you're a rare bird indeed. If you think a corrupt ZFS volume is fun try >> to get your data back from said file after that happens. > > Been there done that though with ext2 rather than UFS.. still got all my > data back... even though it was a nightmare.. Is that an implication that had all your data been on UFS (or ext2:) this time around you would have got it all back? (I've got that impression through this thread from things you've written.) That sort of makes it sound like UFS is bulletproof to me. There are levels of corruption. Maybe what you suffered would have taken down UFS, too? I guess there's no way to know unless there's some way you can recreate exactly the circumstances that took down your original system (but this time your data on UFS). ;-) Cheers, Paul.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7DBA7907-BE8F-4944-9A71-86E5AC1B85CA>