From owner-freebsd-stable@freebsd.org Wed May 1 04:16:30 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4C8BE1584313 for ; Wed, 1 May 2019 04:16:30 +0000 (UTC) (envelope-from walterp@gmail.com) Received: from mail-it1-x142.google.com (mail-it1-x142.google.com [IPv6:2607:f8b0:4864:20::142]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DBD9375ACC for ; Wed, 1 May 2019 04:16:28 +0000 (UTC) (envelope-from walterp@gmail.com) Received: by mail-it1-x142.google.com with SMTP id l140so7112686itb.0 for ; Tue, 30 Apr 2019 21:16:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=IWke8/OuX9QTrnbHJiD1nMJDXKuDWxEIyLIzXJvb6cU=; b=UnvXw1f1vG1wSI4J7NjWaJ0LYnq1zRYe9cxHAOPSmvo0PFalN4wHkCbMWemWb3A1xM 5pYthQB6NoJylbJgd7L9trB6oROH+LV6yYwoThs5SWuAfhHiO3bc3vRn9W9ZJtnRRWKI l+7lynqgIu5FHBc2M1kA2nFE5XbUta35iU2EGRUtIu6sDmIDKysohSqSdxGhTM2OmQ7n wDmcNxUqgaw9TCbRphzisPru5CbjgGeFpn25o3BFNDGHEduD1jlGd25nKfNSMiD/7rNe TmlckBWhs7ItJujvEtFY/YPslaf1Swl40DgFUcksSm3X/O2QPwPq+EJZV5DFQgfRcg2K HgyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=IWke8/OuX9QTrnbHJiD1nMJDXKuDWxEIyLIzXJvb6cU=; b=HrJHSN4BDjqqc8OJgoYtzl/8IjPqL+v/lU5qKEnFeu/Wi8d2RC2hFY46xrUCfyNJXR XLhLrALJymUglI4jjqhBmWN9YIR5Rx58enMxMvy1bk0Ara/U7U7a0XJGP8Oy3mtgfRyX Kxu/O6w4KP0dH0E55gqABTa69fA6cnEjoPsSzs9CNe1QQvZPOv9e9cCgJKtmMKMWFDna wmS0eYdxDBKZX35LZIM+MdQPTQOOnw5aj3/YSmHn1gpxXrm8p0HUmBlYaCnhbzsFjTlZ 1+QFfh6x3TiTIQi466MaEOlHA2sKNaLlZEDZsoYg0fCJyhWONQabhKHPHIiwB6wWgtuD T8NQ== X-Gm-Message-State: APjAAAXomMDC/pkOzuFKA9fBmZIt9wBsiHRhU9YKZvsAOln06X9cA8ES AtHrzAUbbR1oHUa2OVZhiA6MU7JfIhrNJP/ibr4rzg== X-Google-Smtp-Source: APXvYqz+d56yEByn1sIg6sH2u/ie8qgwBnWrkgd3mAaPeyYBKy4KgtBCCZc+/KXJby+6ath/346aVy17lzad2Snze5U= X-Received: by 2002:a24:7b41:: with SMTP id q62mr678826itc.22.1556684187706; Tue, 30 Apr 2019 21:16:27 -0700 (PDT) MIME-Version: 1.0 References: <30506b3d-64fb-b327-94ae-d9da522f3a48@sorbs.net> <56833732-2945-4BD3-95A6-7AF55AB87674@sorbs.net> <3d0f6436-f3d7-6fee-ed81-a24d44223f2f@netfence.it> <17B373DA-4AFC-4D25-B776-0D0DED98B320@sorbs.net> <70fac2fe3f23f85dd442d93ffea368e1@ultra-secure.de> <70C87D93-D1F9-458E-9723-19F9777E6F12@sorbs.net> <5ED8BADE-7B2C-4B73-93BC-70739911C5E3@sorbs.net> <2e4941bf-999a-7f16-f4fe-1a520f2187c0@sorbs.net> <34539589-162B-4891-A68F-88F879B59650@sorbs.net> <576857a5-a5ab-eeb8-2391-992159d9c4f2@denninger.net> In-Reply-To: From: Walter Parker Date: Tue, 30 Apr 2019 21:16:13 -0700 Message-ID: Subject: Re: ZFS... To: freebsd-stable@freebsd.org X-Rspamd-Queue-Id: DBD9375ACC X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=UnvXw1f1; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of walterp@gmail.com designates 2607:f8b0:4864:20::142 as permitted sender) smtp.mailfrom=walterp@gmail.com X-Spamd-Result: default: False [-4.31 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; SUBJ_ALL_CAPS(0.45)[6]; NEURAL_HAM_SHORT(-0.94)[-0.935,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-0.82)[ip: (1.39), ipnet: 2607:f8b0::/32(-3.18), asn: 15169(-2.24), country: US(-0.06)]; RCVD_IN_DNSWL_NONE(0.00)[2.4.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 May 2019 04:16:30 -0000 On Tue, Apr 30, 2019 at 8:19 PM Michelle Sullivan wrote: > > > Michelle Sullivan > http://www.mhix.org/ > Sent from my iPad > > > On 01 May 2019, at 12:37, Karl Denninger wrote: > > > > On 4/30/2019 20:59, Michelle Sullivan wrote > >>> On 01 May 2019, at 11:33, Karl Denninger wrote: > >>> > >>>> On 4/30/2019 19:14, Michelle Sullivan wrote: > >>>> > >>>> Michelle Sullivan > >>>> http://www.mhix.org/ > >>>> Sent from my iPad > >>>> > >>> Nope. I'd much rather *know* the data is corrupt and be forced to > >>> restore from backups than to have SILENT corruption occur and perhaps > >>> screw me 10 years down the road when the odds are my backups have > >>> long-since been recycled. > >> Ahh yes the be all and end all of ZFS.. stops the silent corruption of > data.. but don=E2=80=99t install it on anything unless it=E2=80=99s serve= r grade with > backups and ECC RAM, but it=E2=80=99s good on laptops because it protects= you from > silent corruption of your data when 10 years later the backups have > long-since been recycled... umm is that not a circular argument? > >> > ZFS works fine on systems with ECC. According to one of the original Sun authors of ZFS, the scrub of death is a myth. A non-ECC running ZFS is not more risky than one running UFS. As far as backups go, you should have backups for your important data. This is true for any filesystem. Home users have been using backups for decades. For anybody that has important data, not having a backup is a false economy. Odds are that most people you don't have backup can afford them if they plan and allocate money for this task (as 8TB USB drives are now at Costco for ~$140, getting one of.should not be much of an issue). Tarsnap is great and cheap online backup system. Silent data corruption is a thing. CERN tested it 10-15 years on brand new, high end production hardware. They run a test running on 3000 new servers run for 1 week. They found 147 silent data corruption errors on the server farm (found due to ZFS error checking). > >> Don=E2=80=99t get me wrong here.. and I know you (and some others are)= zfs in > the DC with 10s of thousands in redundant servers and/or backups to keep > your critical data corruption free =3D good thing. > >> > >> ZFS on everything is what some say (because it prevents silent > corruption) but then you have default policies to install it everywhere .= . > including hardware not equipped to function safely with it (in your own > arguments) and yet it=E2=80=99s still good because it will still prevent = silent > corruption even though it relies on hardware that you can trust... umm s= ay > what? > >> I run ZFS on embedded firewalls, it has worked fine for years. ZFS by default is a good idea on systems that ZFS built in (such as FreeBSD and SmartOS). There are great things you do with boot environments and volume management is much nicer. Don't let the blowhards that say ZFS should only be run on server grade (ie ECC) hardware spook you. It works as well as UFS would on any system with at least 1GB of RAM (but I'd suggest getting at least 2GB if you can't get 4GB), you just need to adjust a few memory parameters. > >> Anyhow veered way way off (the original) topic... > >> > >> Modest (part consumer grade, part commercial) suffered irreversible > data loss because of a (very unusual, but not impossible) double power > outage.. and no tools to recover the data (or part data) unless you have > some form of backup because the file system deems the corruption to be to= o > dangerous to let you access any of it (even the known good bits) ... > >> > >> Michelle > > > > IMHO you're dead wrong Michelle. I respect your opinion but disagree > > vehemently. > > I guess we=E2=80=99ll have to agree to disagree then, but I think your at= titude to > pronounce me =E2=80=9Cdead wrong=E2=80=9D is short sighted, because it st= rikes of =E2=80=9CI=E2=80=99m > right because ZFS is the answer to all problems.=E2=80=9D .. I=E2=80=99ve= been around in > the industry long enough to see a variety of issues... some disasters, so= me > not so... > > I also should know better than to run without backups but financial > constraints precluded me.... as will for many non commercial people. > > > > > I run ZFS on both of my laptops under FreeBSD. Both have > > non-power-protected SSDs in them. Neither is mirrored or Raidz-anythin= g. > > > > So why run ZFS instead of UFS? > > > > Because a scrub will detect data corruption that UFS cannot detect *at > all.* > > I get it, I really do, but that balances out against, if you can=E2=80=99= t rebuild > it make sure you have (tested and working) backups and be prepared for > downtime when such corruption does occur. > > > > > It is a balance-of-harms test and you choose. I can make a very clean > > argument that *greater information always wins*; that is, I prefer in > > every case to *know* I'm screwed rather than not. I can defend against > > being screwed with some amount of diligence but in order for that > > diligence to be reasonable I have to know about the screwing in a > > reasonable amount of time after it happens. > > Not disagreeing (and have not been.) > > > > > You may have never had silent corruption bite you. > > I have... but not with data on disks.. most of my silent corruption > issues have been with a layer or two above the hardware... like subversio= n > commits overwriting previous commits without notification (damn I wish I > could reliably replicate it!) > > > > I have had it happen > > several times over my IT career. If that happens to you the odds are > > that it's absolutely unrecoverable and whatever gets corrupted is > > *gone.* > > Every drive corruption I have suffered in my career I have been able to > recover, all or partial data except where the hardware itself was totally > hosed (Ie clean room options only available)... even with brtfs.. yuk.. > puck.. yuk.. oh what a mess that was... still get nightmares on that > one... but I still managed to get most of the data off... in fact I put = it > onto this machine I currently have problems with.. so after the nightmare > of brtfs looks like zfs eventually nailed me. > > > > The defensive measures against silent corruption require > > retention of backup data *literally forever* for the entire useful life > > of the information because from the point of corruption forward *the > > backups are typically going to be complete and correct copies of the > > corrupt data and thus equally worthless to what's on the disk itself.* > > With non-ZFS filesystems quite a lot of thought and care has to go into > > defending against that, and said defense usually requires the active > > cooperation of whatever software wrote said file in the first place > > Say what? > > > (e.g. a database, etc.) > > So dbs (any?) talk actively to the file systems (any?) to actively preven= t > silent corruption? > > Lol... > > I=E2=80=99m guessing you are actually talking about internal checks and b= alances > of data in the DB to ensure that dat retrieved from disk is not > corrupt/altered... you know like writing sha256 checksums of files you > might download from the internet to ensure you got what you asked for and > it wasn=E2=80=99t changed/altered in transit. > > > If said software has no tools to "walk" said > > data or if it's impractical to have it do so you're at severe risk of > > being hosed. > > Umm what? I=E2=80=99m talking about a userland (libzfs) tool (Ie doesn= =E2=80=99t need the > pool imported) such as zfs send (which requires the pool to be imported - > hence me not calling it a userland tool) to allow a sending of data that > can be found to other places where it can be either blindly recovered > (corruption might be present) or can be used to locate files/paths etc th= at > are known to be good (checksums match etc).. walk the structures, feed th= e > data elsewhere where it can be examined/recovered... don=E2=80=99t alter = it.... > it=E2=80=99s a last resort tool when you don=E2=80=99t have working backu= ps.. > > > Prior to ZFS there really wasn't any comprehensive defense > > against this sort of event. There are a whole host of applications tha= t > > manipulate data that are absolutely reliant on that sort of thing not > > happening (e.g. anything using a btree data structure) and recovery if > > it *does* happen is a five-alarm nightmare if it's possible at all. In > > the worst-case scenario you don't detect the corruption and the data > > that has the pointer to it that gets corrupted is overwritten and > > destroyed. > > > > A ZFS scrub on a volume that has no redundancy cannot *fix* that > > corruption but it can and will detect it. > > So you=E2=80=99re advocating restore from backup for every corruption ...= ok... > > > > This puts a boundary on the > > backups that I must keep in order to *not* have that happen. This is o= f > > very high value to me and is why, even on systems without ECC memory an= d > > without redundant disks, provided there is enough RAM to make it > > reasonable (e.g. not on embedded systems I do development on with are > > severely RAM-constrained) I run ZFS. > > > > BTW if you've never had a UFS volume unlink all the blocks within a fil= e > > on an fsck and then recover them back into the free list after a crash > > you're a rare bird indeed. If you think a corrupt ZFS volume is fun tr= y > > to get your data back from said file after that happens. > > Been there done that though with ext2 rather than UFS.. still got all my > data back... even though it was a nightmare.. > > > > > > -- > > Karl Denninger > > karl@denninger.net > > /The Market Ticker/ > > /[S/MIME encrypted email preferred]/ > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > --=20 The greatest dangers to liberty lurk in insidious encroachment by men of zeal, well-meaning but without understanding. -- Justice Louis D. Brandei= s