Date: Sat, 26 Nov 2011 10:41:51 +0200 From: Kostik Belousov <kostikbel@gmail.com> To: Lev Serebryakov <lev@freebsd.org> Cc: Kirk McKusick <mckusick@mckusick.com>, freebsd-fs@freebsd.org Subject: Re: Does UFS2 send BIO_FLUSH to GEOM when update metadata (with softupdates)? Message-ID: <20111126084151.GH50300@deviant.kiev.zoral.com.ua> In-Reply-To: <1961318852.20111126121354@serebryakov.spb.ru> References: <20111123194444.GE50300@deviant.kiev.zoral.com.ua> <201111260725.pAQ7PDow056289@chez.mckusick.com> <20111126080351.GD50300@deviant.kiev.zoral.com.ua> <1961318852.20111126121354@serebryakov.spb.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
--Uu8WkaNTGbYHJAd1 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Nov 26, 2011 at 12:13:54PM +0400, Lev Serebryakov wrote: > Hello, Kostik. > You wrote 26 =CE=CF=D1=C2=D2=D1 2011 =C7., 12:03:51: >=20 > >> You are entirely correct when you say that the requirement for > >> SU and SU+J is that it requires that notification of a disk-write > >> complete mean that the data is on the disk (stable). The problem > >> that arises is that (apparently) some tag-queue implementations > >> report back that tags have been written when in fact they have > >> not been written.=20 > > Right, and my belief that real hardware is not much affected, > You have wrong idea about modern hardware, sorry. >=20 > Again: don't forget multi-megabyte caches, and absence of any > guarantees, in which order these caches will be flashed. Many > controllers and drives itself group writes. And if companion for data > block in cache is found earlier than companion for metadata block (as > drive doesn't distinguish them) or waiting timeout, data block will > be written first. The same applicable to two metadata blocks, of > course. And it is not question of BROKEN QUEUEING. >=20 > Again, I'm speaking not about cheap ATA drivers here, but about > expensive high-performance RAID controllers ands server drives with > huge caches. >=20 > > except probably some ultra-cheap and old ATA disks. Another issue > > is broken-by-design 'drivers' which authors do not understand the > > environment they programming for. > And, again, or you have synchronous from top to bottoms storage > stack and performance, which will be miserable, compared to other > OSes, or you need to give some freedom to driver authors and provide > hints about semantic of personal operations to them. Every drive and > controller, which does write caching and reordering (except old, > cheap broken ATA ones) HAVE flags and knobs to send this individual > block to plactes as soon as possible. But now drivers doesn't have > any idea when they should use these flags. And they don't use them. Sigh. In-kernel i/o subsystem is already asynchronous by design. You have to make special arrangements to get synchronous writes of the blocks (like using bwrite()). And even if you do, buf layer emulates the sync operation by blocking the thread until async even happen. Whole buffer i/o (including reads) operates using bio_done callbacks called on the operation end. In fact, there is inherited uglyness due to async nature, namely, the kernel-owned buffer locks. Getting rid of them would be much more useful then breaking UFS. The non-broken driver must not return the 'completed' bio into the up queue until write is sent to hardware and hardware reported the completion. Raid controllers which aggressively cache the writes use nvram or battery backups, and do not allow to turn on write cache if battery is non-functional. I had not seen SU inconsistencies on RAID 6 on mfi(4), despite one our machine has unfortunate habit of dropping boot disk over SATA channel each week, for 2 years. >=20 > > I do not see how this proposal change much, except limiting potential > > havoc to the last 100ms of system operation. In fact, reordering, > > besides causing fs consistency problems, may cause the security issues > > as well [*]. If user data is written into the reused blocks, but > > metadata update was ordered after data write, we can end with the > > arbitrary override of the sensititive authorization or accounting > > information. > It is why metadata requests should be marked as non-reordable, > non-queuable. Personal requests, not some global barrier every 100ms. >=20 You again missed the point - if metadata is not reordable, but user data is, you get security issues. They are similar (but inverse) to what I described in the previous paragraph. --Uu8WkaNTGbYHJAd1 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk7Qpk8ACgkQC3+MBN1Mb4gpgQCfUVp3nqV+JAnWk+UEsU3DOPIK 5WwAnRf6jq28W3yqR1kOQwBNbOpZ3rPz =IEoy -----END PGP SIGNATURE----- --Uu8WkaNTGbYHJAd1--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20111126084151.GH50300>