From owner-freebsd-scsi@FreeBSD.ORG Sun Sep 21 09:53:39 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5BB6DE6C for ; Sun, 21 Sep 2014 09:53:39 +0000 (UTC) Received: from mail-lb0-x22c.google.com (mail-lb0-x22c.google.com [IPv6:2a00:1450:4010:c04::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D46153F8 for ; Sun, 21 Sep 2014 09:53:38 +0000 (UTC) Received: by mail-lb0-f172.google.com with SMTP id p9so4641133lbv.31 for ; Sun, 21 Sep 2014 02:53:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:message-id:date:user-agent:mime-version:to:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=+qcjZR9ap4tuUMhAwUFNOnTWtaDTakoVZ9VyENAPzdk=; b=YK/ZMvzwlupK1rSZigSuewh6SF66iaXWrXSOy8/mahi7T3/wwTcYBr5TGcX8gFMKWK 3x9gSc9aqLvn21UwBtSFDCdWtYRfsBTjZUGMYBifFtybSfrEkwF15fNOKDsA5hxULdx0 bvU0MGRCreLIo6i80bmQDkW+fTY+u2j4kdLWXPq3IEu6Bm5SWPubakw7N5U+io82gZkg lGlmryGCIDYIO3KrklGapYhWaYiZwVWOc2sFCLXqE8yctBOjt6ccB4kJMjBfAaIAqAT4 wqjI7m3O0fm1u9jTmE8IV5k3trUyxp3qtgoVUju4LL9i1yynIL7TpSdpbyUG0lN2fNFo w+MA== X-Received: by 10.112.155.230 with SMTP id vz6mr1390138lbb.99.1411293216807; Sun, 21 Sep 2014 02:53:36 -0700 (PDT) Received: from mavbook.mavhome.dp.ua ([134.249.139.101]) by mx.google.com with ESMTPSA id v6sm2488370lbb.33.2014.09.21.02.53.34 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 21 Sep 2014 02:53:35 -0700 (PDT) Sender: Alexander Motin From: Alexander Motin X-Google-Original-From: Alexander Motin Message-ID: <541EA01E.6020600@ixsystems.com> Date: Sun, 21 Sep 2014 12:53:34 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: d@delphij.net, Xu Zhe , freebsd-scsi@freebsd.org Subject: Re: How to disable hard disk write cache? References: <541804B0.7070407@gmail.com> <54184484.1070304@delphij.net> <5418FA4A.1050707@gmail.com> <541E974E.7060702@delphij.net> In-Reply-To: <541E974E.7060702@delphij.net> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Sep 2014 09:53:39 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 21.09.2014 12:15, Xin Li wrote: > On 9/17/14 11:04 AM, Xu Zhe wrote: >> I have googled about the "tagged command" but only found >> something related to TCQ, which mainly talks about the queueing >> but not anything related to cache sync. Any more hints? > > Hrm I don't have much thing off hand, but these are defined by the > standards. Alexander knows much more than I do in this area. > >> I saw some pages that mentioned about SATA FUA command support >> on Linux (Which I guess is what I am looking for): > >> https://www.kernel.org/doc/Documentation/block/writeback_cache_control.txt > >> However, I have not found useful information related to >> Freebsd. :( > > Probably g_bio(9) but the documentation do not have much detail on > driver implementation, and callers of the API expects the driver to > do all the right things. This is if you want to implement a file > system, where you would go. > > But back to your question, if you want to know e.g., "how to > disable write cache on an AHCI connected drive", you would go to > the individual driver's manual, e.g. ada(4). I see there was several different topics touched in this threads, so let me start from the beginning. There are several methods to control disk caches, and respectively several approaches to maintaining the data consistency: - Caches can be controlled globally SCSI disks allow to control write cache via the Caching mode page. You may do it with `camcontrol mode` command. ATA disks has specific commands for that, and easiest way to do it is using sysctls/tunables documented in ada(4) manual page. Disabling caches globally heavily affects performance, and in most cases is overkill. It means that _every_ request will go to the media before the operation complete. For disks without command queuing (like legacy ATA) that usually meant _one_ I/O request per platter revolution. Do you want disk doing 120 IOPS peak? If you write huge file in 128K chunks, you will get limited by 120/s * 128K = 15MB/s! Command queuing (NCQ for SATA) significantly improved this situation since OS can now send more operations down to the disk to give it more flexibility, in significant part compensating disabled cache. But number of simultaneously active tags in disk, HBA or application can be limited, creating delays. - Caches can be controlled per-command SCSI disks may support FUA and DPO flags, that allow to do above cache control on per-command basis. SATA disks got FUA flag just recently. This approach has all properties of above, just can be controller per data type or application. Those flags are equivalent of IO_SYNC and IO_DIRECT flags on VFS layer of FreeBSD. Though FreeBSD block layer never had their equivalents. I've heard that Windows NTFS used this technique to keep metadata consistency up to some point, but moved away from it. - Caches can be flushed to media with SYNCHRONIZE CACHE commands Both SATA and SCSI disks provide ways to flush write caches to the media on request. It allows disks to do many writes in any order they prefer within one transaction group, but then reliably push everything to the media before when the group being committed. FreeBSD supports that on both VFS (fflush/VOP_FSYNC) and block layers (BIO_FLUSH). ZFS on FreeBSD relies on it to keep transaction groups atomicity and so metadata correctness. I've heard, this is what Windows NTFS is doing now too. I hope that answered most of questions. - -- Alexander Motin -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQF8BAEBCgBmBQJUHqAeXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRFOThDRjNDNEU2OUNDM0NEMEU1NzlENTU4 MzE4QzM5NTVCQUIyMjdGAAoJEIMYw5VbqyJ/E6UH+QF5AU3KNKjBLyNsgGTOn+UV SZAj3V3JLUNM4Z+8Z+4rxY/26/nbfNPoEJD4SaOt74oEUA574XjVLkHmqZ5JKygV dzOE2m8t0gyDIPurGx2CfRyG7UdJKbVsJ7Ebxafd8lRbwHGoObySUmew8t8WSIHA zBsI3ZiRYzBX7NnejPlJ8UPh498PBl78U+Ak08q/0scdPWsCneCjqHHn0Rx2MEFp 7sllphFh+EBXJXqetFRJfuWmm7yfIrzjO5UJ1mTjq5dCSeXrsIxBMBTSMEG34Yv6 9PqPWYPdVWQKgluKcaTKgrzPVTBgAqefx4gHRm/460a+7qohloCelMsgAsttYuA= =De3m -----END PGP SIGNATURE-----