From owner-freebsd-scsi@FreeBSD.ORG Sun Sep 21 08:35:05 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 74D898CD for ; Sun, 21 Sep 2014 08:35:05 +0000 (UTC) Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0A0B4CEC for ; Sun, 21 Sep 2014 08:35:04 +0000 (UTC) Received: from mandree.no-ip.org ([78.48.101.189]) by mail.gmx.com (mrgmx101) with ESMTPSA (Nemesis) id 0Lm7MT-1Y4xI80ztx-00Zh1H for ; Sun, 21 Sep 2014 10:34:57 +0200 Received: from [IPv6:::1] (localhost6.localdomain6 [IPv6:::1]) by apollo.emma.line.org (Postfix) with ESMTP id BB32223D543 for ; Sun, 21 Sep 2014 10:34:54 +0200 (CEST) Message-ID: <541E8DAE.8010407@gmx.de> Date: Sun, 21 Sep 2014 10:34:54 +0200 From: Matthias Andree User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.1 MIME-Version: 1.0 To: freebsd-scsi@freebsd.org Subject: Re: How to disable hard disk write cache? References: <541804B0.7070407@gmail.com> <54184484.1070304@delphij.net> In-Reply-To: <54184484.1070304@delphij.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:5cQSvSGlU72/SrTcejv21eJdCvYJNZh0mQImVhb0PslH5BiHDUV wtZxsN/ngwVBabCm2XcaE0j14lMy3QgSD1HA7KFG5gYUAVAClKfQPYhDkmaTpYyfodJ2Bxv JVSg/B3QGeTVocco+MQvL3BfPov+9O3EZmH/CfSGtoefMZA8wixcf8ax1UnPh/pE9NigbGf gyVQIii/AbiECUrtfpbDA== X-UI-Out-Filterresults: notjunk:1; X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Sep 2014 08:35:05 -0000 Am 16.09.2014 um 16:09 schrieb Xin Li: > Modern SATA/SAS/SCSI devices usually comes with the capability of > tagged commands, allowing the OS to know when a write buffer is on > stable storage. With this, file systems can easily implement the > right semantics and recover from e.g. a power outage, etc. Yes, they *can easily implement* that, but which file systems in FreeBSD *actually do* that? Do we have a list which file systems are safe to use with WCE=1 as long as they do NCQ? And what do you need to do on those Samsung drives where NCQ is flakey (HD103SI for one)? From owner-freebsd-scsi@FreeBSD.ORG Sun Sep 21 08:42:28 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ACC02A58 for ; Sun, 21 Sep 2014 08:42:28 +0000 (UTC) Received: from anubis.delphij.net (anubis.delphij.net [64.62.153.212]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "anubis.delphij.net", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 92813DAB for ; Sun, 21 Sep 2014 08:42:28 +0000 (UTC) Received: from delphij-macbook.local (unknown [106.120.42.90]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by anubis.delphij.net (Postfix) with ESMTPSA id 6ABF319C05; Sun, 21 Sep 2014 01:42:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphij.net; s=anubis; t=1411288948; x=1411303348; bh=NcV5Nx5P2IOcDlpzA1WUIAdVpmO02vnVWtybK8HNrYE=; h=Date:From:Reply-To:To:Subject:References:In-Reply-To; b=cSEgFl72LBwSF+ffZVIzGevrwc4X5DMjEXU0fgDEfA/T86xSD3sOa8D2eXoXeEefX hZ+HCNqgk02HVatLp5YLeATEYZZHlVoZcWyTTzpTSaaun2xTYFJZcqlnJ60v7mdE1F 9V/yx5RdREyIrnDAcEDSNEVgsEgE4ys7loA4ckoA= Message-ID: <541E8F72.4020202@delphij.net> Date: Sun, 21 Sep 2014 16:42:26 +0800 From: Xin Li Reply-To: d@delphij.net Organization: The FreeBSD Project MIME-Version: 1.0 To: Matthias Andree , freebsd-scsi@freebsd.org Subject: Re: How to disable hard disk write cache? References: <541804B0.7070407@gmail.com> <54184484.1070304@delphij.net> <541E8DAE.8010407@gmx.de> In-Reply-To: <541E8DAE.8010407@gmx.de> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Sep 2014 08:42:28 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 9/21/14 4:34 PM, Matthias Andree wrote: > Am 16.09.2014 um 16:09 schrieb Xin Li: > >> Modern SATA/SAS/SCSI devices usually comes with the capability >> of tagged commands, allowing the OS to know when a write buffer >> is on stable storage. With this, file systems can easily >> implement the right semantics and recover from e.g. a power >> outage, etc. > > Yes, they *can easily implement* that, but which file systems in > FreeBSD *actually do* that? Both UFS (with Soft Updates, which is what it's all about; I haven't checked the Journalled Soft Updates case) and ZFS do that for ages. Cheers, -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJUHo9xAAoJEJW2GBstM+nsMo0P/2oyyJeH0LOvGGMIDJWw4CZf WGbEcecdQ3idVAw/2j/aYrlGpeYpGNZWi7dv5MOc3jdWRQbGkZ5r3E90YaSiwLac CRix+ovXeM+g7KSfb4q5NNrn9SEpQe8AklS5deTiFP3JxfV4juAf/aXUoCtrpdhm acVh0jYpel5ffdKDzXrHpKItvZ7t0YBg2qATK83+TGJ5cdefcEQnvKnOBMShyM4U ReOoeFO1y8ZBh9aVSYJlMarghxVXar212MavvLJxvKruQE++xf2lnhf8BkvCRiSd Xz24LZMJ/Cz92jMuYbCQ7sX4iIDkcCzpZeVOO8NjBE4YCc5XOCeIvw4uUdLwuWEz QqLLZYbepN1cjPraSrmNH8wFA6kZIQuZvMb3JJt6zRR8621VDg2IABAA4KyKs9CB 74shN5alKNCiDNf75EXfcZL30lTx8KDf/dPAil/mtYc0s7Orwc6veRDVmshCvSH+ y+f5NMYyOuVVoSrGNIhdP21c7vRyAywUVKgBT7IbnZg8I5ORNumCNcSFt+Q8c9Az sIC2cR2XVqCRQpH7qnfoot1e0FwUJEdyOHupVVMlTzFb55z06k79/w1/rruY4ylT rSunaKPy5xm4e43JxgzNhXobH6Qd0qO9VxB5nNb4By52GDccS5xIQw6v36TVXorn WfdQb0tZYKfnlE8tjCKH =MkPK -----END PGP SIGNATURE----- From owner-freebsd-scsi@FreeBSD.ORG Sun Sep 21 08:53:02 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 84CA2C13 for ; Sun, 21 Sep 2014 08:53:02 +0000 (UTC) Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1C391E6C for ; Sun, 21 Sep 2014 08:53:01 +0000 (UTC) Received: from mandree.no-ip.org ([78.48.101.189]) by mail.gmx.com (mrgmx101) with ESMTPSA (Nemesis) id 0LlYrb-1Y3Jg1041v-00bL4W for ; Sun, 21 Sep 2014 10:52:54 +0200 Received: from [IPv6:::1] (localhost6.localdomain6 [IPv6:::1]) by apollo.emma.line.org (Postfix) with ESMTP id 169B423D543 for ; Sun, 21 Sep 2014 10:52:53 +0200 (CEST) Message-ID: <541E91E4.4030802@gmx.de> Date: Sun, 21 Sep 2014 10:52:52 +0200 From: Matthias Andree User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.1 MIME-Version: 1.0 To: freebsd-scsi@freebsd.org Subject: Re: How to disable hard disk write cache? References: <541804B0.7070407@gmail.com> <54184484.1070304@delphij.net> <541E8DAE.8010407@gmx.de> <541E8F72.4020202@delphij.net> In-Reply-To: <541E8F72.4020202@delphij.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:2IbUdxKjC4/itDNkDxip1siSJTy5getZUwUMSU3BzyH1dAsUgY1 H789ZwhNDm6ITxdtMolWHc0cRySwLeWrNdWncj3UjqzzJmiWsmRnOyKQVPFtIOEwvs300Ai L+dv44106hu7zLDo/Goim8Bftkne63ec9mfbT8714s7s5/dFb/Z/eH3FSEyAbgDT57k5Yvs EWflf89ui2/rumy0pidqw== X-UI-Out-Filterresults: notjunk:1; X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Sep 2014 08:53:02 -0000 Am 21.09.2014 um 10:42 schrieb Xin Li: > On 9/21/14 4:34 PM, Matthias Andree wrote: >> Am 16.09.2014 um 16:09 schrieb Xin Li: > >>> Modern SATA/SAS/SCSI devices usually comes with the capability >>> of tagged commands, allowing the OS to know when a write buffer >>> is on stable storage. With this, file systems can easily >>> implement the right semantics and recover from e.g. a power >>> outage, etc. > >> Yes, they *can easily implement* that, but which file systems in >> FreeBSD *actually do* that? > > Both UFS (with Soft Updates, which is what it's all about; I haven't > checked the Journalled Soft Updates case) and ZFS do that for ages. Thank you. Is this in any of the 9.3 manpages so I could look that up rather than ask questions on the lists? :-) From owner-freebsd-scsi@FreeBSD.ORG Sun Sep 21 09:16:00 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CB9FB5AB for ; Sun, 21 Sep 2014 09:16:00 +0000 (UTC) Received: from anubis.delphij.net (anubis.delphij.net [IPv6:2001:470:1:117::25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "anubis.delphij.net", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id AB455B8 for ; Sun, 21 Sep 2014 09:16:00 +0000 (UTC) Received: from delphij-macbook.local (unknown [106.120.42.90]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by anubis.delphij.net (Postfix) with ESMTPSA id C343F19DF9; Sun, 21 Sep 2014 02:15:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphij.net; s=anubis; t=1411290960; x=1411305360; bh=vQ0Brxx4ZQjQsTDefy/6IKruWkyHMMqMiDlneuT8SWA=; h=Date:From:Reply-To:To:Subject:References:In-Reply-To; b=dIfBQcpUZPJ1+LJCvRShWDlgcLFKE7NlmclwsFnoskz0+q42OVEd3W1SgtXKO7UNu UIEmIkb0BxWXVxJRAgln2kFNFXkCcPGmgei4bTrdQ/MgAkkBZh81V/wISYoTNiGCul rjYunFcUAeqegjceN4UFN2wVu7m/yLhqrPSa1Ho8= Message-ID: <541E974E.7060702@delphij.net> Date: Sun, 21 Sep 2014 17:15:58 +0800 From: Xin Li Reply-To: d@delphij.net Organization: The FreeBSD Project MIME-Version: 1.0 To: Xu Zhe , d@delphij.net, freebsd-scsi@freebsd.org, Alexander Motin Subject: Re: How to disable hard disk write cache? References: <541804B0.7070407@gmail.com> <54184484.1070304@delphij.net> <5418FA4A.1050707@gmail.com> In-Reply-To: <5418FA4A.1050707@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Sep 2014 09:16:01 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 9/17/14 11:04 AM, Xu Zhe wrote: > My final goal is not to disable write cache. I just want to do a > simple test to see how would the IOPS drop when write cache > disabled (I suppose without disk write cache, the value should be > the same as IOPS of random read, both using O_DIRECT flag to avoid > system cache, etc.). > > Meanwhile, what I really want to know is that, how could I make > sure IO is persistent as long as I got reply from hard disk? Since > I suppose all file systems require this assumption to achieve self > consistency. You should get consistent I/O results assuming you are doing the right benchmark, which unfortunately not all benchmarks do. Doing benchmarks without the on-device caches is interesting but IMHO not very useful, because it's likely no practical application would do that. Traditionally, these write caches are disabled only when you really care about data durability; with modern technologies, now you can get affordable low latency and battery/supercapacitor backed device, for example most enterprise grade SSDs, and use that as your write-ahead journalling device to cover that gap. In order to get more consistent results, you may choose a running data set that is large enough to defeat the cache effect. The size can be chosen by doing a sequence of tests and gradually increase your running set size, and eventually you would find an "inflicting point" where performance would drop significantly. Please note that typical hard drive do not offer consistent latency when you operate on different areas of the drive, and the results could vary because a lot of factors. To do meaningful benchmarks, these factors also need to be considered. > I have googled about the "tagged command" but only found something > related to TCQ, which mainly talks about the queueing but not > anything related to cache sync. Any more hints? Hrm I don't have much thing off hand, but these are defined by the standards. Alexander knows much more than I do in this area. > I saw some pages that mentioned about SATA FUA command support on > Linux (Which I guess is what I am looking for): > > https://www.kernel.org/doc/Documentation/block/writeback_cache_control.txt > > However, I have not found useful information related to Freebsd. > :( Probably g_bio(9) but the documentation do not have much detail on driver implementation, and callers of the API expects the driver to do all the right things. This is if you want to implement a file system, where you would go. But back to your question, if you want to know e.g., "how to disable write cache on an AHCI connected drive", you would go to the individual driver's manual, e.g. ada(4). Cheers, -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJUHpdOAAoJEJW2GBstM+nsRPwP/1FzatW89mmXIdGPYk9IZ+Bo Qdlqvtxy0MCZOJQvug3jUKkiKGxr3+7boo1juQCKgMBqfJzPPhwAef3pRDteuPV4 VGTItEYAKWDo8JXnEl07ChAyxYMiCikiTkplaikOOa5gfmPdyPc9/a0myJ9WQDGk v6D0flS6s4+ZSRolLAMvSRtzpIAnDUIrU4FjJX+D4Mygz1hrHoYigzic3AahCOIm PhQe1nW6cPOjJASknFSFs450cDrQZ2QeVfH2Kr+K49ULdX+dn7ot3etJ4Fw+EBUN m66yPpgjssj/LqXS3m7XLJJYOOBy5FAeBrVUKzGQ6FxkfoCG8+kkD7YsQ2CJZPkc ILLIshVSL35OchuuMJpG9BGcr2BaaUQ23D4d72nIDm+ktsXiLkyD1XbZm7Ze+bR6 aW2KT3t8dU8NhJi8iyoWMS3bksh6lC5k07g8nqAsUNI+vAlKsnbT3cJJuowzbNLc 5Ai6L2+qFIst97BhTWxlpyBWkJM4K2lzB3AhZyc0Y4uXYni4Uj2DPSrC8dxOrp36 rH3SPACSnRP2cyChWQ+vV3l/uNsWM9snqVFAXPrSjmdN0fzN2KzfxnUiK0PtaFyR DoZcwM590GWYianbmzHFPiVCrsqjS6y2z4GK1SVUEfa9VM2qahWUSHFNap1DC3W6 ab4I5sE2eO3BknRXpoMg =tb5a -----END PGP SIGNATURE----- From owner-freebsd-scsi@FreeBSD.ORG Sun Sep 21 09:19:54 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4EA14702 for ; Sun, 21 Sep 2014 09:19:54 +0000 (UTC) Received: from anubis.delphij.net (anubis.delphij.net [64.62.153.212]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "anubis.delphij.net", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 3436FD3 for ; Sun, 21 Sep 2014 09:19:53 +0000 (UTC) Received: from delphij-macbook.local (unknown [106.120.42.90]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by anubis.delphij.net (Postfix) with ESMTPSA id 04F5219E6E; Sun, 21 Sep 2014 02:19:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphij.net; s=anubis; t=1411291193; x=1411305593; bh=KhdudE3yIwcqS8aHd02uR051f7EiBq17xxImiJFEICk=; h=Date:From:Reply-To:To:Subject:References:In-Reply-To; b=VWm1M3ErZy/MoFeZTUp4lRb6yN5Rr7vzr9LT5PruIIXUIuwrboRmSmALvmWHhFRSp LNUeIkAskzdHCodc8Vu/U7WBEEpN+obAPRYTKYi9YQaXpAklz7hGMrkxXnwhvSqDcV Io9Tx6ak1mkKGK1Z9hLFVuC3SBfMJDJVEELLAFuY= Message-ID: <541E9837.3060407@delphij.net> Date: Sun, 21 Sep 2014 17:19:51 +0800 From: Xin Li Reply-To: d@delphij.net Organization: The FreeBSD Project MIME-Version: 1.0 To: freebsd-scsi@freebsd.org Subject: Re: How to disable hard disk write cache? References: <541804B0.7070407@gmail.com> <54184484.1070304@delphij.net> <541E8DAE.8010407@gmx.de> <541E8F72.4020202@delphij.net> <541E91E4.4030802@gmx.de> In-Reply-To: <541E91E4.4030802@gmx.de> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Sep 2014 09:19:54 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 9/21/14 4:52 PM, Matthias Andree wrote: > Am 21.09.2014 um 10:42 schrieb Xin Li: >> On 9/21/14 4:34 PM, Matthias Andree wrote: >>> Am 16.09.2014 um 16:09 schrieb Xin Li: >> >>>> Modern SATA/SAS/SCSI devices usually comes with the >>>> capability of tagged commands, allowing the OS to know when a >>>> write buffer is on stable storage. With this, file systems >>>> can easily implement the right semantics and recover from >>>> e.g. a power outage, etc. >> >>> Yes, they *can easily implement* that, but which file systems >>> in FreeBSD *actually do* that? >> >> Both UFS (with Soft Updates, which is what it's all about; I >> haven't checked the Journalled Soft Updates case) and ZFS do that >> for ages. > > Thank you. Is this in any of the 9.3 manpages so I could look that > up rather than ask questions on the lists? :-) I don't really know... I personally see it as a vital feature for any file systems (or otherwise they become the cat who ate our homework :) so I doubt if it's actually being explicitly documented. This is probably a good FAQ candidate though. Cheers, -----BEGIN PGP SIGNATURE----- iQIbBAEBCgAGBQJUHpg3AAoJEJW2GBstM+nsQDoP+IFrhCT+1LEfDOQYcC29XRi3 SHwrkoMOXgnvw0/h0xTGxQ9pLDl9xLIwTmaWdFNtHM1hYUI1XzZC96GuNSEQfxuR G6VPIsj6mb/sE0+HphZmJHsM3QG2gC9aiFtMr6DggBa4bWokFpjWIigmKP4z5eFD 9a+q4P5tZFIJUoKeDHEyxDdaoxXRWcD7528ALF7MueJXj0Ej1ErPdr4yAi63YUar VrqGqPSRnyEf5h/wak5UGy4ETXZmIScAONYDjI7bwZaJbgO/3qn3AqiV+xVtPJol EVW8b3LjMuV0/d+aym02eVqb4CMPx8JE2Zb8cPl9oVBI386ej9NKzVQVrbL9SC+N Ogm4ScYQxYBI6k7fKg7L2Iy9K2zYHAIdBSRlFjVRipn8O8wt8yrTq0lZYxLpykIY g1+lVeguKZtntF1jg0r4Bnw0+qhJczXf/Xu7icnWIcOjzcAKd+cIXjHzb/0Jh1ZN RM7hEyR7aHW5nwwZPfpV+Q5KWCflc9XgUxxWVVgzXkWDhDMuCpTSGBi/ohjbgPs7 I0cj4IBH+S7fTLuRrkYyUaUg3Sqe3Z3A9Tkyr2q4ePq6RctigcDjVHfbFNpL10b0 PQfF2IqprcMdGoiqIZoQ82qpwSNizFKVzv7vLQnWvFd2Fr3Gh2XQbXWESDo33QfR VMxnrmQ8xrfBxX2Bn9g= =ZkPk -----END PGP SIGNATURE----- From owner-freebsd-scsi@FreeBSD.ORG Sun Sep 21 09:53:39 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5BB6DE6C for ; Sun, 21 Sep 2014 09:53:39 +0000 (UTC) Received: from mail-lb0-x22c.google.com (mail-lb0-x22c.google.com [IPv6:2a00:1450:4010:c04::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D46153F8 for ; Sun, 21 Sep 2014 09:53:38 +0000 (UTC) Received: by mail-lb0-f172.google.com with SMTP id p9so4641133lbv.31 for ; Sun, 21 Sep 2014 02:53:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:message-id:date:user-agent:mime-version:to:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=+qcjZR9ap4tuUMhAwUFNOnTWtaDTakoVZ9VyENAPzdk=; b=YK/ZMvzwlupK1rSZigSuewh6SF66iaXWrXSOy8/mahi7T3/wwTcYBr5TGcX8gFMKWK 3x9gSc9aqLvn21UwBtSFDCdWtYRfsBTjZUGMYBifFtybSfrEkwF15fNOKDsA5hxULdx0 bvU0MGRCreLIo6i80bmQDkW+fTY+u2j4kdLWXPq3IEu6Bm5SWPubakw7N5U+io82gZkg lGlmryGCIDYIO3KrklGapYhWaYiZwVWOc2sFCLXqE8yctBOjt6ccB4kJMjBfAaIAqAT4 wqjI7m3O0fm1u9jTmE8IV5k3trUyxp3qtgoVUju4LL9i1yynIL7TpSdpbyUG0lN2fNFo w+MA== X-Received: by 10.112.155.230 with SMTP id vz6mr1390138lbb.99.1411293216807; Sun, 21 Sep 2014 02:53:36 -0700 (PDT) Received: from mavbook.mavhome.dp.ua ([134.249.139.101]) by mx.google.com with ESMTPSA id v6sm2488370lbb.33.2014.09.21.02.53.34 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 21 Sep 2014 02:53:35 -0700 (PDT) Sender: Alexander Motin From: Alexander Motin X-Google-Original-From: Alexander Motin Message-ID: <541EA01E.6020600@ixsystems.com> Date: Sun, 21 Sep 2014 12:53:34 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: d@delphij.net, Xu Zhe , freebsd-scsi@freebsd.org Subject: Re: How to disable hard disk write cache? References: <541804B0.7070407@gmail.com> <54184484.1070304@delphij.net> <5418FA4A.1050707@gmail.com> <541E974E.7060702@delphij.net> In-Reply-To: <541E974E.7060702@delphij.net> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Sep 2014 09:53:39 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 21.09.2014 12:15, Xin Li wrote: > On 9/17/14 11:04 AM, Xu Zhe wrote: >> I have googled about the "tagged command" but only found >> something related to TCQ, which mainly talks about the queueing >> but not anything related to cache sync. Any more hints? > > Hrm I don't have much thing off hand, but these are defined by the > standards. Alexander knows much more than I do in this area. > >> I saw some pages that mentioned about SATA FUA command support >> on Linux (Which I guess is what I am looking for): > >> https://www.kernel.org/doc/Documentation/block/writeback_cache_control.txt > >> However, I have not found useful information related to >> Freebsd. :( > > Probably g_bio(9) but the documentation do not have much detail on > driver implementation, and callers of the API expects the driver to > do all the right things. This is if you want to implement a file > system, where you would go. > > But back to your question, if you want to know e.g., "how to > disable write cache on an AHCI connected drive", you would go to > the individual driver's manual, e.g. ada(4). I see there was several different topics touched in this threads, so let me start from the beginning. There are several methods to control disk caches, and respectively several approaches to maintaining the data consistency: - Caches can be controlled globally SCSI disks allow to control write cache via the Caching mode page. You may do it with `camcontrol mode` command. ATA disks has specific commands for that, and easiest way to do it is using sysctls/tunables documented in ada(4) manual page. Disabling caches globally heavily affects performance, and in most cases is overkill. It means that _every_ request will go to the media before the operation complete. For disks without command queuing (like legacy ATA) that usually meant _one_ I/O request per platter revolution. Do you want disk doing 120 IOPS peak? If you write huge file in 128K chunks, you will get limited by 120/s * 128K = 15MB/s! Command queuing (NCQ for SATA) significantly improved this situation since OS can now send more operations down to the disk to give it more flexibility, in significant part compensating disabled cache. But number of simultaneously active tags in disk, HBA or application can be limited, creating delays. - Caches can be controlled per-command SCSI disks may support FUA and DPO flags, that allow to do above cache control on per-command basis. SATA disks got FUA flag just recently. This approach has all properties of above, just can be controller per data type or application. Those flags are equivalent of IO_SYNC and IO_DIRECT flags on VFS layer of FreeBSD. Though FreeBSD block layer never had their equivalents. I've heard that Windows NTFS used this technique to keep metadata consistency up to some point, but moved away from it. - Caches can be flushed to media with SYNCHRONIZE CACHE commands Both SATA and SCSI disks provide ways to flush write caches to the media on request. It allows disks to do many writes in any order they prefer within one transaction group, but then reliably push everything to the media before when the group being committed. FreeBSD supports that on both VFS (fflush/VOP_FSYNC) and block layers (BIO_FLUSH). ZFS on FreeBSD relies on it to keep transaction groups atomicity and so metadata correctness. I've heard, this is what Windows NTFS is doing now too. I hope that answered most of questions. - -- Alexander Motin -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQF8BAEBCgBmBQJUHqAeXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRFOThDRjNDNEU2OUNDM0NEMEU1NzlENTU4 MzE4QzM5NTVCQUIyMjdGAAoJEIMYw5VbqyJ/E6UH+QF5AU3KNKjBLyNsgGTOn+UV SZAj3V3JLUNM4Z+8Z+4rxY/26/nbfNPoEJD4SaOt74oEUA574XjVLkHmqZ5JKygV dzOE2m8t0gyDIPurGx2CfRyG7UdJKbVsJ7Ebxafd8lRbwHGoObySUmew8t8WSIHA zBsI3ZiRYzBX7NnejPlJ8UPh498PBl78U+Ak08q/0scdPWsCneCjqHHn0Rx2MEFp 7sllphFh+EBXJXqetFRJfuWmm7yfIrzjO5UJ1mTjq5dCSeXrsIxBMBTSMEG34Yv6 9PqPWYPdVWQKgluKcaTKgrzPVTBgAqefx4gHRm/460a+7qohloCelMsgAsttYuA= =De3m -----END PGP SIGNATURE----- From owner-freebsd-scsi@FreeBSD.ORG Mon Sep 22 15:47:01 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5553AB9C for ; Mon, 22 Sep 2014 15:47:01 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AD308BD1 for ; Mon, 22 Sep 2014 15:47:00 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id 1AC1B9DC60D for ; Mon, 22 Sep 2014 17:40:00 +0200 (CEST) From: Borja Marcos Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Subject: mpr vs mps performance Date: Mon, 22 Sep 2014 17:39:58 +0200 Message-Id: To: FreeBSD-scsi Mime-Version: 1.0 (Apple Message framework v1283) X-Mailer: Apple Mail (2.1283) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Sep 2014 15:47:01 -0000 Hello, I have been playing with the new SAS3 cards supported by the mpr driver, = and I=B4ve found out that they are, in the same hardware configuration, considerably slower writing data. Moreover, running two = simultaneous "bonnie" benchmarks (I am using SSDs, and one "bonnie" = sometimes hits 100% CPU usage, unable to really saturate the I/O) I see = the writing activity somewhat stalling, with disk bandwidth going from = 600 MB/s to around 50 for 20 seconds or so. I'd like to know if this matches anyone else's experiences. Also, I can = try and make some tests if needed. But for now it seems we will stick to = the SAS2 HBAs. The Bonnie results are: With mpr driver, SAS3: (each bonnie instance, so multiply the results by 2 to get the actual = bandwidth achieved) Seq output: (writing) Block: 292155 KB/s Rewrite: 139713 KB/s Seq input: Block: 862861 KB/s With mps driver: SAS2, again, total is 2x the following figures. Seq output: (writing) Block: 587950 KB/s Rewrite: 208239 KB/s Seq. input: (reading) Block: 842169 KB/s The storage is a ZFS pool with a 9-disk raidz2 vdev, made of Samsung = 840 EVO 1 TB SSDs. The pool has been created with an ashift of 12 (zpool = applied it thanks to the 4 KB block quirk for these SSDs) at scbus0 target 9 lun 0 (pass0,da0) at scbus0 target 10 lun 0 (pass1,da1) at scbus0 target 11 lun 0 (pass2,da2) at scbus0 target 12 lun 0 (pass3,da3) at scbus0 target 13 lun 0 (pass4,da4) at scbus0 target 14 lun 0 (pass5,da5) at scbus0 target 17 lun 0 (pass7,da6) at scbus0 target 18 lun 0 (pass8,da7) at scbus0 target 27 lun 0 = (pass14,da12) The mpr card details follow: Sep 17 09:49:39 elibm kernel: mpr0: port 0x3f00-0x3fff mem = 0x912f0000-0x912fffff irq 32 at device 0.0 on pci17 Sep 17 09:49:39 elibm kernel: mpr0: IOCFacts : Sep 17 09:49:39 elibm kernel: MsgVersion: 0x205 Sep 17 09:49:39 elibm kernel: HeaderVersion: 0x1d00 Sep 17 09:49:39 elibm kernel: IOCNumber: 0 Sep 17 09:49:39 elibm kernel: IOCExceptions: 0x0 Sep 17 09:49:39 elibm kernel: MaxChainDepth: 128 Sep 17 09:49:39 elibm kernel: NumberOfPorts: 1 Sep 17 09:49:39 elibm kernel: RequestCredit: 11264 Sep 17 09:49:39 elibm kernel: ProductID: 0x2221 Sep 17 09:49:39 elibm kernel: IOCRequestFrameSize: 32 Sep 17 09:49:39 elibm kernel: MaxInitiators: 1 Sep 17 09:49:39 elibm kernel: MaxTargets: 1024 Sep 17 09:49:39 elibm kernel: MaxSasExpanders: 14 Sep 17 09:49:39 elibm kernel: MaxEnclosures: 15 Sep 17 09:49:39 elibm kernel: HighPriorityCredit: 60 Sep 17 09:49:39 elibm kernel: MaxReplyDescriptorPostQueueDepth: 65504 Sep 17 09:49:39 elibm kernel: ReplyFrameSize: 32 Sep 17 09:49:39 elibm kernel: MaxVolumes: 0 Sep 17 09:49:39 elibm kernel: MaxDevHandle: 1047 Sep 17 09:49:39 elibm kernel: MaxPersistentEntries: 128 Sep 17 09:49:39 elibm kernel: mpr0: Firmware: 01.00.03.00, Driver: = 05.255.05.00-fbsd Sep 17 09:49:39 elibm kernel: mpr0: IOCCapabilities: = 3a85c And the mps card is a classic: Sep 22 17:18:24 elibm kernel: mps0: port 0x3f00-0x3fff mem = 0x90ebc000-0x90ebffff,0x912c0000-0x912fffff irq 32 at device 0.0 on = pci17 Sep 22 17:18:24 elibm kernel: mps0: Firmware: 18.00.00.00, Driver: = 19.00.00.00-fbsd Sep 22 17:18:24 elibm kernel: mps0: IOCCapabilities: = 1285c= The connected devices follow. Both use the same hardware (except for the = cables and HBA of course), but currently there's no way to check this = with the SAS3 card, as sas3ircu nor sas3flash detect it on FreeBSD. # sas2ircu 0 display LSI Corporation SAS2 IR Configuration Utility. Version 18.00.00.00 (2013.11.18)=20 Copyright (c) 2009-2013 LSI Corporation. All rights reserved.=20 Read configuration has been initiated for controller 0 ------------------------------------------------------------------------ Controller information ------------------------------------------------------------------------ Controller type : SAS2008 BIOS version : 7.35.00.00 Firmware version : 18.00.00.00 Channel description : 1 Serial Attached SCSI Initiator ID : 0 Maximum physical devices : 255 Concurrent commands supported : 3432 Slot : 3 Segment : 0 Bus : 17 Device : 0 Function : 0 RAID Support : No ------------------------------------------------------------------------ IR Volume information ------------------------------------------------------------------------ ------------------------------------------------------------------------ Physical device information ------------------------------------------------------------------------ Initiator at ID #0 Device is a Hard disk Enclosure # : 2 Slot # : 16 SAS Address : 5000c50-0-05b5-ce25 State : Ready (RDY) Size (in MB)/(in sectors) : 140014/286749479 Manufacturer : SEAGATE=20 Model Number : ST9146803SS =20 Firmware Revision : FS03 Serial No : 3SD02W5L GUID : N/A Protocol : SAS Drive Type : SAS_HDD Device is a Hard disk Enclosure # : 2 Slot # : 17 SAS Address : 5005076-0-3e8e-81a2 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA =20 Model Number : Samsung SSD 840=20 Firmware Revision : BB0Q Serial No : S1D9NEADA08549F GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 2 Slot # : 18 SAS Address : 5005076-0-3e8e-81a3 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA =20 Model Number : Samsung SSD 840=20 Firmware Revision : BB0Q Serial No : S1D9NEADA08548T GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 2 Slot # : 19 SAS Address : 5005076-0-3e8e-81a4 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA =20 Model Number : Samsung SSD 840=20 Firmware Revision : BB0Q Serial No : S1D9NEADA08568E GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 2 Slot # : 20 SAS Address : 5005076-0-3e8e-81a5 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA =20 Model Number : Samsung SSD 840=20 Firmware Revision : BB0Q Serial No : S1D9NEADA08547X GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 2 Slot # : 21 SAS Address : 5005076-0-3e8e-81a6 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA =20 Model Number : Samsung SSD 840=20 Firmware Revision : BB0Q Serial No : S1D9NEADA08518Y GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 2 Slot # : 22 SAS Address : 5005076-0-3e8e-81a7 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA =20 Model Number : Samsung SSD 840=20 Firmware Revision : BB0Q Serial No : S1D9NEADA08556K GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Enclosure services device Enclosure # : 2 Slot # : 255 SAS Address : 5005076-0-3e8e-81b9 State : Standby (SBY) Manufacturer : IBM-ESXS Model Number : SAS EXP BP =20 Firmware Revision : 61A6 Serial No : 00000006 GUID : N/A Protocol : SAS Device Type : Enclosure services device Device is a Hard disk Enclosure # : 3 Slot # : 0 SAS Address : 5005076-0-3e8e-86e9 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA =20 Model Number : Samsung SSD 840=20 Firmware Revision : BB0Q Serial No : S1D9NEADA08550R GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 3 Slot # : 1 SAS Address : 5005076-0-3e8e-86ea State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA =20 Model Number : Samsung SSD 840=20 Firmware Revision : BB0Q Serial No : S1D9NEADA08911Y GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 3 Slot # : 2 SAS Address : 5005076-0-3e8e-86eb State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA =20 Model Number : Samsung SSD 840=20 Firmware Revision : BB0Q Serial No : S1D9NEADA08811L GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 3 Slot # : 13 SAS Address : 5000c50-0-05b5-e531 State : Ready (RDY) Size (in MB)/(in sectors) : 140014/286749479 Manufacturer : SEAGATE=20 Model Number : ST9146803SS =20 Firmware Revision : FS03 Serial No : 3SD02STR GUID : N/A Protocol : SAS Drive Type : SAS_HDD Device is a Hard disk Enclosure # : 3 Slot # : 14 SAS Address : 5000c50-0-05b5-d489 State : Ready (RDY) Size (in MB)/(in sectors) : 140014/286749479 Manufacturer : SEAGATE=20 Model Number : ST9146803SS =20 Firmware Revision : FS03 Serial No : 3SD02TV1 GUID : N/A Protocol : SAS Drive Type : SAS_HDD Device is a Hard disk Enclosure # : 3 Slot # : 15 SAS Address : 5000c50-0-05b5-f0ad State : Ready (RDY) Size (in MB)/(in sectors) : 140014/286749479 Manufacturer : SEAGATE=20 Model Number : ST9146803SS =20 Firmware Revision : FS03 Serial No : 3SD03F4C GUID : N/A Protocol : SAS Drive Type : SAS_HDD Device is a Enclosure services device Enclosure # : 3 Slot # : 255 SAS Address : 5005076-0-3e8e-86f9 State : Standby (SBY) Manufacturer : IBM-ESXS Model Number : SAS EXP BP =20 Firmware Revision : 61A6 Serial No : 00000006 GUID : N/A Protocol : SAS Device Type : Enclosure services device ------------------------------------------------------------------------ Enclosure information ------------------------------------------------------------------------ Enclosure# : 1 Logical ID : 500605b0:07ba2100 Numslots : 8 StartSlot : 0 Enclosure# : 2 Logical ID : 50050760:3e8e81a0 Numslots : 25 StartSlot : 0 Enclosure# : 3 Logical ID : 50050760:3e8e86e0 Numslots : 25 StartSlot : 0 ------------------------------------------------------------------------ SAS2IRCU: Command DISPLAY Completed Successfully. SAS2IRCU: Utility Completed Successfully. From owner-freebsd-scsi@FreeBSD.ORG Mon Sep 22 23:10:49 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0AA45BA3 for ; Mon, 22 Sep 2014 23:10:49 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E57D8776 for ; Mon, 22 Sep 2014 23:10:48 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id s8MNAmfI024233 for ; Mon, 22 Sep 2014 23:10:48 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-scsi@FreeBSD.org Subject: [Bug 193696] CAM AC_FOUND_DEVICE calls malloc(M_WAITOK) from THREAD_NO_SLEEPING() context Date: Mon, 22 Sep 2014 23:10:49 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: Needs Triage X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-scsi@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Sep 2014 23:10:49 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193696 --- Comment #5 from Garrett Cooper --- The patch Scott provided looked ok (doesn't panic on boot with simple cases with a VM), but I didn't get an opportunity to test it out more extensively. I didn't try out a patch that incorporates the feedback noted in comment # 4 yet. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-scsi@FreeBSD.ORG Tue Sep 23 08:11:37 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1BAF3ABB; Tue, 23 Sep 2014 08:11:37 +0000 (UTC) Received: from mail-pd0-x235.google.com (mail-pd0-x235.google.com [IPv6:2607:f8b0:400e:c02::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DD8ACE9B; Tue, 23 Sep 2014 08:11:36 +0000 (UTC) Received: by mail-pd0-f181.google.com with SMTP id r10so5872755pdi.26 for ; Tue, 23 Sep 2014 01:11:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=88zcngsJ6wBfkVu2BNiTx4qtAat/tz5iFJuUaPSdXDE=; b=soTAKZ/S1vLWBuB08iBxViJWJZE2GkRfYq0BEunKdnov3Idq2wnoxAM/IJMGZYM2Zg NBwENfgpDcanSiNBb0unCrQibRbxBeySoZmBEg3fRO/SJJln1ScdaZ40BwsNco6Bv8wX hl7ZejjFPYv52twy0/paV8LdMxiCdMUG/S/Cn+BeeIpGpQfA/Ca1NjJcUIAORP+d4DQ7 Q21Ecbzw5KeAETYMTeaVbCOh4mGM/u6sopwvVKZz3R235NVq6i4R/jg1vY2+Fuxl9msl c4sg79pDjN3fDNOWVUl+P8kKunGpBGbDn9rpYLQARWfbX4Def/qb56bzOIFtDTSGoHaD hhyg== X-Received: by 10.70.131.13 with SMTP id oi13mr35964706pdb.23.1411459896054; Tue, 23 Sep 2014 01:11:36 -0700 (PDT) Received: from Peters-MacAir.local ([106.39.255.55]) by mx.google.com with ESMTPSA id iq3sm11304345pbb.71.2014.09.23.01.11.32 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 23 Sep 2014 01:11:35 -0700 (PDT) Message-ID: <54212B31.7050608@gmail.com> Date: Tue, 23 Sep 2014 16:11:29 +0800 From: Xu Zhe User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Alexander Motin , d@delphij.net, freebsd-scsi@freebsd.org Subject: Re: How to disable hard disk write cache? References: <541804B0.7070407@gmail.com> <54184484.1070304@delphij.net> <5418FA4A.1050707@gmail.com> <541E974E.7060702@delphij.net> <541EA01E.6020600@ixsystems.com> In-Reply-To: <541EA01E.6020600@ixsystems.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Sep 2014 08:11:37 -0000 Hi, Alexandar, Xin, This is exactly what I am looking for. Thanks alot. :) Peter 于 14-9-21 17:53, Alexander Motin 写道: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > On 21.09.2014 12:15, Xin Li wrote: >> On 9/17/14 11:04 AM, Xu Zhe wrote: >>> I have googled about the "tagged command" but only found >>> something related to TCQ, which mainly talks about the queueing >>> but not anything related to cache sync. Any more hints? >> Hrm I don't have much thing off hand, but these are defined by the >> standards. Alexander knows much more than I do in this area. >> >>> I saw some pages that mentioned about SATA FUA command support >>> on Linux (Which I guess is what I am looking for): >>> https://www.kernel.org/doc/Documentation/block/writeback_cache_control.txt >>> However, I have not found useful information related to >>> Freebsd. :( >> Probably g_bio(9) but the documentation do not have much detail on >> driver implementation, and callers of the API expects the driver to >> do all the right things. This is if you want to implement a file >> system, where you would go. >> >> But back to your question, if you want to know e.g., "how to >> disable write cache on an AHCI connected drive", you would go to >> the individual driver's manual, e.g. ada(4). > I see there was several different topics touched in this threads, so > let me start from the beginning. There are several methods to control > disk caches, and respectively several approaches to maintaining the > data consistency: > > - Caches can be controlled globally > SCSI disks allow to control write cache via the Caching mode page. You > may do it with `camcontrol mode` command. ATA disks has specific > commands for that, and easiest way to do it is using sysctls/tunables > documented in ada(4) manual page. Disabling caches globally heavily > affects performance, and in most cases is overkill. It means that > _every_ request will go to the media before the operation complete. > For disks without command queuing (like legacy ATA) that usually meant > _one_ I/O request per platter revolution. Do you want disk doing 120 > IOPS peak? If you write huge file in 128K chunks, you will get limited > by 120/s * 128K = 15MB/s! Command queuing (NCQ for SATA) significantly > improved this situation since OS can now send more operations down to > the disk to give it more flexibility, in significant part compensating > disabled cache. But number of simultaneously active tags in disk, HBA > or application can be limited, creating delays. > > - Caches can be controlled per-command > SCSI disks may support FUA and DPO flags, that allow to do above cache > control on per-command basis. SATA disks got FUA flag just recently. > This approach has all properties of above, just can be controller per > data type or application. Those flags are equivalent of IO_SYNC and > IO_DIRECT flags on VFS layer of FreeBSD. Though FreeBSD block layer > never had their equivalents. I've heard that Windows NTFS used this > technique to keep metadata consistency up to some point, but moved > away from it. > > - Caches can be flushed to media with SYNCHRONIZE CACHE commands > Both SATA and SCSI disks provide ways to flush write caches to the > media on request. It allows disks to do many writes in any order they > prefer within one transaction group, but then reliably push everything > to the media before when the group being committed. FreeBSD supports > that on both VFS (fflush/VOP_FSYNC) and block layers (BIO_FLUSH). ZFS > on FreeBSD relies on it to keep transaction groups atomicity and so > metadata correctness. I've heard, this is what Windows NTFS is doing > now too. > > I hope that answered most of questions. > > - -- > Alexander Motin > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2 > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQF8BAEBCgBmBQJUHqAeXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w > ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRFOThDRjNDNEU2OUNDM0NEMEU1NzlENTU4 > MzE4QzM5NTVCQUIyMjdGAAoJEIMYw5VbqyJ/E6UH+QF5AU3KNKjBLyNsgGTOn+UV > SZAj3V3JLUNM4Z+8Z+4rxY/26/nbfNPoEJD4SaOt74oEUA574XjVLkHmqZ5JKygV > dzOE2m8t0gyDIPurGx2CfRyG7UdJKbVsJ7Ebxafd8lRbwHGoObySUmew8t8WSIHA > zBsI3ZiRYzBX7NnejPlJ8UPh498PBl78U+Ak08q/0scdPWsCneCjqHHn0Rx2MEFp > 7sllphFh+EBXJXqetFRJfuWmm7yfIrzjO5UJ1mTjq5dCSeXrsIxBMBTSMEG34Yv6 > 9PqPWYPdVWQKgluKcaTKgrzPVTBgAqefx4gHRm/460a+7qohloCelMsgAsttYuA= > =De3m > -----END PGP SIGNATURE----- From owner-freebsd-scsi@FreeBSD.ORG Wed Sep 24 14:41:26 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0C3A5400 for ; Wed, 24 Sep 2014 14:41:26 +0000 (UTC) Received: from SNT004-OMC1S23.hotmail.com (snt004-omc1s23.hotmail.com [65.55.90.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "*.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D0C78793 for ; Wed, 24 Sep 2014 14:41:25 +0000 (UTC) Received: from SNT151-W8 ([65.55.90.7]) by SNT004-OMC1S23.hotmail.com with Microsoft SMTPSVC(7.5.7601.22724); Wed, 24 Sep 2014 07:40:19 -0700 X-TMN: [H6vHDbLje8TcBimXFS9LDC8gkfDecL+I] X-Originating-Email: [kylie.liang2@outlook.com] Message-ID: From: Kylie Liang To: "freebsd-scsi@freebsd.org" Subject: scatter/gather list support Date: Wed, 24 Sep 2014 14:40:19 +0000 Importance: Normal MIME-Version: 1.0 X-OriginalArrivalTime: 24 Sep 2014 14:40:19.0662 (UTC) FILETIME=[76CD0EE0:01CFD805] Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 14:41:26 -0000 T24gaHR0cDovL3d3dy5mcmVlYnNkLm9yZy9kb2MvZW4vYm9va3MvYXJjaC1oYW5kYm9vay9zY3Np LWdlbmVyYWwuaHRtbCwgIGl0IG1lbnRpb25zICIgQWN0dWFsbHksIGl0CSAgc2VlbXMgbGlrZSB0 aGUgc2NhdHRlci1nYXRoZXIgYWJpbGl0eSBpcyBub3QgdXNlZCBhbnl3aGVyZQkgIGluIHRoZSBD QU0gY29kZSBub3cuICBCdXQgYXQgbGVhc3QgdGhlIGNhc2UgZm9yIGEgc2luZ2xlCSAgbm9uLXNj YXR0ZXJlZCB2aXJ0dWFsIGJ1ZmZlciBtdXN0IGJlIGltcGxlbWVudGVkLCBpdCBpcwkgIGFjdGl2 ZWx5IHVzZWQgYnkgQ0FNLiIuIA0KIA0KV2hhdCBkb2VzIGl0IG1lYW4/IEhvdyBkb2VzIENBTSBs ZXZlbCBoYW5kbGUgc2NhdHRlci1nYXRoZXIgSS9PIHJlcXVlc3Q/IFRoYW5rIHlvdS4gDQogCQkg CSAgIAkJICA= From owner-freebsd-scsi@FreeBSD.ORG Thu Sep 25 00:58:32 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2F99D2E6 for ; Thu, 25 Sep 2014 00:58:32 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 16001622 for ; Thu, 25 Sep 2014 00:58:32 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id s8P0wVVV094883 for ; Thu, 25 Sep 2014 00:58:31 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-scsi@FreeBSD.org Subject: [Bug 193696] CAM AC_FOUND_DEVICE calls malloc(M_WAITOK) from THREAD_NO_SLEEPING() context Date: Thu, 25 Sep 2014 00:58:32 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: bdrewery@FreeBSD.org X-Bugzilla-Status: Needs Triage X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-scsi@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Sep 2014 00:58:32 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193696 --- Comment #6 from Bryan Drewery --- Trace from xpt_done_td from pulling a device out of the system: KASSERT failed: malloc(M_WAITOK) in no_sleeping context KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe349829a340 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe349829a3f0 _kassert_panic() at _kassert_panic+0xd7/frame 0xfffffe349829a470 malloc() at malloc+0x2e4/frame 0xfffffe349829a4c0 g_post_event_x() at g_post_event_x+0x84/frame 0xfffffe349829a510 g_post_event() at g_post_event+0x5d/frame 0xfffffe349829a580 adacleanup() at adacleanup+0x62/frame 0xfffffe349829a5a0 cam_periph_release_locked_buses() at cam_periph_release_locked_buses+0xde/frame 0xfffffe349829aaa0 cam_periph_release_locked() at cam_periph_release_locked+0x1b/frame 0xfffffe349829aac0 adadone() at adadone+0x26e/frame 0xfffffe349829ab20 xpt_done_process() at xpt_done_process+0x3a4/frame 0xfffffe349829ab60 xpt_done_td() at xpt_done_td+0x136/frame 0xfffffe349829abb0 fork_exit() at fork_exit+0x84/frame 0xfffffe349829abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe349829abf0 -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-scsi@FreeBSD.ORG Thu Sep 25 01:09:22 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 83EE5420 for ; Thu, 25 Sep 2014 01:09:22 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6AAAC756 for ; Thu, 25 Sep 2014 01:09:22 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id s8P19MMP047012 for ; Thu, 25 Sep 2014 01:09:22 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-scsi@FreeBSD.org Subject: [Bug 193696] CAM AC_FOUND_DEVICE calls malloc(M_WAITOK) from THREAD_NO_SLEEPING() context Date: Thu, 25 Sep 2014 01:09:22 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: bdrewery@FreeBSD.org X-Bugzilla-Status: Needs Triage X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-scsi@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Sep 2014 01:09:22 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193696 --- Comment #7 from Bryan Drewery --- https://reviews.freebsd.org/D829 - KASSERT_WARN https://reviews.freebsd.org/D830 - Use KASSERT_WARN in malloc(9) and uma_zalloc_arg(9) -- You are receiving this mail because: You are the assignee for the bug.