From owner-freebsd-questions@FreeBSD.ORG Fri Mar 1 20:39:18 2013 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id D82C2A06 for ; Fri, 1 Mar 2013 20:39:18 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.dawidek.net (garage.dawidek.net [91.121.88.72]) by mx1.freebsd.org (Postfix) with ESMTP id A448016E2 for ; Fri, 1 Mar 2013 20:39:18 +0000 (UTC) Received: from localhost (89-73-195-149.dynamic.chello.pl [89.73.195.149]) by mail.dawidek.net (Postfix) with ESMTPSA id DA8D6DD4; Fri, 1 Mar 2013 21:36:14 +0100 (CET) Date: Fri, 1 Mar 2013 21:40:32 +0100 From: Pawel Jakub Dawidek To: Mikolaj Golub Subject: Re: hast, zfs, unable to flush disk cache Message-ID: <20130301204032.GC16664@garage.freebsd.pl> References: <3367E416-493C-44EC-9259-89877EDF6DE8@balius.com> <20130301202549.GA2874@gmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="GZVR6ND4mMseVXL/" Content-Disposition: inline In-Reply-To: <20130301202549.GA2874@gmail.com> X-OS: FreeBSD 10.0-CURRENT amd64 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Chad M Stewart , freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Mar 2013 20:39:18 -0000 --GZVR6ND4mMseVXL/ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Mar 01, 2013 at 10:25:50PM +0200, Mikolaj Golub wrote: > On Fri, Mar 01, 2013 at 11:39:23AM -0600, Chad M Stewart wrote: >=20 > > I've setup a 2 node cluster using HAST. I'd previously had this > > running 9.0 and then rebuilt the nodes using 9.1. Under 9.0 I used > > vfs.zfs.vdev.bio_flush_disable=3D1 but that setting does not appear to > > work in 9.1. The other difference, previous build had root disk on > > UFS, while this build has only ZFS based file systems. > >=20 > > FreeBSD node1.san 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243826: Tue Dec = 4 06:55:39 UTC 2012 root@obrian.cse.buffalo.edu:/usr/obj/usr/src/sys/GE= NERIC i386 > >=20 > >=20 > > Mar 1 17:07:25 node1 hastd[1446]: [disk5] (primary) Unable to flush di= sk cache on activemap update: Operation not supported by device. > > Mar 1 17:07:28 node1 hastd[1440]: [disk3] (primary) Unable to flush di= sk cache on activemap update: Operation not supported by device. > > Mar 1 17:07:28 node1 hastd[1437]: [disk2] (primary) Unable to flush di= sk cache on activemap update: Operation not supported by device. > > Mar 1 17:07:28 node1 hastd[1434]: [disk1] (primary) Unable to flush di= sk cache on activemap update: Operation not supported by device. > > Mar 1 17:07:28 node1 hastd[1446]: [disk5] (primary) Unable to flush di= sk cache on activemap update: Operation not supported by device. > > Mar 1 17:07:28 node1 hastd[1443]: [disk4] (primary) Unable to flush di= sk cache on activemap update: Operation not supported by device. > >=20 > > I tried setting zfs set zfs:zfs_nocacheflush=3D1 but that > > diddn't appear to work either. I get a lot of lines in my log file > > because of this. I have also tried zfs set sync=3Ddisabled but > > HAST still outputs those lines in the log file. >=20 > These flushes were generated by HAST itself when it tried to flush > activemap updates, that is why disabling BIO_FLUSH for ZFS dis not > help much. >=20 > Setting "metaflush off" in hast.conf should help though. >=20 > BTW, hastd tries to detect devices that do not support BIO_FLUSH, > checking for the returned errno, and automatically disable flushes if > the errno is EOPNOTSUPP (Operation not supported). Unfortunately, your > device returned ENODEV (Operation not supported by device). >=20 > What device do you have? >=20 > Pawel, do you think it would be a good idea to automatically disable > activemap flush for ENODEV case too? It would be better to find the driver that returns ENODEV and fix it, IMHO. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com FreeBSD committer http://www.FreeBSD.org Am I Evil? Yes, I Am! http://tupytaj.pl --GZVR6ND4mMseVXL/ Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlExEkAACgkQForvXbEpPzR9jgCeIL26Y9KHjEPc/TtS8NJaulQA gEUAn2HQ1zumTSEOTmGOfh2CJZu6xGZm =4SMe -----END PGP SIGNATURE----- --GZVR6ND4mMseVXL/--