From owner-freebsd-stable@FreeBSD.ORG Thu Mar 7 07:22:08 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 804B4B7F for ; Thu, 7 Mar 2013 07:22:08 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vps.rulingia.com (host-122-100-2-194.octopus.com.au [122.100.2.194]) by mx1.freebsd.org (Postfix) with ESMTP id 1680A391 for ; Thu, 7 Mar 2013 07:22:07 +0000 (UTC) Received: from server.rulingia.com (c220-239-237-213.belrs5.nsw.optusnet.com.au [220.239.237.213]) by vps.rulingia.com (8.14.5/8.14.5) with ESMTP id r277Loed053569 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Thu, 7 Mar 2013 18:21:50 +1100 (EST) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.14.5/8.14.5) with ESMTP id r277LjpT011114 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 7 Mar 2013 18:21:45 +1100 (EST) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.14.5/8.14.5/Submit) id r277LjAf011113; Thu, 7 Mar 2013 18:21:45 +1100 (EST) (envelope-from peter) Date: Thu, 7 Mar 2013 18:21:45 +1100 From: Peter Jeremy To: Karl Denninger Subject: Re: ZFS "stalls" -- and maybe we should be talking about defaults? Message-ID: <20130307072145.GA2923@server.rulingia.com> References: <513524B2.6020600@denninger.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="zYM0uCDKw75PZbzx" Content-Disposition: inline In-Reply-To: <513524B2.6020600@denninger.net> X-PGP-Key: http://www.rulingia.com/keys/peter.pgp User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Mar 2013 07:22:08 -0000 --zYM0uCDKw75PZbzx Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2013-Mar-04 16:48:18 -0600, Karl Denninger wrote: >The subject machine in question has 12GB of RAM and dual Xeon >5500-series processors. It also has an ARECA 1680ix in it with 2GB of >local cache and the BBU for it. The ZFS spindles are all exported as >JBOD drives. I set up four disks under GPT, have a single freebsd-zfs >partition added to them, are labeled and the providers are then >geli-encrypted and added to the pool. What sort of disks? SAS or SATA? >also known good. I began to get EXTENDED stalls with zero I/O going on, >some lasting for 30 seconds or so. The system was not frozen but >anything that touched I/O would lock until it cleared. Dedup is off, >incidentally. When the system has stalled: - Do you see very low free memory? - What happens to all the different CPU utilisation figures? Do they all go to zero? Do you get high system or interrupt CPU (including going to 1 core's worth)? - What happens to interrupt load? Do you see any disk controller interrupts? Would you be able to build a kernel with WITNESS (and WITNESS_SKIPSPIN) and see if you get any errors when stalls happen. On 2013-Mar-05 14:09:36 -0800, Jeremy Chadwick wrote: >On Tue, Mar 05, 2013 at 01:09:41PM +0200, Andriy Gapon wrote: >> Completely unrelated to the main thread: >>=20 >> on 05/03/2013 07:32 Jeremy Chadwick said the following: >> > That said, I still do not recommend ZFS for a root filesystem >> Why? >Too long a history of problems with it and weird edge cases (keep >reading); the last thing an administrator wants to deal with is a system >where the root filesystem won't mount/can't be used. It makes >recovery or problem-solving (i.e. the server is not physically accessible >given geographic distances) very difficult. I've had lots of problems with a gmirrored UFS root as well. The biggest issue is that gmirror has no audit functionality so you can't verify that both sides of a mirror really do have the same data. >My point/opinion: UFS for a root filesystem is guaranteed to work >without any fiddling about and, barring drive failures or controller >issues, is (again, my opinion) a lot more risk-free than ZFS-on-root. AFAIK, you can't boot from anything other than a single disk (ie no graid). --=20 Peter Jeremy --zYM0uCDKw75PZbzx Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlE4QAkACgkQ/opHv/APuId2wQCgs8WOllSrjKtPxNbBDDqtW9wG Tz8An26LiYxeg46x2+kr6cT9dgakLkKN =vgwF -----END PGP SIGNATURE----- --zYM0uCDKw75PZbzx--