Date: Wed, 31 Oct 2018 14:42:52 +0100 From: Stefan Bethke <stb@lassitu.de> To: FreeBSD Stable <freebsd-stable@freebsd.org> Subject: ZFS high write IO in single user mode Message-ID: <9D56C3BA-753B-472F-8FD7-AF90E1F14301@lassitu.de>
next in thread | raw e-mail | index | archive | help
I have two hosts that are configured identically (in a kind of manual = hot-standby configuration), running a set of jails each. ZFS datasets = for the jails and bhyve VMs are synced across regularly. When one of the = machines exhibits a problem, I can shut down the problematic jails or = the whole machine, and start the jails/VMs on the other host. This has = been working really well for the past ~10 years. A couple of years ago, one of the ZFS pools on one of the machines = developed some logical inconsistencies that were not detected by zpool = scrub. The only indication that something was amiss was high disk IO, in = particular, writes, even when no processes were running. I eventually = resolved that situation by recreating the zpool and restoring the = datasets from the working machine. About a year ago, I upgraded the hardware and in the process created = fresh pools. This has been running well. Since about two days ago, I now = have the situation again where I have a steady write rate even in single = user mode, with the root dataset mounted read only, and the second pool = that contains the jail datasets not mounted at all. I only have a video console (via IPMI KVM) so I won=E2=80=99t transcribe = the complete output, but here=E2=80=99s what I think are significant = observations: gstat reports ~30 writes/sec on each of the two disks that make up the = zmirror pool. mount shows the root dataset to be mounted read-only. zpool status takes a really long time, and then reports that everything = is fine for both pools (boot/os and jails). smartctl doesn=E2=80=99t show any problems for either of the disks. I=E2=80=99m happy to just wipe the pools and start fresh, but I=E2=80=99d = like to use this opportunity to hopefully figure out why ZFS appears to = act weirdly, and hopefully find a permanent fix. This is 11-stable from = September 13th. Stefan --=20 Stefan Bethke <stb@lassitu.de> Fon +49 151 14070811
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9D56C3BA-753B-472F-8FD7-AF90E1F14301>