Date: Sun, 10 Feb 2013 13:31:10 +0100 From: Fabian Keil <freebsd-listen@fabiankeil.de> To: Jeremy Chadwick <jdc@koitsu.org> Cc: freebsd-stable@freebsd.org Subject: Re: patch which implements ZFS LZ4 compression Message-ID: <20130210133110.1a5a3ca0@fabiankeil.de> In-Reply-To: <20130209204436.GA67890@icarus.home.lan> References: <C1BC7885-9C79-4534-8047-7408ED46A78A@langille.org> <511581C9.5040608@delphij.net> <20130208230830.GA45081@icarus.home.lan> <20130209151918.221176cd@fabiankeil.de> <20130209204436.GA67890@icarus.home.lan>
next in thread | previous in thread | raw e-mail | index | archive | help
--Sig_//+Dl4ssRW1LsCxzojoB9S8J Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Jeremy Chadwick <jdc@koitsu.org> wrote: > On Sat, Feb 09, 2013 at 03:19:18PM +0100, Fabian Keil wrote: > > Jeremy Chadwick <jdc@koitsu.org> wrote: =20 > > > If you want a PR for it, I'll file one, but all it's going to contain= is > > > the contents of this Email. > >=20 > > My impression is that your emails describe symptoms and contain > > some speculation about what the cause might be. I didn't see any > > sched traces, so it's unclear (to me) that priorities are actual > > the problem. >=20 > They contain no speculation. >=20 > Bob Friesenhahn, who has a lot of experience and familiarity with ZFS on > Solaris, seemed to know exactly the behaviour I described. Others on > FreeBSD have reported the same behaviour as well, just not in that > thread circa 2011. Similar symptoms can have different causes. =20 > Regarding "sched traces", please expand and include instructions. I'm referring to the stuff that is fed into: /usr/src/tools/sched/schedgraph.py It can be created with ktrace and dtrace and I believe the "documentation" is buried in the various "the scheduler sucks" threads. > > It's also unclear to me why the dedup and compression issues should > > be related. There are lots of dedup performance issues reported for > > Solaris as well and I doubt that they can be fixed for FreeBSD without > > significantly deviating from the ZFS upstream. >=20 > What part of Bob's statement did you not understand? Here, let me > repeat it verbatim: >=20 > "Solaris solved the problem by putting the zfs writer threads into a=20 > special scheduling class so that they are usually lower priority than=20 > normal processing. Before this change, a desktop system would become=20 > almost unusable (intermittent loss of keyboard/mouse) while writing=20 > lots of data with compression enabled. Some NFS servers encountered=20 > severe enough issues that NFS clients reported NFS timeouts." My impression from reading zfs-discuss@ is that dedup performance and some interactivity issues actually still exist in Illumos and that they are completely unrelated to zfs writer threads. As I can't use dedup on my systems I don't really pay attention to them, though. > > I'm not saying a PR would be useless, but in my experience PRs > > with insufficient information just stay open and if the problem > > isn't important enough for you to provide additional information > > filing a PR is unlikely to have a great impact: > > http://www.freebsd.org/cgi/query-pr-summary.cgi?category=3D&text=3Dzfs >=20 > Then someone in the know needs to explain exactly *what* data would help > and (more importantly) *how* to go about providing it (i.e. what to > enable in the kernel, what commands to issue, etc.). Eidan has > repeatedly insisted that PRs are a Good Thing(tm) because they allow for > an official way to track issues vs. mailing list threads that start and > turn into tumbleweeds (just like the one I've referenced). And how many of those PRs are actually solved? This is a rhetoric question and I don't expect you to look it up. I'm not saying that PRs are a bad thing, but filing PRs is the easy part and in my experience issues that don't spark developer interest on the mailing lists are usually also ignored when filed as PR, especially when they don't contain 100% of the information that may be relevant. Even if you provide "proof" that the priorities are indeed the cause of the problem there's a fair chance that the PR gets ignored anyway. I currently have four somewhat ZFS-related PRs open, the first was filed in 2007. I still don't think that the solution is that nobody works on ZFS improvements until my PRs are solved. I'm looking forward to using LZ4 which promises better compression than lzjb with less interactivity impact than gzip. It might even work for your /dev/random test as it's supposed to better deal with poorly compressible data. > Without those necessary instructions, in effect what you're asking me to > do is "prove" that the problem exists, which I have already done so. > You just don't like the data I've provided. I don't expect you to "prove" that the problem exists. My impression is that the interactivity issues with gzip have been well known for years and exist since the ZFS import. I also don't dislike your data, all I'm saying is that there could be other explanations. > Bottom line: people enable compression on an fs, issue large amounts of > write I/O to that fs (say hundreds of megabytes, or gigabytes), and > start to see the entire system intermittently stalling hard (for > multiple seconds at a time). This affects everything from switching VTs > on physical console to packets going across SSH. The stalls vary in > duration depending on what compression type is used (lzjb vs. gzip-1 -- > I cannot even imagine what gzip-9 would be like). I described it as > verbosely as I could, including going back and "re-testing" because > people felt the "ZFSv28 import might have addressed it" (it did not): >=20 > http://lists.freebsd.org/pipermail/freebsd-fs/2011-October/012752.html I'm aware that the interactivity issues with gzip still exist with ZFSv"5000" and don't deny that similar issues might exist with lzjb (even though it works for me). I don't think anyone is denying that gzip compression currently works poorly for some work loads. Fixing it apparently just isn't considered a priority and thus the issue still exists. Fabian --Sig_//+Dl4ssRW1LsCxzojoB9S8J Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlEXkxMACgkQBYqIVf93VJ1w1ACcCzjZQJqDHz7T9sdhajDBBz7Y ZPQAnizFbH2RpjnFvb+yJILwOiS8UzvN =icGT -----END PGP SIGNATURE----- --Sig_//+Dl4ssRW1LsCxzojoB9S8J--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130210133110.1a5a3ca0>