Date: Sat, 9 Feb 2013 12:44:36 -0800 From: Jeremy Chadwick <jdc@koitsu.org> To: Fabian Keil <freebsd-listen@fabiankeil.de> Cc: freebsd-stable@freebsd.org, d@delphij.net, Dan Langille <dan@langille.org> Subject: Re: patch which implements ZFS LZ4 compression Message-ID: <20130209204436.GA67890@icarus.home.lan> In-Reply-To: <20130209151918.221176cd@fabiankeil.de> References: <C1BC7885-9C79-4534-8047-7408ED46A78A@langille.org> <511581C9.5040608@delphij.net> <20130208230830.GA45081@icarus.home.lan> <20130209151918.221176cd@fabiankeil.de>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Feb 09, 2013 at 03:19:18PM +0100, Fabian Keil wrote: > Jeremy Chadwick <jdc@koitsu.org> wrote: > > > On Fri, Feb 08, 2013 at 02:52:57PM -0800, Xin Li wrote: > > > On 02/08/13 14:29, Dan Langille wrote: > > > > Here is a patch against FreeBSD 9.1 STABLE which implements ZFS LZ4 > > > > compression. > > > > > > > > https://plus.google.com/106386350930626759085/posts/PLbkNfndPiM > > > > > > > > short link: http://bpaste.net/show/76095 > > > > > > Please DO NOT use this patch! It will ruin your data silently. > > > > > > As I already posted on Ivan's Google+ post, I'm doing final universe > > > builds to make sure that there is no regression and will merge my > > > changes to -HEAD later today. > > > > Another compression algorithm, this time 50%+ faster than lzjb. Great, > > fine, wonderful, awesome, kudos, huzzah, blah blah blah. > > > > So when is someone going to step up to the plate and fix how compression > > (as well as dedup) destroys interactivity on FreeBSD? Do I need to > > remind folks of this issue once again? Here you have it, dated October > > 2011, including the root cause and how it was fixed in Solaris et al: > > > > Description: > > > > http://lists.freebsd.org/pipermail/freebsd-fs/2011-October/012718.html > > > > Explanation and how Solaris et al fixed it, and how on Solaris the > > problem was major enough that it even caused NFS timeouts (sound > > familiar to anyone?): > > > > http://lists.freebsd.org/pipermail/freebsd-fs/2011-October/012726.html > > > > Further testing showing gzip-1 vs. lzjb and interactivity stalls: > > > > http://lists.freebsd.org/pipermail/freebsd-fs/2011-October/012752.html > > > > This is still a problem with base/stable/9. And as I have said > > elsewhere on lists, do not ask me to run CURRENT -- it will be a cold > > day in hell before I ever do that. I assume this same problem exists in > > CURRENT unless I have some key developer/committer say "I backported > > this fix in CURRENT, absolutely 100% sure". > > > > I'm also wondering why iXSystems hasn't stepped up to the plate to > > contribute to making this happen, given their business focus. I do not > > have the knowledge of the kernel (or of threading) to fix this myself, > > and for that I do apologise. > > > > But every time I see compression or dedup mentioned, I use the > > opportunity to bring up this subject. STOP ADDING FEATURES AND FIX > > STUFF LIKE THIS INSTEAD -- while new algorithms are neat/fun toys, they > > do not truly fix issues like this. How this problem has continually > > gotten overlooked is beyond me. > > Did you consider that other people may have different priorities > than you do? I see what you did there. > > If you want a PR for it, I'll file one, but all it's going to contain is > > the contents of this Email. > > My impression is that your emails describe symptoms and contain > some speculation about what the cause might be. I didn't see any > sched traces, so it's unclear (to me) that priorities are actual > the problem. They contain no speculation. Bob Friesenhahn, who has a lot of experience and familiarity with ZFS on Solaris, seemed to know exactly the behaviour I described. Others on FreeBSD have reported the same behaviour as well, just not in that thread circa 2011. Regarding "sched traces", please expand and include instructions. > It's also unclear to me why the dedup and compression issues should > be related. There are lots of dedup performance issues reported for > Solaris as well and I doubt that they can be fixed for FreeBSD without > significantly deviating from the ZFS upstream. What part of Bob's statement did you not understand? Here, let me repeat it verbatim: "Solaris solved the problem by putting the zfs writer threads into a special scheduling class so that they are usually lower priority than normal processing. Before this change, a desktop system would become almost unusable (intermittent loss of keyboard/mouse) while writing lots of data with compression enabled. Some NFS servers encountered severe enough issues that NFS clients reported NFS timeouts." > I'm not saying a PR would be useless, but in my experience PRs > with insufficient information just stay open and if the problem > isn't important enough for you to provide additional information > filing a PR is unlikely to have a great impact: > http://www.freebsd.org/cgi/query-pr-summary.cgi?category=&text=zfs Then someone in the know needs to explain exactly *what* data would help and (more importantly) *how* to go about providing it (i.e. what to enable in the kernel, what commands to issue, etc.). Eidan has repeatedly insisted that PRs are a Good Thing(tm) because they allow for an official way to track issues vs. mailing list threads that start and turn into tumbleweeds (just like the one I've referenced). Without those necessary instructions, in effect what you're asking me to do is "prove" that the problem exists, which I have already done so. You just don't like the data I've provided. Bottom line: people enable compression on an fs, issue large amounts of write I/O to that fs (say hundreds of megabytes, or gigabytes), and start to see the entire system intermittently stalling hard (for multiple seconds at a time). This affects everything from switching VTs on physical console to packets going across SSH. The stalls vary in duration depending on what compression type is used (lzjb vs. gzip-1 -- I cannot even imagine what gzip-9 would be like). I described it as verbosely as I could, including going back and "re-testing" because people felt the "ZFSv28 import might have addressed it" (it did not): http://lists.freebsd.org/pipermail/freebsd-fs/2011-October/012752.html The exact same behaviour happens if dedup is used. There is no relation between compression (the feature) and dedup (the feature), obviously, but the symptom I've described matches Bob's explanation perfectly. If you want to provide the aforementioned instructions, I'll happily follow them. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130209204436.GA67890>