Date: Wed, 15 Sep 2010 18:15:52 +0300 From: Andriy Gapon <avg@freebsd.org> To: Steven Hartland <killing@multiplay.co.uk> Cc: freebsd-fs@freebsd.org, jhell <jhell@DataIX.net>, Pawel Jakub Dawidek <pjd@freebsd.org> Subject: Re: zfs very poor performance compared to ufs due to lack of cache? Message-ID: <4C90E328.20606@freebsd.org> In-Reply-To: <0B1A90A08DFE4ADA9540F9F3846FDF38@multiplay.co.uk> References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk><AANLkTikNhsj5myhQCoPaNytUbpHtox1vg9AZm1N-OcMO@mail.gmail.com><4C85E91E.1010602@icyb.net.ua><4C873914.40404@freebsd.org><20100908084855.GF2465@deviant.kiev.zoral.com.ua><4C874F00.3050605@freebsd.org><A6D7E134B24F42E395C30A375A6B50AF@multiplay.co.uk><4C8D087B.5040404@freebsd.org><03537796FAB54E02959E2D64FC83004F@multiplay.co.uk><4C8D280F.3040803@freebsd.org><3FBF66BF11AA4CBBA6124CA435A4A31B@multiplay.co.uk><4C8E4212.30000@freebsd.org> <B98EBECBD399417CA5390C20627384B1@multiplay.co.uk> <D79F15FEB5794315BD8668E40B414BF0@multiplay.co.uk> <4C90B4C8.90203@freebsd.org> <6DFACB27CA8A4A22898BC81E55C4FD36@multiplay.co.uk> <4C90D3A1.7030008@freebsd.org> <0B1A90A08DFE4ADA9540F9F3846FDF38@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
on 15/09/2010 18:04 Steven Hartland said the following: > ----- Original Message ----- From: "Andriy Gapon" <avg@freebsd.org> > >>> Indeed. Where would this need to be addressed as ufs doesn't suffer from this? >> >> In ZFS. But I don't think that this is going to happen any time soon if at all. >> Authors of ZFS specifically chose to use a dedicated cache, which is ARC. >> Talk to them, or don't use ZFS, or get used to it. >> ARC has a price, but it supposedly has benefits too. >> Changing ZFS to use buffer cache is a lot of work and effectively means not using >> ARC, IMO. > > Hmm, so taking a different track on the issue is the a way to make sendfile use data > directly from ARC instead of having to copy it first? Well, theoretically everything is possible, but I am not sure if it's feasible. It's a lot of work anyways, it should be a very specialized sendfile and a lot if inter-layer knowledge and dependencies. Don't hold your breath for it. >> Well, I thought that you hurried when you applied the patches and changed the >> settings at the same time. This made it impossible for you to judge properly what >> patches do and don't do for you. > > No hurry just applying the patches that where suggested, retest, apply new retest > etc but > in parrallel been reading up on the arc tunables. > >>> Now we have a very simple setup so we can make sensible values for min / max but >>> it still means that for every file being sent when sendfile is enabled: >>> 1. There are two copies in memory which is still going to mean that only half the >>> amount files can be successfully cached and served without resorting to disk IO. >> >> Can't really say, depends on the size of the files. >> Though, it's approximately a half of what could have fit in memory with e.g. >> UFS, yes. > > Out of interest if a copy of the data is being made from ARC whats ties those > two copies together, in order to prevent the next request for the same file having to > create a third copy etc... Read about FreeBSD VM, particularly about a notion of VM object. http://www.informit.com/store/product.aspx?isbn=0201702452 >>> 2. sendfile isn't achieving what it states it should be i.e. a zero-copy. Does >>> this explain >>> the other odd behaviour we noticed, high CPU usage from nginx? >> >> sendfile should achieve zero copy with all the patches applied once both copies of >> data are settled in memory. If you have insufficient memory to hold the workset, >> then that's a different issue of moving competing data in and out of memory. And >> that may explain the CPU load, but it's just a speculation. > > Yes, more investigation needed ;-) > >> At present I don't see any other way but brute force - throw even more RAM at the >> problem. >> >> Perhaps, a miracle would happen and someone would post patches that radically >> change ZFS behavior with respect to caches. But I don't expect it >> (pessimist/realist). > > Or alternatively make sendfile work directly from ARC, would that be possible? I'll be glad to review the patches :-) > Thanks for all the info :) You are welcome. You did a lot of testing and investigative work here and I hope this thread will be useful for other people researching the topic. -- Andriy Gapon
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4C90E328.20606>