Date: Fri, 6 Apr 2018 09:08:06 -0600 From: Warner Losh <imp@bsdimp.com> To: Borja Marcos <borjam@sarenet.es> Cc: Steven Hartland <killing@multiplay.co.uk>, "Eugene M. Zheganin" <eugene@zhegan.in>, FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org> Subject: Re: TRIM, iSCSI and %busy waves Message-ID: <CANCZdfr63TLb6Trv_L3i4nXKTAH59qH=Xrwzv1epiZ2fGhk__Q@mail.gmail.com> In-Reply-To: <9AC3F9CE-B715-4894-B047-38758ACD913C@sarenet.es> References: <92b92a3d-3262-c006-ed5a-dc2f9f4a5cb9@zhegan.in> <CANCZdfrQBqcb5ASPFGjLN=nif4bQ9vTjMtBuDaOvvUKkM7u3XA@mail.gmail.com> <8935E1F4-10DE-4621-8153-26AF851CC26E@sarenet.es> <CAHEMsqaLSBURNSsx4k8YKBckAWtN2uva36vck5G8jWjhkLLgmQ@mail.gmail.com> <9AC3F9CE-B715-4894-B047-38758ACD913C@sarenet.es>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Apr 6, 2018 at 2:56 AM, Borja Marcos <borjam@sarenet.es> wrote: > > > On 6 Apr 2018, at 10:41, Steven Hartland <killing@multiplay.co.uk> wrote: > > That is very hw and use case dependent. > > The reason we originally sponsored the project to add TRIM to ZFS was tha= t > in our case without TRIM the performance got so bad that we had to secure > erase disks every couple of weeks as they simply became so slow they wher= e > unusable. > > Now admittedly that was a fair few years ago and hw has moved on since > then, but the point remains that it=E2=80=99s not totally true that just = not > TRIMing is an option. > > > As far as I know, letting the device know that you have released a block > should be a Good Thing. > So I prefer to TRIM. I have also seen cases where not doing TRIMs resulte= d > in a terrible performance > although modern SSDs should have better algorithms to deal with it. > > But I have also seen bad cases of TRIM requests piling up and clogging th= e > queues for long periods > of time. Admittedly it was a purely synthetic situation (running bonnie++= ) > but, again, performing > operations like destroying a huge snapshot or dataset could trigger a > similar condition. It was > especially nasty when I tried NVMEs. I mentioned this here: > > https://lists.freebsd.org/pipermail/freebsd-fs/2016-May/023134.html > > > That=E2=80=99s why I think there is a mid point between the radical appro= ach of > not doing TRIMs at all > and clogging the queues with too many of them. > nvd is especially troublesome. It doesn't do any trim shaping *AT*ALL* and we machine gun a huge number of TRIM requests when we delete large files. TRIM is normally not a fast path operation inside the drives, so sending lots of TRIMs down will often starve read/write resources for a variety of reasons (it often forces single threading limiting the parallelism in the drive, it often kicks off GC which causes lots of block erases which are slow and block reads/writes to other pages/blocks on the die (some or all mitigated by planes, but still), etc. > If the TRIM queue gets very large and the =E2=80=9Cmandatory=E2=80=9D ope= rations (read, > writes, etc) begin to > get large delays I think it=E2=80=99s safe to assume that *there is* a pr= oblem and > discarding at least part > of those TRIMs could help to solve it. > That's often a property of the devices, however. And how quickly we shove TRIMs down its throat. I've seen situations where we play out the TRIMs slowly enough that we can see huge latencies in TRIMs (seconds to minutes), but see little to no effect effect on read latency, even in the P99 number since that's often where effects in TRIMs show up and are easiest to measure. Warner > A =E2=80=9CTRIM when you can=E2=80=9D should still be better than a =E2= =80=9CDon=E2=80=99t TRIM at all=E2=80=9D. > > Cheers, > > > > > > Borja. > > P.S: Attaching the graphs that were lost. > > > > > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfr63TLb6Trv_L3i4nXKTAH59qH=Xrwzv1epiZ2fGhk__Q>