From owner-freebsd-stable@freebsd.org Fri Apr 6 15:08:09 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 309F9F8CA72 for ; Fri, 6 Apr 2018 15:08:09 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-it0-x236.google.com (mail-it0-x236.google.com [IPv6:2607:f8b0:4001:c0b::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AC1E680152 for ; Fri, 6 Apr 2018 15:08:08 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-it0-x236.google.com with SMTP id f6-v6so2380236ita.2 for ; Fri, 06 Apr 2018 08:08:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=NpDWHozfjRN1EYYFE5FUW7yTWGdX6F9rzVVLfwHegFU=; b=udpWOjVf+Jb7rP8jf2m4p5vESNII/FJqGomoVnFhrnCcG/m/U4KZK9Fcl1rqu9IB+4 gZqf9hl1fuQ3MtepWfLlcXDd6ZpIw8XkonEQyClRsWv9cSocJJcpGvBoAiRJ1stc0lEw XTFw1okuKWTk+Gfmnc+o0bP9wOsBdRJEYMCZ4ugMlmTLH4OrYxxQhcnVbiPSdZ9g4JuN TrgADTyWfX8j7TbUpHAippEy91z8zibalTvbeFpYGYROwQoCuXLwlUliHpsqba3h0uk9 9NlS85+YNwNYp/4et83RGqBXcUS7DCaT3oIisaq5MPPmMMq2HPAU3xlJWLtgXcn6bYis o3aA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=NpDWHozfjRN1EYYFE5FUW7yTWGdX6F9rzVVLfwHegFU=; b=e7jtsLP9L38p9wddrJMuHAE8XOxHHxO89h9II/j8lamehfpuk3szUCDGB37GCClVL8 /9YdSh58aMosXlWfJwKgp0I0+OKIaTGOSPL1iCo9kDdi3fkdDMlVNCm3nf5kXl38QB6E KqUeE6py6bsrlg2SyYPiSpXOp6xeS3HdHcwuhkZqD+GIf0fvZnAmm+/Psg1EgQCBRDs1 pLw7eOqAFnPHuQDJwNwhXvHAm00cHeY9NAAgX7FX7H8ivyIgwjgvR9MVyGjqUGCwlJaO N+lpNkWSW1nlpUQWea4y37eaW9qTbh9bqzL9Z/1HG8wo7qEZmroqtWhTSydBqvdLRZmU AqCw== X-Gm-Message-State: AElRT7GQVK6YXdHz1SH00J0dJd7yLzZq0ZYCSPLyWeKFKmRCX/0F2ZYV n7/73sz7C4wGLUOVMSAc03XVYE/A5EG8/J4g1qs1TQ== X-Google-Smtp-Source: AIpwx4/12HGj2+VwIO1jczL+gl8vRImjxwDhtU39uGUywzdPx1r/6hOh1QKQsVZ9MLtMc9Jm/BSpRR7egyyJvFlK5uo= X-Received: by 2002:a24:4286:: with SMTP id i128-v6mr18696994itb.73.1523027287554; Fri, 06 Apr 2018 08:08:07 -0700 (PDT) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.79.203.196 with HTTP; Fri, 6 Apr 2018 08:08:06 -0700 (PDT) X-Originating-IP: [2603:300b:6:5100:1052:acc7:f9de:2b6d] In-Reply-To: <9AC3F9CE-B715-4894-B047-38758ACD913C@sarenet.es> References: <92b92a3d-3262-c006-ed5a-dc2f9f4a5cb9@zhegan.in> <8935E1F4-10DE-4621-8153-26AF851CC26E@sarenet.es> <9AC3F9CE-B715-4894-B047-38758ACD913C@sarenet.es> From: Warner Losh Date: Fri, 6 Apr 2018 09:08:06 -0600 X-Google-Sender-Auth: YAqYLyUc8pY6PVTDsSaE2VbgfbI Message-ID: Subject: Re: TRIM, iSCSI and %busy waves To: Borja Marcos Cc: Steven Hartland , "Eugene M. Zheganin" , FreeBSD-STABLE Mailing List Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Apr 2018 15:08:09 -0000 On Fri, Apr 6, 2018 at 2:56 AM, Borja Marcos wrote: > > > On 6 Apr 2018, at 10:41, Steven Hartland wrote: > > That is very hw and use case dependent. > > The reason we originally sponsored the project to add TRIM to ZFS was tha= t > in our case without TRIM the performance got so bad that we had to secure > erase disks every couple of weeks as they simply became so slow they wher= e > unusable. > > Now admittedly that was a fair few years ago and hw has moved on since > then, but the point remains that it=E2=80=99s not totally true that just = not > TRIMing is an option. > > > As far as I know, letting the device know that you have released a block > should be a Good Thing. > So I prefer to TRIM. I have also seen cases where not doing TRIMs resulte= d > in a terrible performance > although modern SSDs should have better algorithms to deal with it. > > But I have also seen bad cases of TRIM requests piling up and clogging th= e > queues for long periods > of time. Admittedly it was a purely synthetic situation (running bonnie++= ) > but, again, performing > operations like destroying a huge snapshot or dataset could trigger a > similar condition. It was > especially nasty when I tried NVMEs. I mentioned this here: > > https://lists.freebsd.org/pipermail/freebsd-fs/2016-May/023134.html > > > That=E2=80=99s why I think there is a mid point between the radical appro= ach of > not doing TRIMs at all > and clogging the queues with too many of them. > nvd is especially troublesome. It doesn't do any trim shaping *AT*ALL* and we machine gun a huge number of TRIM requests when we delete large files. TRIM is normally not a fast path operation inside the drives, so sending lots of TRIMs down will often starve read/write resources for a variety of reasons (it often forces single threading limiting the parallelism in the drive, it often kicks off GC which causes lots of block erases which are slow and block reads/writes to other pages/blocks on the die (some or all mitigated by planes, but still), etc. > If the TRIM queue gets very large and the =E2=80=9Cmandatory=E2=80=9D ope= rations (read, > writes, etc) begin to > get large delays I think it=E2=80=99s safe to assume that *there is* a pr= oblem and > discarding at least part > of those TRIMs could help to solve it. > That's often a property of the devices, however. And how quickly we shove TRIMs down its throat. I've seen situations where we play out the TRIMs slowly enough that we can see huge latencies in TRIMs (seconds to minutes), but see little to no effect effect on read latency, even in the P99 number since that's often where effects in TRIMs show up and are easiest to measure. Warner > A =E2=80=9CTRIM when you can=E2=80=9D should still be better than a =E2= =80=9CDon=E2=80=99t TRIM at all=E2=80=9D. > > Cheers, > > > > > > Borja. > > P.S: Attaching the graphs that were lost. > > > > > >