From owner-freebsd-fs@freebsd.org Tue May 17 07:59:44 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AA3D4B3EE1F for ; Tue, 17 May 2016 07:59:44 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 9A5841C97 for ; Tue, 17 May 2016 07:59:44 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: by mailman.ysv.freebsd.org (Postfix) id 99C27B3EE1D; Tue, 17 May 2016 07:59:44 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 994FDB3EE1B; Tue, 17 May 2016 07:59:44 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6038B1C96; Tue, 17 May 2016 07:59:43 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.36] (izaro.sarenet.es [192.148.167.11]) by proxypop01.sare.net (Postfix) with ESMTPSA id E7CBC9DD374; Tue, 17 May 2016 09:49:46 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ZFS and NVMe, trim caused stalling From: Borja Marcos In-Reply-To: <5E710EA5-C9B0-4521-85F1-3FE87555B0AF@bsdimp.com> Date: Tue, 17 May 2016 09:49:46 +0200 Cc: fs@freebsd.org, FreeBSD-STABLE Mailing List Content-Transfer-Encoding: quoted-printable Message-Id: References: <5E710EA5-C9B0-4521-85F1-3FE87555B0AF@bsdimp.com> To: Warner Losh X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 May 2016 07:59:44 -0000 > On 05 May 2016, at 16:39, Warner Losh wrote: >=20 >> What do you think? In some cases it=E2=80=99s clear that TRIM can do = more harm than good. >=20 > I think it=E2=80=99s best we not overreact. I agree. But with this issue the system is almost unusable for now. > This particular case is cause by the nvd driver, not the Intel P3500 = NVME drive. You need > a solution (3): Fix the driver. >=20 > Specifically, ZFS is pushing down a boatload of BIO_DELETE requests. = In ata/da land, these > requests are queued up, then collapsed together as much as makes sense = (or is possible). > This vastly helps performance (even with the extra sorting that I = forced to be in there that I > need to fix before 11). The nvd driver needs to do the same thing. I understand that, but I don=E2=80=99t think it=E2=80=99s a good that = ZFS depends blindly on a driver feature such as that. Of course, it=E2=80=99s great to exploit it. I have also noticed that ZFS has a good throttling mechanism for write = operations. A similar mechanism should throttle trim requests so that trim requests don=E2=80=99= t clog the whole system. > I=E2=80=99d be extremely hesitant to tossing away TRIMs. They are = actually quite important for > the FTL in the drive=E2=80=99s firmware to proper manage the NAND = wear. More free space always > reduces write amplification. It tends to go as 1 / freespace, so = simply dropping them on > the floor should be done with great reluctance. I understand. I was wondering about choosing the lesser between two = evils. A 15 minute I/O stall (I deleted 2 TB of data, that=E2=80=99s a lot, but not so = unrealistic) or settings trims aside during the peak activity. I see that I was wrong on that, as a throttling mechanism would be more = than enough probably, unless the system is close to running out of space. I=E2=80=99ve filed a bug report anyway. And copying to -stable. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D209571 Thanks! Borja.