From owner-freebsd-stable@freebsd.org Thu Apr 5 14:08:29 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6E97FF88315 for ; Thu, 5 Apr 2018 14:08:29 +0000 (UTC) (envelope-from eugene@zhegan.in) Received: from elf.hq.norma.perm.ru (mail.norma.perm.ru [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D03A57DB4D for ; Thu, 5 Apr 2018 14:08:28 +0000 (UTC) (envelope-from eugene@zhegan.in) Received: from bsdrookie.norma.com. (asterisk.enaza.ru [91.237.76.254]) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTPS id w35E8MNe034336 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Thu, 5 Apr 2018 19:08:23 +0500 (YEKT) (envelope-from eugene@zhegan.in) To: freebsd-stable@FreeBSD.ORG From: "Eugene M. Zheganin" Subject: TRIM, iSCSI and %busy waves Message-ID: <92b92a3d-3262-c006-ed5a-dc2f9f4a5cb9@zhegan.in> Date: Thu, 5 Apr 2018 19:08:22 +0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Spamd-Result: default: False [-998.10 / 25.00] WHITELISTED_IPS(-999.00)[91.237.76.254] AUTH_NA(1.00)[] RCVD_COUNT_TWO(0.00)[2] MID_RHS_MATCH_FROM(0.00)[] DMARC_NA(0.00)[zhegan.in] FROM_EQ_ENVFROM(0.00)[] ARC_NA(0.00)[] ASN(0.00)[asn:57973, ipnet:91.237.76.0/24, country:RU] FROM_HAS_DN(0.00)[] R_SPF_NA(0.00)[] TO_MATCH_ENVRCPT_ALL(0.00)[] TO_DN_NONE(0.00)[] MIME_GOOD(-0.10)[text/plain] RCPT_COUNT_ONE(0.00)[1] RCVD_TLS_ALL(0.00)[] R_DKIM_NA(0.00)[] IP_SCORE(0.00)[ip: (-9.89), ipnet: 91.237.76.0/24(-7.82), asn: 57973(-4.89), country: RU(0.12)] X-Rspamd-Server: localhost X-Rspamd-Scan-Time: 0.85 X-Rspamd-Queue-ID: w35E8MNe034336 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Apr 2018 14:08:29 -0000 Hi, I have a production iSCSI system (on zfs of course) with 15 ssd disks and it's often suffering from TRIMs. Well, I know what TRIM is for, and I know it's a good thing, but sometimes (actually often) I'm seeing my disks in gstat are overwhelmed by the TRIM waves, this looks like a "wave" of 20K 100%busy delete operations starting on first pool disk, then reaching second, then third,... - at the time it reaches the 15th disk the first one if freed from TRIM operations, and in 20-40 seconds this wave begins again. I'm also having a couple of iSCSI issues that I'm dealing through bounty with, so may be this is related somehow. Or may be not. Due to some issues in iSCSI stack my system sometimes reboots, and then these "waves" are stopped for some time. So, my question is - can I fine-tune TRIM operations ? So they don't consume the whole disk at 100%. I see several sysctl oids, but they aren't well-documented. P.S. This is 11.x, disks are Toshibas, and they are attached via LSI HBA. Thanks. Eugene.