From owner-freebsd-fs@freebsd.org Tue Jun 20 19:52:50 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0FC8CDA0EC6 for ; Tue, 20 Jun 2017 19:52:50 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wr0-x235.google.com (mail-wr0-x235.google.com [IPv6:2a00:1450:400c:c0c::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 99062713E1 for ; Tue, 20 Jun 2017 19:52:49 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wr0-x235.google.com with SMTP id c11so49732887wrc.3 for ; Tue, 20 Jun 2017 12:52:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=q8c5PqP+eTHC/+rERyrMX3TUUkTb63FJD1lQTBimLdk=; b=GVgLmHTjHIhjzWiRKAn+AuXzmS9iyzJswbyblOrOZJ/ID7VLX5ml/9Dde16M0m/O12 briB49yBzZDKsCUYIp5G4j36qK7N/6RLlBjSBshCGRWBae/kCJ3Md3qDHCMC3td9qjAT KADeN3Or/tusUv/NlJYkk5Wm6XOdSD5acf7GgiWJgW0nobcbPb08mv2mzvV8cBX2Y5fs RQcMwpEXp+uQjVoegBWT4G3ssesBLSRSEBlOqvxu9lUHJDXw62qaDElGD1AgFwEiULsL OFXXkDaBxIBNZyn71MZnffi0tZqRZAAYiqUKG7A9E8dpHM4dLbf+sQ3IbGyQsiYi5l6V LI+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=q8c5PqP+eTHC/+rERyrMX3TUUkTb63FJD1lQTBimLdk=; b=Wbs4PFAhCyZcPxHmTY49ttsXdMlBKI8/jKVO48vpAwSmoeVh5qnNi5Z9KDEgqdkbCt ZMy9Sk84dRww9FySFIn6KMuCY8mfUAiBw0SDxwZdGsbPv2r7KklMuT8iMEfXgDKCRJ1l m2BAQClMyuOjjtQDhCGfHf+SuXOJL5jqBAc2Uzgm9AD/rVstZm8dUVzxIZoPHCugeHvt hRBQ1JcU/gkfiK8G53HpYOAxGHFSBn4s85MgRUmJe0hFKd66AYc1M/0HWmi7YgnTByvI XX/KP2pGEGHqPw2sBXL48u++og2TqcFk9xTerwutgJIQGav6pQItKiQlJauAHZA/Wp5R 1fgA== X-Gm-Message-State: AKS2vOwchPEOHBnvBH830qC6HAfjMTQnLUlbntLFEe6Dmp+SD1ooE1YK w8x+txAeky/nURHu9JvI8g== X-Received: by 10.223.144.69 with SMTP id h63mr20259206wrh.187.1497988367249; Tue, 20 Jun 2017 12:52:47 -0700 (PDT) Received: from [10.10.1.111] ([185.97.61.15]) by smtp.gmail.com with ESMTPSA id k86sm13475604wmi.16.2017.06.20.12.52.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 20 Jun 2017 12:52:46 -0700 (PDT) Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs To: "Caza, Aaron" , "freebsd-fs@freebsd.org" References: <431d958c658a408d8bfd4c574a565439@DM2PR58MB013.032d.mgd.msft.net> <990886ae-b7c1-8630-1cef-f6678f0b5b63@multiplay.co.uk> From: Steven Hartland Message-ID: Date: Tue, 20 Jun 2017 20:52:45 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 19:52:50 -0000 On 20/06/2017 17:58, Caza, Aaron wrote: > >> Can you show the output from gstat -pd during this DD please. > dT: 1.001s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > 0 4318 4318 34865 0.0 0 0 0.0 0 0 0.0 14.2| ada0 > 0 4402 4402 35213 0.0 0 0 0.0 0 0 0.0 14.4| ada1 > > dT: 1.002s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > 1 4249 4249 34136 0.0 0 0 0.0 0 0 0.0 14.1| ada0 > 0 4393 4393 35287 0.0 0 0 0.0 0 0 0.0 14.5| ada1 You %busy is very low, so sounds like the bottleneck isn't in raw disk performance but somewhere else. Might be interesting to see if anything stands out in top -Sz and then press h for threads. You also mention ada, could you share: sysctl kern.cam And: sysctl vfs.zfs Finally when its performing well can you repeat the gstat -pd > Every now and again, I was seeing d/s hit, which I understand to be TRIM operations - it would briefly show 16 then go back to 0. > > test@f111beta2:~ # dd if=/testdb/test of=/dev/null bs=1m > 16000+0 records in > 16000+0 records out > 16777216000 bytes transferred in 43.447343 secs (386150561 bytes/sec) > test@f111beta2:~ # uptime > 9:54AM up 19:38, 2 users, load averages: 2.92, 1.01, 0.44 > root@f111beta2:~ # dd if=/testdb/test of=/dev/null bs=1m > 16000+0 records in > 16000+0 records out > 16777216000 bytes transferred in 236.097011 secs (71060688 bytes/sec) > test@f111beta2:~ # uptime > 10:36AM up 20:20, 2 users, load averages: 0.90, 0.62, 0.36 > > As can be seen in the above 'dd' test results, I'm back to seeing the original issue I reported on freebsd-hackers - performance degrading after < 24 hours of uptime going from ~386MB/sec to ~71MB/sec inexpicably - this server isn't doing anything other than running this test hourly. > > Please note in the gstat -pd output above, this was after the performance degradation hit. Prior to this, I was seeing %busy of ~60%. In this particular instance, the performance degradation hit ~20hrs into the test but I've see it hit as soon as ~5hrs. > > Previously, Allan Jude had advised zfs.vfs.trim.enabled=0 to see if this changed the behavior. I did this; however, it had no impact - but that was when I was using the GEOM ELI 4k sector emulation and not the ashift 4k sector emulation. The GEOM ELI 4k sector emulation does not appear to work with TRIM operations as gstat -d in that case always stayed at 0 ops/s. I can try disabling trim, but did not want to reboot the server to restart the test in case there was some additional info worth capturing. > > I have captured an hourly log that can be provided containing the initial dmsg, zpool status, zfs list, zfs get all along with an hourly capture of the results of running the above 'dd' test with associated zfs-stats -a and sysctl -a output though it's currently 2.8MB hence too large to post to this list. > > Also, there seems to be a problem with my freebsd-fs subscription as I'm not getting e-mail notifications despite having submitted a subscription request so apologies for my slow responses. >