From owner-freebsd-stable@FreeBSD.ORG Thu Jan 15 17:22:52 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B647AF60 for ; Thu, 15 Jan 2015 17:22:52 +0000 (UTC) Received: from mail-we0-x233.google.com (mail-we0-x233.google.com [IPv6:2a00:1450:400c:c03::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5B0D2239 for ; Thu, 15 Jan 2015 17:22:52 +0000 (UTC) Received: by mail-we0-f179.google.com with SMTP id q59so15962004wes.10 for ; Thu, 15 Jan 2015 09:22:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=pzHKFLMphev5AI1tANp4xbmaIbTWPJxsfP/95SRgqOU=; b=RLN5npwaDRuEGF4XFPpUbvVppu24bV38nKgOkcNYJHVZI99LA1xh+KqANDDp2NaUdr 1p+7FnJX2OGjAKrEnkNYbsLJNLjbOm9KrUC1fYkMYV1DSUKvNQsCAH/XhKGbZIu0Qxge 94jaoU3BnntQ4s47g2hsq+dB0ZyG63PdYd/qHGCirUThwrgCnvjYTneR3Z3EaKDVqUTK Blat19zrdwn+pJfVsNpwF0MkjcjvR8lvaGS4ixF9sxSQTL9IbDy8iFPQa6noVC8xCWf3 pr8qM9MNY3/7KRFTn7s10b+G3t83x6u69jMfGAe37vNEpziDb9cdCbtNcKwHsrD/cc1G m37A== X-Received: by 10.194.203.104 with SMTP id kp8mr15888541wjc.103.1421342570667; Thu, 15 Jan 2015 09:22:50 -0800 (PST) Received: from [192.168.3.105] ([193.148.0.35]) by mx.google.com with ESMTPSA id u18sm2843427wjq.42.2015.01.15.09.22.49 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 15 Jan 2015 09:22:49 -0800 (PST) Message-ID: <54B7F769.40605@gmail.com> Date: Thu, 15 Jan 2015 19:22:49 +0200 From: Mihai Vintila User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Poor performance on Intel P3600 NVME driver Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jan 2015 17:22:52 -0000 Hi all, I've got a: 10.1-RELEASE FreeBSD 10.1-RELEASE #0 r274401: Tue Nov 11 21:02:49 UTC 2014 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 hw.machine: amd64 hw.model: Intel(R) Xeon(R) CPU E5-2603 v3 @ 1.60GHz hw.ncpu: 12 hw.machine_arch: amd64 With 2 Intel P3600 nvmecontrol devlist nvme0: INTEL SSDPE2ME020T4 nvme0ns1 (1907729MB) nvme1: INTEL SSDPE2ME020T4 nvme1ns1 (1907729MB) That i've put in a mirror or single drive zfs pool. Issue that i'm having is that performance is very poor on the P3600 drives, they are really close to a S3500 which is really poor: Here is a iozone benchmark on a single drive pool with recordsize set to 4k and compression to lz4 Command line used: iozone -Rb /root/output.wks -O -i 0 -i 1 -i 2 -e -+n -r4K -r 8K -r 32K -r 64K -r 128K -s 1G Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 1048576 4 74609 0 104268 0 95699 49975 1048576 8 36554 0 62927 0 59419 25778 1048576 32 9869 0 19148 0 18757 7134 1048576 64 5014 0 9612 0 9528 3813 1048576 128 2586 0 4908 0 4883 1962 While on S3500 i have: Command line used: iozone -Rb /root/output_nexenta.wks -O -i 0 -i 1 -i 2 -e -+n -r4K -r 8K -r 32K -r 64K -r 128K -s 1G Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 1048576 4 66215 0 204121 0 162069 35408 1048576 8 54523 0 168679 0 137393 29711 1048576 32 10293 0 80652 0 75063 10462 1048576 64 23065 0 49044 0 46684 20179 1048576 128 16755 0 25715 0 25240 16125 Settings that i have apart default are: cat /boot/loader.conf zfs_load="YES" kern.geom.label.gptid.enable="0" nvme_load="YES" nvd_load="YES" vfs.zfs.vdev.trim_on_init=0 vfs.zfs.trim.enabled=0 kern.ipc.nmbjumbo16=262144 kern.ipc.nmbjumbo9=262144 kern.ipc.nmbclusters=262144 kern.ipc.nmbjumbop=262144 net.inet.tcp.maxtcptw=163840 hw.intr_storm_threshold="9000" vfs.zfs.cache_flush_disable=1 #avoid sending flushes to prevent useless delays in buggy low end ssds vfs.zfs.vdev.cache.bshift=13 net.inet.tcp.tcbhashsize=32768 vfs.zfs.arc_max=34359738368 /etc/sysctl.conf net.inet.tcp.fast_finwait2_recycle=1 net.inet.ip.portrange.randomized=0 net.inet.ip.portrange.first=1024 net.inet.ip.portrange.last=65535 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_inc=65536 vfs.zfs.vdev.trim_on_init=0 #close time_wait connections at 2*7500ms net.inet.tcp.msl=7500 kern.ipc.somaxconn=4096 net.inet.icmp.icmplim=2000 #zfs vfs.zfs.txg.timeout=5 vfs.zfs.prefetch_disable=1 What i've tried is setting hw.nvme.per_cpu_io_queues to 0, but doesn't seem to have any effect. Setting hw.nvme.force_intx=1 leads to system not booting at all. From what it seems issues is on nvme driver as perftest outputs: nvmecontrol perftest -n 32 -o read -s 512 -t30 nvme0ns1 Threads: 32 Size: 512 READ Time: 30 IO/s: 270212 MB/s: 131 nvmecontrol perftest -n 32 -o write -s 512 -t30 nvme0ns1 Threads: 32 Size: 512 WRITE Time: 30 IO/s: 13658 MB/s: 6 Any help to bring this device to proper speed will be welcomed. -- Best regards, Vintila Mihai Alexandru