From owner-freebsd-stable@FreeBSD.ORG  Thu Jan 15 17:59:31 2015
Return-Path: <owner-freebsd-stable@FreeBSD.ORG>
Delivered-To: freebsd-stable@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 0A4B2DAC
 for <freebsd-stable@freebsd.org>; Thu, 15 Jan 2015 17:59:31 +0000 (UTC)
Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id B9BEC843
 for <freebsd-stable@freebsd.org>; Thu, 15 Jan 2015 17:59:30 +0000 (UTC)
Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD))
 (envelope-from <slw@zxy.spb.ru>)
 id 1YBohb-00056m-GY; Thu, 15 Jan 2015 20:59:27 +0300
Date: Thu, 15 Jan 2015 20:59:27 +0300
From: Slawa Olhovchenkov <slw@zxy.spb.ru>
To: Mihai Vintila <unixro@gmail.com>
Subject: Re: Poor performance on Intel P3600 NVME driver
Message-ID: <20150115175927.GA19071@zxy.spb.ru>
References: <54B7F769.40605@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <54B7F769.40605@gmail.com>
User-Agent: Mutt/1.5.23 (2014-03-12)
X-SA-Exim-Connect-IP: <locally generated>
X-SA-Exim-Mail-From: slw@zxy.spb.ru
X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false
Cc: freebsd-stable@freebsd.org
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-stable>,
 <mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable/>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
 <mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 15 Jan 2015 17:59:31 -0000

On Thu, Jan 15, 2015 at 07:22:49PM +0200, Mihai Vintila wrote:

> /etc/sysctl.conf
> 
> net.inet.tcp.fast_finwait2_recycle=1
> net.inet.ip.portrange.randomized=0
> net.inet.ip.portrange.first=1024
> net.inet.ip.portrange.last=65535
> net.inet.tcp.recvbuf_max=16777216
> net.inet.tcp.sendbuf_max=16777216
> net.inet.tcp.recvbuf_inc=65536
> vfs.zfs.vdev.trim_on_init=0
> #close time_wait connections at 2*7500ms
> net.inet.tcp.msl=7500
> kern.ipc.somaxconn=4096
> net.inet.icmp.icmplim=2000
> 
> #zfs
> vfs.zfs.txg.timeout=5
> vfs.zfs.prefetch_disable=1


> Any help to bring this device to proper speed will be welcomed.

Do you try to increase vfs.zfs.top_maxinflight?