From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 1 10:16:34 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D8F2B907; Wed, 1 Apr 2015 10:16:34 +0000 (UTC) Received: from mail-wg0-x233.google.com (mail-wg0-x233.google.com [IPv6:2a00:1450:400c:c00::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7EE783DF; Wed, 1 Apr 2015 10:16:34 +0000 (UTC) Received: by wgdm6 with SMTP id m6so47897868wgd.2; Wed, 01 Apr 2015 03:16:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :content-type:content-transfer-encoding; bh=ed5Uz9Ct7ankujhrn3o/I9MLizxCg+7SzlI9XCE8/EE=; b=eBOPpvphcM7gQ/KKcJ6l0lYDBuF8Sa7fvbQF5wIuyPHjVZhRDvlhygb0b51p0UmHjy 4t9kQ4hf175usJMOA/fw+ZqAB7Bdk4ZoVQpx0EZobNIPHxbkM3w3rNTOlG1nX0hilepW /tbX9P8+cxY3OZrw94mbFaTfMLfVZGN/15FmYTMNFMZFyMrENGIt5zzywWQXF8x4z4Uc +P0ok2Y9fBVWLe0BkQrX/ljOwTVoI6NTecnX14EvjmPYNL+okctJEZyh6LRadTXiDS7q sFM4F3WedlGZu3CUZvRyYs/MmoPelNBRXc1nt8xVwn3STZPM40Uj/PZFakAGGbjvtkl6 CycA== X-Received: by 10.180.87.66 with SMTP id v2mr13088738wiz.51.1427883392751; Wed, 01 Apr 2015 03:16:32 -0700 (PDT) Received: from [192.168.43.73] ([89.204.135.212]) by mx.google.com with ESMTPSA id r3sm1998719wjw.7.2015.04.01.03.16.31 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 01 Apr 2015 03:16:31 -0700 (PDT) Message-ID: <551BC57D.5070101@gmail.com> Date: Wed, 01 Apr 2015 12:16:29 +0200 From: Tobias Oberstein User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: NVMe performance 4x slower than expected Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Cc: jimharris@freebsd.org, Michael Fuckner , kib@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Apr 2015 10:16:34 -0000 Hi, I am testing performance of a NVMe device (Intel P3700) using FIO at the block device level and get 4x slower performance than expected: 4kB Random Read Intel Datasheet FIO Measurement Match P3700 450,000 107,092 24% DC S3700 75,000 67,186 90% The 2nd line are results for an Intel DC S3700 for comparison (with this device, I do see the performance expected, but not for the P3700). Hardware: - 4 sockets, 48 core x86-64, 3TB RAM - 8 x Intel P3700 2TB - 12 x Intel DC S3700 800GB (via LSI HBAs) Software: FreeBSD 11 Current with patches (DMAR and ZFS patches, otherwise the box doesn't boot at all .. because of 3TB RAM and the amount of periphery). Complete info and test logs are here: https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/perftests.md Right now I am running Linux on the box (openSUSE 13.2). Using the exact same FIO control file, the values for the DC S3700 are very close to FreeBSD, but the values for the P3700 are much higher: https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/perftests.md#more-numbers-linux I am looking for tuning hints or general advice for FreeBSD and NVMe. I would like to go with FreeBSD (a major aspect is ZFS), but the performance issues with NVMe might be a deal breaker. Cheers, /Tobias