From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 1 14:43:05 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 347932D9; Wed, 1 Apr 2015 14:43:05 +0000 (UTC) Received: from mail-ob0-x22e.google.com (mail-ob0-x22e.google.com [IPv6:2607:f8b0:4003:c01::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E3E05859; Wed, 1 Apr 2015 14:43:04 +0000 (UTC) Received: by obvd1 with SMTP id d1so82780002obv.0; Wed, 01 Apr 2015 07:43:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=TisC5IgROSeSNWtZIjwsQ83UedHNwjPEoekuaU+qQ1k=; b=r0Igkcta5XEa3CwJcBhb5W6g7REWoZV5d72vDKnv3gBJj6A40EjtHnkEqXDQJvHHHZ C0tRRxMIp9f7VYx8DU31cl2GLvpjcJyhDrZaxrHoie+zFmFQuS6fZqke0fCHibeGM9eF N7WPGARbOy3oG/HLQsKXKKhCmnPQvSPKQNJlmwnbXr5w/3UE6f8QwMtp2y8SbHm8Y/Mt Hyy6ZZmYJaRw3IEz49FXn2De6aRJ1/0vBzdsa+WWEwQ5TxZ9S+/pAXYKhN36mWVwGg6d uV6Tj+RKiHc6sTZIVNZB8Tt4Vbw0NW3eM4D4St1tcqWXdCaOyy+rgL0WX3iar7hKlqJw Mupg== MIME-Version: 1.0 X-Received: by 10.182.241.197 with SMTP id wk5mr11706681obc.0.1427899371005; Wed, 01 Apr 2015 07:42:51 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.202.215.7 with HTTP; Wed, 1 Apr 2015 07:42:50 -0700 (PDT) In-Reply-To: <551BC57D.5070101@gmail.com> References: <551BC57D.5070101@gmail.com> Date: Wed, 1 Apr 2015 08:42:50 -0600 X-Google-Sender-Auth: X7Cma5wLHbeOkUkQxetYaXHX_1I Message-ID: Subject: Re: NVMe performance 4x slower than expected From: Alan Somers To: Tobias Oberstein Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" , Michael Fuckner , kib@freebsd.org, jimharris@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Apr 2015 14:43:05 -0000 On Wed, Apr 1, 2015 at 4:16 AM, Tobias Oberstein wrote: > Hi, > > I am testing performance of a NVMe device (Intel P3700) using FIO at the > block device level and get 4x slower performance than expected: > > 4kB Random Read > > Intel Datasheet FIO Measurement Match > P3700 450,000 107,092 24% > DC S3700 75,000 67,186 90% > > The 2nd line are results for an Intel DC S3700 for comparison (with this > device, I do see the performance expected, but not for the P3700). > > Hardware: > > - 4 sockets, 48 core x86-64, 3TB RAM > - 8 x Intel P3700 2TB > - 12 x Intel DC S3700 800GB (via LSI HBAs) > > Software: > > FreeBSD 11 Current with patches (DMAR and ZFS patches, otherwise the box > doesn't boot at all .. because of 3TB RAM and the amount of periphery). Do you still have WITNESS and INVARIANTS turned on in your kernel config? They're turned on by default for Current, but they do have some performance impact. To turn them off, just build a GENERIC-NODEBUG kernel . > > Complete info and test logs are here: > > https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/perftests.md > > Right now I am running Linux on the box (openSUSE 13.2). Using the exact > same FIO control file, the values for the DC S3700 are very close to > FreeBSD, but the values for the P3700 are much higher: > > https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/perftests.md#more-numbers-linux > > I am looking for tuning hints or general advice for FreeBSD and NVMe. > > I would like to go with FreeBSD (a major aspect is ZFS), but the performance > issues with NVMe might be a deal breaker. > > Cheers, > /Tobias > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"