From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 1 20:52:21 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 78FEEF67; Wed, 1 Apr 2015 20:52:21 +0000 (UTC) Received: from mail-wi0-x22b.google.com (mail-wi0-x22b.google.com [IPv6:2a00:1450:400c:c05::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 069A7924; Wed, 1 Apr 2015 20:52:21 +0000 (UTC) Received: by widdi4 with SMTP id di4so58280309wid.0; Wed, 01 Apr 2015 13:52:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=ae8+fGkJBec3jxpbWZRCYEVqh3a+tJMkWwBzsCbTiSo=; b=uFb7PLQTG1hTCWs0ANYKF15ZjBDGEdevhCxHKXkcv1aweO5n00VcgdstJSaFPIt5nz XuTlfhL+NNejSu+8uzcnNttrDbBOAoaig9c+EwABlZUssS3yB0g+wFMC4D9pBdm4Epz1 kz01GIMTDRdf8WdORALAZK1PfY3yK0i/dg/YgTL99FWPg3lGuMx+UizS1P1A1TRi46ZE j2SC1NX8ekhLu4yMRhcy7ZyfezRllIqK0v+rp5OuZTlo8zHGv1F3w4kOzX0nBsxJiE// nZLOG316/f/wDvydQT8MAp7pMaqCVIsA+Xinr38iCa1wVAwZ2uLVWRBvJbP82klRW5zl mk6w== X-Received: by 10.194.77.230 with SMTP id v6mr89147447wjw.25.1427921539485; Wed, 01 Apr 2015 13:52:19 -0700 (PDT) Received: from [172.22.3.210] (p578b7aa0.dip0.t-ipconnect.de. [87.139.122.160]) by mx.google.com with ESMTPSA id dx11sm4267099wjb.23.2015.04.01.13.52.18 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 01 Apr 2015 13:52:18 -0700 (PDT) Message-ID: <551C5A82.2090306@gmail.com> Date: Wed, 01 Apr 2015 22:52:18 +0200 From: Tobias Oberstein User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: Jim Harris , Alan Somers Subject: Re: NVMe performance 4x slower than expected References: <551BC57D.5070101@gmail.com> In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-hackers@freebsd.org" , Michael Fuckner , Konstantin Belousov X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Apr 2015 20:52:21 -0000 > > FreeBSD 11 Current with patches (DMAR and ZFS patches, otherwise the box > > doesn't boot at all .. because of 3TB RAM and the amount of periphery). > > Do you still have WITNESS and INVARIANTS turned on in your kernel > config? They're turned on by default for Current, but they do have > some performance impact. To turn them off, just build a > GENERIC-NODEBUG kernel . WITNESS is off, INVARIANTS is still on. Here is complete config: https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_kernel_conf.md This is the aggregated patch (work was done by Konstantin - thanks again btw!) https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_patch.md > Could you also post full dmesg output as well as vmstat -i? dmesg: https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_dmesg.md vmstat: https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_vmstat.md === Here are results from FIO under FreeBSD: https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd.md Here are results using _same_ FIO control file under Linux: https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/linux.md === The firmware for the P3700 cards was updated to the very latest as of today (using isdct under Linux). /Tobias