From owner-freebsd-fs@freebsd.org Tue May 17 12:08:57 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9B074B3F73D for ; Tue, 17 May 2016 12:08:57 +0000 (UTC) (envelope-from lexa@lexa.ru) Received: from mx3.lexa.ru (ns503534.ip-198-27-68.net [198.27.68.102]) by mx1.freebsd.org (Postfix) with ESMTP id 808251480 for ; Tue, 17 May 2016 12:08:56 +0000 (UTC) (envelope-from lexa@lexa.ru) Received: by mx3.lexa.ru (Postfix, from userid 66) id C8433224A5E; Tue, 17 May 2016 08:00:07 -0400 (EDT) Received: from [193.124.130.166] (unknown [193.124.130.166]) by home-gw.lexa.ru (Postfix) with ESMTP id 6F73616DE for ; Tue, 17 May 2016 15:00:03 +0300 (MSK) To: freebsd-fs@freebsd.org From: Alex Tutubalin Subject: ZFS performance bottlenecks: CPU or RAM or anything else? Message-ID: <8441f4c0-f8d1-f540-b928-7ae60998ba8e@lexa.ru> Date: Tue, 17 May 2016 15:00:03 +0300 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.1.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 May 2016 12:08:57 -0000 Hi, I'm new to the list, sorry if the subject was discussed earlier (for many times), just point to archives.... I'm building new storage server for 'linear read/linear write' performance with limited number of parallel data streams (load like read/write multi-gigabyte photoshop files, or read many large raw photo files). Target is to saturate 10G link using SMB or iSCSI. Several years ago I've tested small zpool (5x3Tb 7200rpm drives in RAIDZ) with different CPU/memory combos and have got these results for linear write speed by big chunks: 440 Mb/sec with Core i3-2120/DDR3-1600 ram (2 channel) 360 Mb/sec with core i7-920/DDR3-1333 (3 channel RAM) 280 Mb/sec with Core 2Q Q9300 /DDR2-800 (2 channel) Mixed thoughts: i7-920 is fastest of the three, RAM linear access also fastest, but beaten by i3-2120 with lower latency memory. Also, I've found this link: https://calomel.org/zfs_raid_speed_capacity.html For 6x SSD and 10x SSD in RAIDZ2, there is very similar read speed (1.7Gb/sec) and very close in write speed (721/806 Mb/sec for 6/10 drives). Assuming HBA/PCIe performance to be very same for read and write operations, write speed is not limited by HBA/bus... so it is limited by what? CPU or RAM or ...? So, my question is 'what CPU/memory is optimal for ZFS performance'? In particular: - DDR3 or DDR4 (twice the bandwidth) ? - limited number of cores and high clock rate (e.g. i3-6xxxx) or many cores/slower clock ? No plans to use compression or deduplication, only raidz2 with 8-10 HDD spindles and 3-4-5 SSDs for L2ARC. Alex Tutubalin lexa@lexa.ru