Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 09 Sep 2016 14:14:50 -0700
From:      Dennis Glatting <>
To:        Christoph Pilka <>,
Subject:   Re: 40 cores, 48 NVMe disks, feel free to take over
Message-ID:  <>
In-Reply-To: <>
References:  <>

Next in thread | Previous in thread | Raw E-Mail | Index | Archive | Help
On Fri, 2016-09-09 at 22:51 +0200, Christoph Pilka wrote:
> Hi,
> we've just been granted a short-term loan of a server from Supermicro
> with 40 physical cores (plus HTT) and 48 NVMe drives. After a bit of
> mucking about, we managed to get 11-RC running. A couple of things
> are preventing the system from being terribly useful:
> - We have to use hw.nvme.force_intx=1 for the server to boot
> If we don't, it panics around the 9th NVMe drive with "panic:
> couldn't find an APIC vector for IRQ...". Increasing
> hw.nvme.min_cpus_per_ioq brings it further, but it still panics later
> in the NVMe enumeration/init. hw.nvme.per_cpu_io_queues=0 causes it
> to panic later (I suspect during ixl init - the box has 4x10gb
> ethernet ports).
> - zfskern seems to be the limiting factor when doing ~40 parallel "dd
> if=/dev/zer of=<file> bs=1m" on a zpool stripe of all 48 drives. Each
> drive shows ~30% utilization (gstat), I can do ~14GB/sec write and 16
> read.
> - direct writing to the NVMe devices (dd from /dev/zero) gives about
> 550MB/sec and ~91% utilization per device 
> Obviously, the first item is the most troublesome. The rest is based
> on entirely synthetic testing and may have little or no actual impact
> on the server's usability or fitness for our purposes. 
> There is nothing but sshd running on the server, and if anyone wants
> to play around you'll have IPMI access (remote kvm, virtual media,
> power) and root.
> Any takers?

I'm curious to know what board you have. I have had FreeBSD, including
release 11 candidates, running on SM boards without any trouble
although some of them are older boards. I haven't looked at ZFS
performance because mine are typically low disk use. That said, my
virtual server (also a SM) IOPs suck but so do its disks.

I recently found the Intel RAID chip on one SM isn't real RAID, rather
it's pseudo RAID but for a few dollars more it could be real RAID. :(
It was killing IOPs so I popped in an old LSI board, routed the cables
from the Intel chip, and the server is now a happy camper. I then
replaced 11-RC with Ubuntu 16.10 due to a specific application but I am
also running RAIDz2 under Ubuntu on three trash 2.5T disks (I didn't do
this for any reason other than fun). 

root@Tuck3r:/opt/bin# zpool status
  pool: opt
 state: ONLINE
  scan: none requested

	opt         ONLINE       0     0     0
	  raidz2-0  ONLINE       0     0     0
	    sda     ONLINE       0     0     0
	    sdb     ONLINE       0     0     0
	    sdc     ONLINE       0     0     0

> Wbr
> Christoph Pilka
> Modirum MDpay
> Sent from my iPhone
> _______________________________________________
> mailing list
> To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freeb

Want to link to this message? Use this URL: <>