Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 10 Sep 2016 10:57:07 +0200
From:      Christoph Pilka <c.pilka@asconix.com>
To:        freebsd-questions@freebsd.org
Subject:   Re: 40 cores, 48 NVMe disks, feel free to take over 
Message-ID:  <CCE56C3D-19ED-4A10-942C-16C5379028C1@asconix.com>
In-Reply-To: <1473455690.58708.93.camel@pki2.com>
References:  <E264C60F-7317-4D99-882C-8F76191238BE@asconix.com> <1473455690.58708.93.camel@pki2.com>

Next in thread | Previous in thread | Raw E-Mail | Index | Archive | Help
Hi,

the server we got to experiment with is the SuperMicro 2028R-NR48N =
(https://www.supermicro.nl/products/system/2U/2028/SSG-2028R-NR48N.cfm =
<https://www.supermicro.nl/products/system/2U/2028/SSG-2028R-NR48N.cfm>), =
the board itself is a X10DSC+

//Chris

> On 09 Sep 2016, at 23:14, Dennis Glatting <freebsd@pki2.com> wrote:
>=20
> On Fri, 2016-09-09 at 22:51 +0200, Christoph Pilka wrote:
>> Hi,
>>=20
>> we've just been granted a short-term loan of a server from Supermicro
>> with 40 physical cores (plus HTT) and 48 NVMe drives. After a bit of
>> mucking about, we managed to get 11-RC running. A couple of things
>> are preventing the system from being terribly useful:
>>=20
>> - We have to use hw.nvme.force_intx=3D1 for the server to boot
>> If we don't, it panics around the 9th NVMe drive with "panic:
>> couldn't find an APIC vector for IRQ...". Increasing
>> hw.nvme.min_cpus_per_ioq brings it further, but it still panics later
>> in the NVMe enumeration/init. hw.nvme.per_cpu_io_queues=3D0 causes it
>> to panic later (I suspect during ixl init - the box has 4x10gb
>> ethernet ports).
>>=20
>> - zfskern seems to be the limiting factor when doing ~40 parallel "dd
>> if=3D/dev/zer of=3D<file> bs=3D1m" on a zpool stripe of all 48 =
drives. Each
>> drive shows ~30% utilization (gstat), I can do ~14GB/sec write and 16
>> read.
>>=20
>> - direct writing to the NVMe devices (dd from /dev/zero) gives about
>> 550MB/sec and ~91% utilization per device=20
>>=20
>> Obviously, the first item is the most troublesome. The rest is based
>> on entirely synthetic testing and may have little or no actual impact
>> on the server's usability or fitness for our purposes.=20
>>=20
>> There is nothing but sshd running on the server, and if anyone wants
>> to play around you'll have IPMI access (remote kvm, virtual media,
>> power) and root.
>>=20
>> Any takers?
>>=20
>=20
>=20
> I'm curious to know what board you have. I have had FreeBSD, including
> release 11 candidates, running on SM boards without any trouble
> although some of them are older boards. I haven't looked at ZFS
> performance because mine are typically low disk use. That said, my
> virtual server (also a SM) IOPs suck but so do its disks.
>=20
> I recently found the Intel RAID chip on one SM isn't real RAID, rather
> it's pseudo RAID but for a few dollars more it could be real RAID. :(
> It was killing IOPs so I popped in an old LSI board, routed the cables
> from the Intel chip, and the server is now a happy camper. I then
> replaced 11-RC with Ubuntu 16.10 due to a specific application but I =
am
> also running RAIDz2 under Ubuntu on three trash 2.5T disks (I didn't =
do
> this for any reason other than fun).=20
>=20
> root@Tuck3r:/opt/bin# zpool status
>   pool: opt
>  state: ONLINE
>   scan: none requested
> config:
>=20
> 	NAME        STATE     READ WRITE CKSUM
> 	opt         ONLINE       0     0     0
> 	  raidz2-0  ONLINE       0     0     0
> 	    sda     ONLINE       0     0     0
> 	    sdb     ONLINE       0     0     0
> 	    sdc     ONLINE       0     0     0
>=20
>=20
>=20
>> Wbr
>> Christoph Pilka
>> Modirum MDpay
>>=20
>> Sent from my iPhone
>> _______________________________________________
>> freebsd-questions@freebsd.org <mailto:freebsd-questions@freebsd.org> =
mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-questions =
<https://lists.freebsd.org/mailman/listinfo/freebsd-questions>;
>> To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freeb
>> sd.org <http://sd.org/>"
> _______________________________________________
> freebsd-questions@freebsd.org <mailto:freebsd-questions@freebsd.org> =
mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-questions =
<https://lists.freebsd.org/mailman/listinfo/freebsd-questions>;
> To unsubscribe, send any mail to =
"freebsd-questions-unsubscribe@freebsd.org =
<mailto:freebsd-questions-unsubscribe@freebsd.org>"




Want to link to this message? Use this URL: <http://docs.FreeBSD.org/cgi/mid.cgi?CCE56C3D-19ED-4A10-942C-16C5379028C1>