Date: Thu, 4 Apr 2019 10:37:58 +0200 From: "Patrick M. Hausen" <hausen@punkt.de> To: freebsd-stable@freebsd.org Subject: NVME aborting outstanding i/o Message-ID: <CF2365AE-23EA-4F18-9520-C998216155D5@punkt.de> References: <818CF16A-D71C-47C0-8A1B-35C9D8F68F4E@punkt.de>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi all,
I=E2=80=99m currently doing some load tests/burn in for two new servers.
These feature all NVME SSDs and run FreeNAS, i.e. FreeBSD 11.2-STABLE.
pcib17: <ACPI PCI-PCI bridge> at device 3.2 numa-domain 1 on =
pci15
pcib17: [GIANT-LOCKED]
pci17: <ACPI PCI bus> numa-domain 1 on pcib17
nvme7: <Generic NVMe Device> mem 0xeca10000-0xeca13fff at device =
0.0 numa-domain 1 on pci17
When putting some moderate i/o load on the system, the log fills with =
these
messages:
nvme7: aborting outstanding i/o
nvme7: DATASET MANAGEMENT sqid:41 cid:91 nsid:1
nvme7: ABORTED - BY REQUEST (00/07) sqid:41 cid:91 cdw0:0
There has been some discussion of this on on the iX Systems forum as =
well as various
FreeBSD media and one person suggested setting:
hw.nvme.per_cpu_io_queues=3D0
This is where I need some help now. This is from the manpage for =
nvme(4):
----------
To force a single I/O queue pair shared by all CPUs, set the =
following
tunable value in loader.conf(5):
hw.nvme.per_cpu_io_queues=3D0
To assign more than one CPU per I/O queue pair, thereby reducing the
number of MSI-X vectors consumed by the device, set the following =
tunable
value in loader.conf(5):
hw.nvme.min_cpus_per_ioq=3DX
To force legacy interrupts for all nvme driver instances, set the
following tunable value in loader.conf(5):
hw.nvme.force_intx=3D1
Note that use of INTx implies disabling of per-CPU I/O queue pairs.
----------
But:
root@freenas01[~]# sysctl hw.nvme.per_cpu_io_queues
sysctl: unknown oid 'hw.nvme.per_cpu_io_queues'
root@freenas01[~]# sysctl hw.nvme.min_cpus_per_ioq
sysctl: unknown oid 'hw.nvme.min_cpus_per_ioq'
root@freenas01[~]# sysctl hw.nvme.force_intx
sysctl: unknown oid 'hw.nvme.force_intx'
Where do I go from here?
Thanks!
Patrick
--=20
punkt.de GmbH Internet - Dienstleistungen - Beratung
Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100
76133 Karlsruhe info@punkt.de http://punkt.de
AG Mannheim 108285 Gf: Juergen Egeling
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CF2365AE-23EA-4F18-9520-C998216155D5>