Date: Mon, 11 Sep 2017 18:48:09 +0000 (UTC) From: Alexander Motin <mav@FreeBSD.org> To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-11@freebsd.org Subject: svn commit: r323453 - in stable/11: share/man/man4 sys/amd64/conf sys/conf sys/dev/ntb/ntb_hw sys/i386/conf sys/modules/ntb sys/modules/ntb/ntb_hw sys/modules/ntb/ntb_hw_intel sys/modules/ntb/ntb_h... Message-ID: <201709111848.v8BIm9xr075694@repo.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: mav Date: Mon Sep 11 18:48:09 2017 New Revision: 323453 URL: https://svnweb.freebsd.org/changeset/base/323453 Log: MFC r323032, r323053, r323058, r323059, r323084, r323114, r323127: Add NTB driver for PLX/Avago/Broadcom PCIe switches. This driver supports both NTB-to-NTB and NTB-to-Root Port modes (though the second with predictable complications on hot-plug and reboot events). I tested it with PEX 8717 and PEX 8733 chips, but expect it should work with many other compatible ones too. It supports up to two NT bridges per chip, each of which can have up to 2 64-bit or 4 32-bit memory windows, 6 or 12 scratchpad registers and 16 doorbells. There are also 4 DMA engines in those chips, but they are not yet supported. While there, rename Intel NTB driver from generic ntb_hw(4) to more specific ntb_hw_intel(4), so now it is on par with this new ntb_hw_plx(4) driver and alike to Linux naming. Added: stable/11/share/man/man4/ntb.4 - copied, changed from r323032, head/share/man/man4/ntb.4 stable/11/share/man/man4/ntb_hw_intel.4 - copied unchanged from r323032, head/share/man/man4/ntb_hw_intel.4 stable/11/share/man/man4/ntb_hw_plx.4 - copied unchanged from r323032, head/share/man/man4/ntb_hw_plx.4 stable/11/sys/dev/ntb/ntb_hw/ntb_hw_intel.c - copied unchanged from r323032, head/sys/dev/ntb/ntb_hw/ntb_hw_intel.c stable/11/sys/dev/ntb/ntb_hw/ntb_hw_intel.h - copied unchanged from r323032, head/sys/dev/ntb/ntb_hw/ntb_hw_intel.h stable/11/sys/dev/ntb/ntb_hw/ntb_hw_plx.c - copied, changed from r323032, head/sys/dev/ntb/ntb_hw/ntb_hw_plx.c stable/11/sys/modules/ntb/ntb_hw_intel/ - copied from r323032, head/sys/modules/ntb/ntb_hw_intel/ stable/11/sys/modules/ntb/ntb_hw_plx/ - copied from r323032, head/sys/modules/ntb/ntb_hw_plx/ Deleted: stable/11/share/man/man4/ntb_hw.4 stable/11/sys/dev/ntb/ntb_hw/ntb_hw.c stable/11/sys/dev/ntb/ntb_hw/ntb_regs.h stable/11/sys/modules/ntb/ntb_hw/ Modified: stable/11/share/man/man4/Makefile stable/11/share/man/man4/if_ntb.4 stable/11/share/man/man4/ntb_transport.4 stable/11/sys/amd64/conf/NOTES stable/11/sys/conf/files.amd64 stable/11/sys/conf/files.i386 stable/11/sys/i386/conf/NOTES stable/11/sys/modules/ntb/Makefile Directory Properties: stable/11/ (props changed) Modified: stable/11/share/man/man4/Makefile ============================================================================== --- stable/11/share/man/man4/Makefile Mon Sep 11 18:31:51 2017 (r323452) +++ stable/11/share/man/man4/Makefile Mon Sep 11 18:48:09 2017 (r323453) @@ -383,7 +383,9 @@ MAN= aac.4 \ ng_vlan.4 \ nmdm.4 \ nsp.4 \ - ${_ntb_hw.4} \ + ${_ntb.4} \ + ${_ntb_hw_intel.4} \ + ${_ntb_hw_plx.4} \ ${_ntb_transport.4} \ ${_if_ntb.4} \ null.4 \ @@ -707,7 +709,6 @@ MLINKS+=netintro.4 net.4 \ netintro.4 networking.4 MLINKS+=${_nfe.4} ${_if_nfe.4} MLINKS+=nge.4 if_nge.4 -MLINKS+=${_ntb_hw.4} ${_ntb.4} MLINKS+=${_nxge.4} ${_if_nxge.4} MLINKS+=ow.4 onewire.4 MLINKS+=patm.4 if_patm.4 @@ -853,7 +854,8 @@ _xnb.4= xnb.4 _if_ntb.4= if_ntb.4 _ioat.4= ioat.4 _ntb.4= ntb.4 -_ntb_hw.4= ntb_hw.4 +_ntb_hw_intel.4= ntb_hw_intel.4 +_ntb_hw_plx.4= ntb_hw_plx.4 _ntb_transport.4=ntb_transport.4 _qlxge.4= qlxge.4 _qlxgb.4= qlxgb.4 Modified: stable/11/share/man/man4/if_ntb.4 ============================================================================== --- stable/11/share/man/man4/if_ntb.4 Mon Sep 11 18:31:51 2017 (r323452) +++ stable/11/share/man/man4/if_ntb.4 Mon Sep 11 18:48:09 2017 (r323453) @@ -25,7 +25,7 @@ .\" .\" $FreeBSD$ .\" -.Dd July 29, 2016 +.Dd September 2, 2017 .Dt IF_NTB 4 .Os .Sh NAME @@ -35,6 +35,8 @@ To compile this driver into your kernel, place the following lines in your kernel configuration file: .Bd -ragged -offset indent +.Cd "device ntb" +.Cd "device ntb_transport" .Cd "device if_ntb" .Ed .Pp @@ -48,7 +50,7 @@ The following tunables are settable from the .Xr loader 8 : .Bl -ohang .It Va hw.if_ntb.num_queues -Number of transport queues to use per interface. +Limits maximal number of queues per interface. Default is unlimited. .El .Sh DESCRIPTION @@ -56,12 +58,10 @@ The .Nm driver attaches on top of the .Xr ntb_transport 4 -driver to utilize its resources to create virtual Ethernet interface between -the systems. -Interface capabilities depend on the underlying transport. -Typical MTU is about 64KB to reduce overhead. -By default one queue is used, but more may be configured. -The MAC address for interface is randomly generated. +driver to utilize one or more of its packet queues to create virtual +Ethernet network interface between the systems. +Typical MTU for the interface is about 64KB to reduce overhead. +Default MAC address for the interface is randomly generated. .Pp The .Nm Copied and modified: stable/11/share/man/man4/ntb.4 (from r323032, head/share/man/man4/ntb.4) ============================================================================== --- head/share/man/man4/ntb.4 Wed Aug 30 21:16:32 2017 (r323032, copy source) +++ stable/11/share/man/man4/ntb.4 Mon Sep 11 18:48:09 2017 (r323453) @@ -25,7 +25,7 @@ .\" .\" $FreeBSD$ .\" -.Dd August 30, 2017 +.Dd September 2, 2017 .Dt NTB 4 .Os .Sh NAME @@ -51,19 +51,19 @@ The following tunables are settable from the Driver debug level. The default value is 0, higher means more verbose. .It Va hint.ntb_hw. Ns Ar X Ns Va .config -Configures NTB resources split between several consumer devices. -Configuration of multiple consumer devices separated by commas. -Each device can be configured as: "[<name>][:<mw>[:<spad>[:<db>]]]", where: +Configures a set of NTB functions, separated by commas, +and their resource allocation. +Each function can be configured as: "[<name>][:<mw>[:<spad>[:<db>]]]", where: .Va name -is a name of the driver which should attach the device (empty means any), +is a name of the driver to attach (empty means any), .Va mw is a number of memory windows to allocate (empty means all available), .Va spad is a number of scratchpad registers to allocate (empty means all available), .Va db is a number of doorbells to allocate (empty means all available). -The default configuration is empty string, which means single device -with all available resources allowing any driver attachment. +The default configuration is empty string, which means single function +with all available resources, allowing any driver to attach. .El .Sh DESCRIPTION Non-Transparent Bridges allow to connect two computer systems with PCIe @@ -72,7 +72,7 @@ scratchpad registers and interrupts. The .Nm subsystem uses those resources provided in generic way by hardware drivers -and splits them between multiple consumer drivers, according to specified +and splits them between several functions, according to specified configuration. .Sh SEE ALSO .Xr if_ntb 4 , Copied: stable/11/share/man/man4/ntb_hw_intel.4 (from r323032, head/share/man/man4/ntb_hw_intel.4) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ stable/11/share/man/man4/ntb_hw_intel.4 Mon Sep 11 18:48:09 2017 (r323453, copy of r323032, head/share/man/man4/ntb_hw_intel.4) @@ -0,0 +1,97 @@ +.\" +.\" Copyright (c) 2016-2017 Alexander Motin <mav@FreeBSD.org> +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +.\" SUCH DAMAGE. +.\" +.\" $FreeBSD$ +.\" +.Dd August 30, 2017 +.Dt NTB_HW_INTEL 4 +.Os +.Sh NAME +.Nm ntb_hw_intel +.Nd Intel(R) Non-Transparent Bridge driver +.Sh SYNOPSIS +To compile this driver into your kernel, +place the following lines in your kernel configuration file: +.Bd -ragged -offset indent +.Cd "device ntb" +.Cd "device ntb_hw_intel" +.Ed +.Pp +Or, to load the driver as a module at boot, place the following line in +.Xr loader.conf 5 : +.Bd -literal -offset indent +ntb_hw_intel_load="YES" +.Ed +.Sh DESCRIPTION +The +.Nm ntb_hw_intel +driver provides support for the Non-Transparent Bridge (NTB) hardware in +Intel Xeon E3/E5 and S1200 processor families, which allow one of their PCIe +ports to be switched from transparent to non-transparent bridge mode. +In this mode bridge looks not as a PCI bridge, but as PCI endpoint device. +The driver hides hardware details, exposing memory windows, scratchpads +and doorbells of the other side via hardware independent KPI to +.Xr ntb 4 +subsystem. +.Pp +The hardware provides 2 or 3 memory windows to the other system's memory, +16 scratchpad registers and 14 or 34 doorbells to interrupt the other system. +On Xeon processors one of memory windows is typically consumed by the driver +itself to workaround multiple hardware erratas. +.Sh CONFIGURATION +The NTB configuration should be set by BIOS. +It includes enabling NTB, choosing between NTB-to-NTB (back-to-back) or +NTB-to-Root Port mode, +enabling split BAR mode (one of two 64-bit BARs can be split into two 32-bit +ones) and configuring BAR sizes in bits (from 12 to 29/39) for both NTB sides. +.Pp +The recommended configuration is NTB-to-NTB mode, split bar is enabled and +all BAR sizes are set to 20 (1 MiB). +This needs to be done on both systems. +.Sh SEE ALSO +.Xr if_ntb 4 , +.Xr ntb_transport 4 , +.Xr ntb 4 , +.Sh AUTHORS +.An -nosplit +The +.Nm +driver was developed by Intel and originally written by +.An Carl Delsey Aq Mt carl@FreeBSD.org . +Later improvements were done by +.An Conrad E. Meyer Aq Mt cem@FreeBSD.org +and +.An Alexander Motin Aq Mt mav@FreeBSD.org . +.Sh BUGS +NTB-to-Root Port mode is not yet supported, but it doesn't look very useful. +.Pp +On Xeon v2/v3/v4 processors split BAR mode should be enabled to allow +SB01BASE_LOCKUP errata workaround to be applied by the driver. +.Pp +There is no way to protect your system from malicious behavior on the other +system once the link is brought up. +Anyone with root or kernel access on the other system can read or write to +any location on your system. +In other words, only connect two systems that completely trust each other. Copied: stable/11/share/man/man4/ntb_hw_plx.4 (from r323032, head/share/man/man4/ntb_hw_plx.4) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ stable/11/share/man/man4/ntb_hw_plx.4 Mon Sep 11 18:48:09 2017 (r323453, copy of r323032, head/share/man/man4/ntb_hw_plx.4) @@ -0,0 +1,118 @@ +.\" +.\" Copyright (c) 2017 Alexander Motin <mav@FreeBSD.org> +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +.\" SUCH DAMAGE. +.\" +.\" $FreeBSD$ +.\" +.Dd August 30, 2017 +.Dt NTB_HW_PLX 4 +.Os +.Sh NAME +.Nm ntb_hw_plx +.Nd PLX/Avago/Broadcom Non-Transparent Bridge driver +.Sh SYNOPSIS +To compile this driver into your kernel, +place the following lines in your kernel configuration file: +.Bd -ragged -offset indent +.Cd "device ntb" +.Cd "device ntb_hw_plx" +.Ed +.Pp +Or, to load the driver as a module at boot, place the following line in +.Xr loader.conf 5 : +.Bd -literal -offset indent +ntb_hw_plx_load="YES" +.Ed +.Pp +The following tunables are settable from the +.Xr loader 8 : +.Bl -ohang +.It Va hint.ntb_hw. Ns Ar X Ns Va .b2b +Being set to 1 (default) tells the driver attached to Virtual Interface of the +NTB that it works in NTB-to-NTB (back-to-back) mode, 0 -- NTB-to-Root Port. +Driver attached to Link Interface (visible from Root Port side) switches to +NTB-to-Root Port mode automatically, but one attached to Virtual Interface +can't detect what is on the other side and require external knowledge. +.El +.Sh DESCRIPTION +The +.Nm +driver provides support for the Non-Transparent Bridge (NTB) hardware in +PLX PCIe bridge chips, which allow up to two of their PCIe ports to be +switched from transparent to non-transparent bridge mode. +In this mode bridge looks not as a PCI bridge, but as PCI endpoint device. +The driver hides hardware details, exposing memory windows, scratchpads +and doorbells of the other side via hardware independent KPI to +.Xr ntb 4 +subsystem. +.Pp +Each PLX NTB provides up to 2 64-bit or 4 32-bit memory windows to the +other system's memory, 6 or 12 scratchpad registers and 16 doorbells to +interrupt the other system. +In NTB-to-NTB mode one of memory windows (or half of it, if bigger then 1MB) +is consumed by the driver itself to access scratchpad and doorbell registers +of the other side. +.Sh HARDWARE +The following PLX/Avago/Broadcom chips are supported by the +.Nm +driver: +.Pp +.Bl -bullet -compact +.It +PEX 8713 +.It +PEX 8717 +.It +PEX 8725 +.It +PEX 8733 +.It +PEX 8749 +.El +.Pp +, but it may also work with other compatible ones. +.Sh CONFIGURATION +The basic chip configuration should be done by serial EEPROM or via i2c. +It includes enabling NTB on one or both sides (choosing between NTB-to-NTB +(back-to-back) and NTB-to-Root Port modes) and configuring BARs sizes. +.Pp +The recommended mode is NTB-to-NTB mode, since while NTB-to-Root Port is +generally supported by the driver, it require PCI hotplug handling on the +Root Port, that may be difficult or cause different kinds of problems. +.Sh SEE ALSO +.Xr if_ntb 4 , +.Xr ntb_transport 4 , +.Xr ntb 4 , +.Sh AUTHORS +.An -nosplit +The +.Nm +driver was written by +.An Alexander Motin Aq Mt mav@FreeBSD.org . +.Sh BUGS +There is no way to protect your system from malicious behavior on the other +system once the link is brought up. +Anyone with root or kernel access on the other system can read or write to +any location on your system. +In other words, only connect two systems that completely trust each other. Modified: stable/11/share/man/man4/ntb_transport.4 ============================================================================== --- stable/11/share/man/man4/ntb_transport.4 Mon Sep 11 18:31:51 2017 (r323452) +++ stable/11/share/man/man4/ntb_transport.4 Mon Sep 11 18:48:09 2017 (r323453) @@ -1,5 +1,5 @@ .\" -.\" Copyright (c) 2016 Alexander Motin <mav@FreeBSD.org> +.\" Copyright (c) 2016-2017 Alexander Motin <mav@FreeBSD.org> .\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without @@ -25,14 +25,21 @@ .\" .\" $FreeBSD$ .\" -.Dd July 29, 2016 +.Dd September 2, 2017 .Dt NTB_TRANSPORT 4 .Os .Sh NAME .Nm ntb_transport .Nd Packet-oriented transport for Non-Transparent Bridges .Sh SYNOPSIS -To load the driver as a module at boot, place the following line in +To compile this driver into your kernel, +place the following lines in your kernel configuration file: +.Bd -ragged -offset indent +.Cd "device ntb" +.Cd "device ntb_transport" +.Ed +.Pp +Or, to load the driver as a module at boot, place the following line in .Xr loader.conf 5 : .Bd -literal -offset indent ntb_transport_load="YES" @@ -45,28 +52,44 @@ The following tunables are settable from the Driver debug level. The default value is 0, higher means more verbose. .It Va hint.ntb_transport. Ns Ar X Ns Va .config -Configures queues allocation for consumer devices, separated by commas. -Each device can be configured as: "<name>[:<queues>]", where: +Configures a set of the transport consumers, separated by commas. +Each consumer can be configured as: "[<name>][:<queues>]", where: .Va name -is a name of the driver which should attach the device (empty means any), +is a name of the driver to attach (empty means any), .Va queues -is a number of queues to allocate (empty means automatic), -The default configuration is empty string, which means single device -with one queue per memory window allowing any driver attachment. +is a number of queues to allocate (empty means automatic). +The default configuration is empty string, which means single consumer +with one queue per memory window, allowing any driver to attach. .El .Sh DESCRIPTION The .Nm driver attaches on top of the .Nm ntb -driver to utilize its resources to create set of bidirectional queues, +driver to utilize its resources to create a set of bidirectional queues, delivering packets between the systems. -The primary purpose of this is to be used by +The primary purpose of this driver is to be used by .Nm if_ntb network interface, but other consumers may also be developed using KPI. +.Pp +Each +.Nm +require from underlying +.Nm ntb +instance: +.Bl -bullet -compact +.It +1 or more memory windows; +.It +6 scratchpads, plus 2 more for each additional memory window; +.It +1 doorbell for each memory window or configured queue. +.El .Sh SEE ALSO .Xr if_ntb 4 , -.Xr ntb_hw 4 +.Xr ntb 4 , +.Xr ntb_hw_intel 4 , +.Xr ntb_hw_plx 4 .Sh AUTHORS .An -nosplit The Modified: stable/11/sys/amd64/conf/NOTES ============================================================================== --- stable/11/sys/amd64/conf/NOTES Mon Sep 11 18:31:51 2017 (r323452) +++ stable/11/sys/amd64/conf/NOTES Mon Sep 11 18:48:09 2017 (r323453) @@ -402,9 +402,14 @@ device iwn6000g2bfw device iwn6050fw device wpifw -# Intel Non-Transparent Bridge (NTB) hardware -device ntb_hw # Hardware Abstraction Layer for the NTB -device if_ntb # Simulated ethernet device using the NTB +# +# Non-Transparent Bridge (NTB) drivers +# +device if_ntb # Virtual NTB network interface +device ntb_transport # NTB packet transport driver +device ntb # NTB hardware interface +device ntb_hw_intel # Intel NTB hardware driver +device ntb_hw_plx # PLX NTB hardware driver # #XXX this stores pointers in a 32bit field that is defined by the hardware Modified: stable/11/sys/conf/files.amd64 ============================================================================== --- stable/11/sys/conf/files.amd64 Mon Sep 11 18:31:51 2017 (r323452) +++ stable/11/sys/conf/files.amd64 Mon Sep 11 18:48:09 2017 (r323453) @@ -324,10 +324,11 @@ dev/hyperv/vmbus/amd64/vmbus_vector.S optional hyper dev/nctgpio/nctgpio.c optional nctgpio dev/nfe/if_nfe.c optional nfe pci dev/ntb/if_ntb/if_ntb.c optional if_ntb -dev/ntb/ntb_transport.c optional if_ntb -dev/ntb/ntb.c optional if_ntb | ntb_hw -dev/ntb/ntb_if.m optional if_ntb | ntb_hw -dev/ntb/ntb_hw/ntb_hw.c optional ntb_hw +dev/ntb/ntb_transport.c optional ntb_transport | if_ntb +dev/ntb/ntb.c optional ntb | ntb_transport | if_ntb | ntb_hw_intel | ntb_hw_plx | ntb_hw +dev/ntb/ntb_if.m optional ntb | ntb_transport | if_ntb | ntb_hw_intel | ntb_hw_plx | ntb_hw +dev/ntb/ntb_hw/ntb_hw_intel.c optional ntb_hw_intel | ntb_hw +dev/ntb/ntb_hw/ntb_hw_plx.c optional ntb_hw_plx | ntb_hw dev/nvd/nvd.c optional nvd nvme dev/nvme/nvme.c optional nvme dev/nvme/nvme_ctrlr.c optional nvme Modified: stable/11/sys/conf/files.i386 ============================================================================== --- stable/11/sys/conf/files.i386 Mon Sep 11 18:31:51 2017 (r323452) +++ stable/11/sys/conf/files.i386 Mon Sep 11 18:48:09 2017 (r323453) @@ -294,10 +294,11 @@ dev/mse/mse_isa.c optional mse isa dev/nctgpio/nctgpio.c optional nctgpio dev/nfe/if_nfe.c optional nfe pci dev/ntb/if_ntb/if_ntb.c optional if_ntb -dev/ntb/ntb_transport.c optional if_ntb -dev/ntb/ntb.c optional if_ntb | ntb_hw -dev/ntb/ntb_if.m optional if_ntb | ntb_hw -dev/ntb/ntb_hw/ntb_hw.c optional ntb_hw +dev/ntb/ntb_transport.c optional ntb_transport | if_ntb +dev/ntb/ntb.c optional ntb | ntb_transport | if_ntb | ntb_hw_intel | ntb_hw_plx | ntb_hw +dev/ntb/ntb_if.m optional ntb | ntb_transport | if_ntb | ntb_hw_intel | ntb_hw_plx | ntb_hw +dev/ntb/ntb_hw/ntb_hw_intel.c optional ntb_hw_intel | ntb_hw +dev/ntb/ntb_hw/ntb_hw_plx.c optional ntb_hw_plx | ntb_hw dev/nvd/nvd.c optional nvd nvme dev/nvme/nvme.c optional nvme dev/nvme/nvme_ctrlr.c optional nvme Copied: stable/11/sys/dev/ntb/ntb_hw/ntb_hw_intel.c (from r323032, head/sys/dev/ntb/ntb_hw/ntb_hw_intel.c) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ stable/11/sys/dev/ntb/ntb_hw/ntb_hw_intel.c Mon Sep 11 18:48:09 2017 (r323453, copy of r323032, head/sys/dev/ntb/ntb_hw/ntb_hw_intel.c) @@ -0,0 +1,3121 @@ +/*- + * Copyright (c) 2016 Alexander Motin <mav@FreeBSD.org> + * Copyright (C) 2013 Intel Corporation + * Copyright (C) 2015 EMC Corporation + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +/* + * The Non-Transparent Bridge (NTB) is a device that allows you to connect + * two or more systems using a PCI-e links, providing remote memory access. + * + * This module contains a driver for NTB hardware in Intel Xeon/Atom CPUs. + * + * NOTE: Much of the code in this module is shared with Linux. Any patches may + * be picked up and redistributed in Linux with a dual GPL/BSD license. + */ + +#include <sys/cdefs.h> +__FBSDID("$FreeBSD$"); + +#include <sys/param.h> +#include <sys/kernel.h> +#include <sys/systm.h> +#include <sys/bus.h> +#include <sys/endian.h> +#include <sys/interrupt.h> +#include <sys/malloc.h> +#include <sys/module.h> +#include <sys/mutex.h> +#include <sys/pciio.h> +#include <sys/queue.h> +#include <sys/rman.h> +#include <sys/sbuf.h> +#include <sys/sysctl.h> +#include <vm/vm.h> +#include <vm/pmap.h> +#include <machine/bus.h> +#include <machine/intr_machdep.h> +#include <machine/resource.h> +#include <dev/pci/pcireg.h> +#include <dev/pci/pcivar.h> + +#include "ntb_hw_intel.h" +#include "../ntb.h" + +#define MAX_MSIX_INTERRUPTS MAX(XEON_DB_COUNT, ATOM_DB_COUNT) + +#define NTB_HB_TIMEOUT 1 /* second */ +#define ATOM_LINK_RECOVERY_TIME 500 /* ms */ +#define BAR_HIGH_MASK (~((1ull << 12) - 1)) + +#define NTB_MSIX_VER_GUARD 0xaabbccdd +#define NTB_MSIX_RECEIVED 0xe0f0e0f0 + +/* + * PCI constants could be somewhere more generic, but aren't defined/used in + * pci.c. + */ +#define PCI_MSIX_ENTRY_SIZE 16 +#define PCI_MSIX_ENTRY_LOWER_ADDR 0 +#define PCI_MSIX_ENTRY_UPPER_ADDR 4 +#define PCI_MSIX_ENTRY_DATA 8 + +enum ntb_device_type { + NTB_XEON, + NTB_ATOM +}; + +/* ntb_conn_type are hardware numbers, cannot change. */ +enum ntb_conn_type { + NTB_CONN_TRANSPARENT = 0, + NTB_CONN_B2B = 1, + NTB_CONN_RP = 2, +}; + +enum ntb_b2b_direction { + NTB_DEV_USD = 0, + NTB_DEV_DSD = 1, +}; + +enum ntb_bar { + NTB_CONFIG_BAR = 0, + NTB_B2B_BAR_1, + NTB_B2B_BAR_2, + NTB_B2B_BAR_3, + NTB_MAX_BARS +}; + +enum { + NTB_MSIX_GUARD = 0, + NTB_MSIX_DATA0, + NTB_MSIX_DATA1, + NTB_MSIX_DATA2, + NTB_MSIX_OFS0, + NTB_MSIX_OFS1, + NTB_MSIX_OFS2, + NTB_MSIX_DONE, + NTB_MAX_MSIX_SPAD +}; + +/* Device features and workarounds */ +#define HAS_FEATURE(ntb, feature) \ + (((ntb)->features & (feature)) != 0) + +struct ntb_hw_info { + uint32_t device_id; + const char *desc; + enum ntb_device_type type; + uint32_t features; +}; + +struct ntb_pci_bar_info { + bus_space_tag_t pci_bus_tag; + bus_space_handle_t pci_bus_handle; + int pci_resource_id; + struct resource *pci_resource; + vm_paddr_t pbase; + caddr_t vbase; + vm_size_t size; + vm_memattr_t map_mode; + + /* Configuration register offsets */ + uint32_t psz_off; + uint32_t ssz_off; + uint32_t pbarxlat_off; +}; + +struct ntb_int_info { + struct resource *res; + int rid; + void *tag; +}; + +struct ntb_vec { + struct ntb_softc *ntb; + uint32_t num; + unsigned masked; +}; + +struct ntb_reg { + uint32_t ntb_ctl; + uint32_t lnk_sta; + uint8_t db_size; + unsigned mw_bar[NTB_MAX_BARS]; +}; + +struct ntb_alt_reg { + uint32_t db_bell; + uint32_t db_mask; + uint32_t spad; +}; + +struct ntb_xlat_reg { + uint32_t bar0_base; + uint32_t bar2_base; + uint32_t bar4_base; + uint32_t bar5_base; + + uint32_t bar2_xlat; + uint32_t bar4_xlat; + uint32_t bar5_xlat; + + uint32_t bar2_limit; + uint32_t bar4_limit; + uint32_t bar5_limit; +}; + +struct ntb_b2b_addr { + uint64_t bar0_addr; + uint64_t bar2_addr64; + uint64_t bar4_addr64; + uint64_t bar4_addr32; + uint64_t bar5_addr32; +}; + +struct ntb_msix_data { + uint32_t nmd_ofs; + uint32_t nmd_data; +}; + +struct ntb_softc { + /* ntb.c context. Do not move! Must go first! */ + void *ntb_store; + + device_t device; + enum ntb_device_type type; + uint32_t features; + + struct ntb_pci_bar_info bar_info[NTB_MAX_BARS]; + struct ntb_int_info int_info[MAX_MSIX_INTERRUPTS]; + uint32_t allocated_interrupts; + + struct ntb_msix_data peer_msix_data[XEON_NONLINK_DB_MSIX_BITS]; + struct ntb_msix_data msix_data[XEON_NONLINK_DB_MSIX_BITS]; + bool peer_msix_good; + bool peer_msix_done; + struct ntb_pci_bar_info *peer_lapic_bar; + struct callout peer_msix_work; + + struct callout heartbeat_timer; + struct callout lr_timer; + + struct ntb_vec *msix_vec; + + uint32_t ppd; + enum ntb_conn_type conn_type; + enum ntb_b2b_direction dev_type; + + /* Offset of peer bar0 in B2B BAR */ + uint64_t b2b_off; + /* Memory window used to access peer bar0 */ +#define B2B_MW_DISABLED UINT8_MAX + uint8_t b2b_mw_idx; + uint32_t msix_xlat; + uint8_t msix_mw_idx; + + uint8_t mw_count; + uint8_t spad_count; + uint8_t db_count; + uint8_t db_vec_count; + uint8_t db_vec_shift; + + /* Protects local db_mask. */ +#define DB_MASK_LOCK(sc) mtx_lock_spin(&(sc)->db_mask_lock) +#define DB_MASK_UNLOCK(sc) mtx_unlock_spin(&(sc)->db_mask_lock) +#define DB_MASK_ASSERT(sc,f) mtx_assert(&(sc)->db_mask_lock, (f)) + struct mtx db_mask_lock; + + volatile uint32_t ntb_ctl; + volatile uint32_t lnk_sta; + + uint64_t db_valid_mask; + uint64_t db_link_mask; + uint64_t db_mask; + uint64_t fake_db; /* NTB_SB01BASE_LOCKUP*/ + uint64_t force_db; /* NTB_SB01BASE_LOCKUP*/ + + int last_ts; /* ticks @ last irq */ + + const struct ntb_reg *reg; + const struct ntb_alt_reg *self_reg; + const struct ntb_alt_reg *peer_reg; + const struct ntb_xlat_reg *xlat_reg; +}; + +#ifdef __i386__ +static __inline uint64_t +bus_space_read_8(bus_space_tag_t tag, bus_space_handle_t handle, + bus_size_t offset) +{ + + return (bus_space_read_4(tag, handle, offset) | + ((uint64_t)bus_space_read_4(tag, handle, offset + 4)) << 32); +} + +static __inline void +bus_space_write_8(bus_space_tag_t tag, bus_space_handle_t handle, + bus_size_t offset, uint64_t val) +{ + + bus_space_write_4(tag, handle, offset, val); + bus_space_write_4(tag, handle, offset + 4, val >> 32); +} +#endif + +#define intel_ntb_bar_read(SIZE, bar, offset) \ + bus_space_read_ ## SIZE (ntb->bar_info[(bar)].pci_bus_tag, \ + ntb->bar_info[(bar)].pci_bus_handle, (offset)) +#define intel_ntb_bar_write(SIZE, bar, offset, val) \ + bus_space_write_ ## SIZE (ntb->bar_info[(bar)].pci_bus_tag, \ + ntb->bar_info[(bar)].pci_bus_handle, (offset), (val)) +#define intel_ntb_reg_read(SIZE, offset) \ + intel_ntb_bar_read(SIZE, NTB_CONFIG_BAR, offset) +#define intel_ntb_reg_write(SIZE, offset, val) \ + intel_ntb_bar_write(SIZE, NTB_CONFIG_BAR, offset, val) +#define intel_ntb_mw_read(SIZE, offset) \ + intel_ntb_bar_read(SIZE, intel_ntb_mw_to_bar(ntb, ntb->b2b_mw_idx), \ + offset) +#define intel_ntb_mw_write(SIZE, offset, val) \ + intel_ntb_bar_write(SIZE, intel_ntb_mw_to_bar(ntb, ntb->b2b_mw_idx), \ + offset, val) + +static int intel_ntb_probe(device_t device); +static int intel_ntb_attach(device_t device); +static int intel_ntb_detach(device_t device); +static uint64_t intel_ntb_db_valid_mask(device_t dev); +static void intel_ntb_spad_clear(device_t dev); +static uint64_t intel_ntb_db_vector_mask(device_t dev, uint32_t vector); +static bool intel_ntb_link_is_up(device_t dev, enum ntb_speed *speed, + enum ntb_width *width); +static int intel_ntb_link_enable(device_t dev, enum ntb_speed speed, + enum ntb_width width); +static int intel_ntb_link_disable(device_t dev); +static int intel_ntb_spad_read(device_t dev, unsigned int idx, uint32_t *val); +static int intel_ntb_peer_spad_write(device_t dev, unsigned int idx, uint32_t val); + +static unsigned intel_ntb_user_mw_to_idx(struct ntb_softc *, unsigned uidx); +static inline enum ntb_bar intel_ntb_mw_to_bar(struct ntb_softc *, unsigned mw); +static inline bool bar_is_64bit(struct ntb_softc *, enum ntb_bar); +static inline void bar_get_xlat_params(struct ntb_softc *, enum ntb_bar, + uint32_t *base, uint32_t *xlat, uint32_t *lmt); +static int intel_ntb_map_pci_bars(struct ntb_softc *ntb); +static int intel_ntb_mw_set_wc_internal(struct ntb_softc *, unsigned idx, + vm_memattr_t); +static void print_map_success(struct ntb_softc *, struct ntb_pci_bar_info *, + const char *); +static int map_mmr_bar(struct ntb_softc *ntb, struct ntb_pci_bar_info *bar); +static int map_memory_window_bar(struct ntb_softc *ntb, + struct ntb_pci_bar_info *bar); +static void intel_ntb_unmap_pci_bar(struct ntb_softc *ntb); +static int intel_ntb_remap_msix(device_t, uint32_t desired, uint32_t avail); +static int intel_ntb_init_isr(struct ntb_softc *ntb); +static int intel_ntb_setup_legacy_interrupt(struct ntb_softc *ntb); +static int intel_ntb_setup_msix(struct ntb_softc *ntb, uint32_t num_vectors); +static void intel_ntb_teardown_interrupts(struct ntb_softc *ntb); +static inline uint64_t intel_ntb_vec_mask(struct ntb_softc *, uint64_t db_vector); +static void intel_ntb_interrupt(struct ntb_softc *, uint32_t vec); +static void ndev_vec_isr(void *arg); +static void ndev_irq_isr(void *arg); +static inline uint64_t db_ioread(struct ntb_softc *, uint64_t regoff); +static inline void db_iowrite(struct ntb_softc *, uint64_t regoff, uint64_t); +static inline void db_iowrite_raw(struct ntb_softc *, uint64_t regoff, uint64_t); +static int intel_ntb_create_msix_vec(struct ntb_softc *ntb, uint32_t num_vectors); +static void intel_ntb_free_msix_vec(struct ntb_softc *ntb); +static void intel_ntb_get_msix_info(struct ntb_softc *ntb); +static void intel_ntb_exchange_msix(void *); +static struct ntb_hw_info *intel_ntb_get_device_info(uint32_t device_id); +static void intel_ntb_detect_max_mw(struct ntb_softc *ntb); +static int intel_ntb_detect_xeon(struct ntb_softc *ntb); +static int intel_ntb_detect_atom(struct ntb_softc *ntb); +static int intel_ntb_xeon_init_dev(struct ntb_softc *ntb); +static int intel_ntb_atom_init_dev(struct ntb_softc *ntb); +static void intel_ntb_teardown_xeon(struct ntb_softc *ntb); +static void configure_atom_secondary_side_bars(struct ntb_softc *ntb); +static void xeon_reset_sbar_size(struct ntb_softc *, enum ntb_bar idx, + enum ntb_bar regbar); +static void xeon_set_sbar_base_and_limit(struct ntb_softc *, + uint64_t base_addr, enum ntb_bar idx, enum ntb_bar regbar); +static void xeon_set_pbar_xlat(struct ntb_softc *, uint64_t base_addr, + enum ntb_bar idx); +static int xeon_setup_b2b_mw(struct ntb_softc *, + const struct ntb_b2b_addr *addr, const struct ntb_b2b_addr *peer_addr); +static inline bool link_is_up(struct ntb_softc *ntb); +static inline bool _xeon_link_is_up(struct ntb_softc *ntb); +static inline bool atom_link_is_err(struct ntb_softc *ntb); +static inline enum ntb_speed intel_ntb_link_sta_speed(struct ntb_softc *); +static inline enum ntb_width intel_ntb_link_sta_width(struct ntb_softc *); +static void atom_link_hb(void *arg); +static void recover_atom_link(void *arg); +static bool intel_ntb_poll_link(struct ntb_softc *ntb); +static void save_bar_parameters(struct ntb_pci_bar_info *bar); +static void intel_ntb_sysctl_init(struct ntb_softc *); +static int sysctl_handle_features(SYSCTL_HANDLER_ARGS); +static int sysctl_handle_link_admin(SYSCTL_HANDLER_ARGS); +static int sysctl_handle_link_status_human(SYSCTL_HANDLER_ARGS); +static int sysctl_handle_link_status(SYSCTL_HANDLER_ARGS); +static int sysctl_handle_register(SYSCTL_HANDLER_ARGS); + +static unsigned g_ntb_hw_debug_level; +SYSCTL_UINT(_hw_ntb, OID_AUTO, debug_level, CTLFLAG_RWTUN, + &g_ntb_hw_debug_level, 0, "ntb_hw log level -- higher is more verbose"); +#define intel_ntb_printf(lvl, ...) do { \ + if ((lvl) <= g_ntb_hw_debug_level) { \ + device_printf(ntb->device, __VA_ARGS__); \ + } \ +} while (0) + +#define _NTB_PAT_UC 0 +#define _NTB_PAT_WC 1 +#define _NTB_PAT_WT 4 +#define _NTB_PAT_WP 5 +#define _NTB_PAT_WB 6 +#define _NTB_PAT_UCM 7 +static unsigned g_ntb_mw_pat = _NTB_PAT_UC; +SYSCTL_UINT(_hw_ntb, OID_AUTO, default_mw_pat, CTLFLAG_RDTUN, + &g_ntb_mw_pat, 0, "Configure the default memory window cache flags (PAT): " + "UC: " __XSTRING(_NTB_PAT_UC) ", " + "WC: " __XSTRING(_NTB_PAT_WC) ", " + "WT: " __XSTRING(_NTB_PAT_WT) ", " + "WP: " __XSTRING(_NTB_PAT_WP) ", " + "WB: " __XSTRING(_NTB_PAT_WB) ", " + "UC-: " __XSTRING(_NTB_PAT_UCM)); + +static inline vm_memattr_t +intel_ntb_pat_flags(void) +{ + + switch (g_ntb_mw_pat) { + case _NTB_PAT_WC: + return (VM_MEMATTR_WRITE_COMBINING); + case _NTB_PAT_WT: + return (VM_MEMATTR_WRITE_THROUGH); + case _NTB_PAT_WP: + return (VM_MEMATTR_WRITE_PROTECTED); + case _NTB_PAT_WB: + return (VM_MEMATTR_WRITE_BACK); + case _NTB_PAT_UCM: + return (VM_MEMATTR_WEAK_UNCACHEABLE); + case _NTB_PAT_UC: + /* FALLTHROUGH */ + default: + return (VM_MEMATTR_UNCACHEABLE); + } +} + +/* + * Well, this obviously doesn't belong here, but it doesn't seem to exist + * anywhere better yet. + */ +static inline const char * +intel_ntb_vm_memattr_to_str(vm_memattr_t pat) +{ + + switch (pat) { + case VM_MEMATTR_WRITE_COMBINING: + return ("WRITE_COMBINING"); + case VM_MEMATTR_WRITE_THROUGH: + return ("WRITE_THROUGH"); + case VM_MEMATTR_WRITE_PROTECTED: + return ("WRITE_PROTECTED"); + case VM_MEMATTR_WRITE_BACK: + return ("WRITE_BACK"); + case VM_MEMATTR_WEAK_UNCACHEABLE: + return ("UNCACHED"); + case VM_MEMATTR_UNCACHEABLE: + return ("UNCACHEABLE"); + default: + return ("UNKNOWN"); + } +} + +static int g_ntb_msix_idx = 1; +SYSCTL_INT(_hw_ntb, OID_AUTO, msix_mw_idx, CTLFLAG_RDTUN, &g_ntb_msix_idx, + 0, "Use this memory window to access the peer MSIX message complex on " + "certain Xeon-based NTB systems, as a workaround for a hardware errata. " + "Like b2b_mw_idx, negative values index from the last available memory " + "window. (Applies on Xeon platforms with SB01BASE_LOCKUP errata.)"); + +static int g_ntb_mw_idx = -1; +SYSCTL_INT(_hw_ntb, OID_AUTO, b2b_mw_idx, CTLFLAG_RDTUN, &g_ntb_mw_idx, + 0, "Use this memory window to access the peer NTB registers. A " + "non-negative value starts from the first MW index; a negative value " + "starts from the last MW index. The default is -1, i.e., the last " + "available memory window. Both sides of the NTB MUST set the same " + "value here! (Applies on Xeon platforms with SDOORBELL_LOCKUP errata.)"); + +/* Hardware owns the low 16 bits of features. */ +#define NTB_BAR_SIZE_4K (1 << 0) +#define NTB_SDOORBELL_LOCKUP (1 << 1) +#define NTB_SB01BASE_LOCKUP (1 << 2) +#define NTB_B2BDOORBELL_BIT14 (1 << 3) +/* Software/configuration owns the top 16 bits. */ +#define NTB_SPLIT_BAR (1ull << 16) + +#define NTB_FEATURES_STR \ + "\20\21SPLIT_BAR4\04B2B_DOORBELL_BIT14\03SB01BASE_LOCKUP" \ + "\02SDOORBELL_LOCKUP\01BAR_SIZE_4K" + +static struct ntb_hw_info pci_ids[] = { + /* XXX: PS/SS IDs left out until they are supported. */ + { 0x0C4E8086, "BWD Atom Processor S1200 Non-Transparent Bridge B2B", + NTB_ATOM, 0 }, + + { 0x37258086, "JSF Xeon C35xx/C55xx Non-Transparent Bridge B2B", + NTB_XEON, NTB_SDOORBELL_LOCKUP | NTB_B2BDOORBELL_BIT14 }, + { 0x3C0D8086, "SNB Xeon E5/Core i7 Non-Transparent Bridge B2B", + NTB_XEON, NTB_SDOORBELL_LOCKUP | NTB_B2BDOORBELL_BIT14 }, + { 0x0E0D8086, "IVT Xeon E5 V2 Non-Transparent Bridge B2B", NTB_XEON, + NTB_SDOORBELL_LOCKUP | NTB_B2BDOORBELL_BIT14 | + NTB_SB01BASE_LOCKUP | NTB_BAR_SIZE_4K }, + { 0x2F0D8086, "HSX Xeon E5 V3 Non-Transparent Bridge B2B", NTB_XEON, + NTB_SDOORBELL_LOCKUP | NTB_B2BDOORBELL_BIT14 | + NTB_SB01BASE_LOCKUP }, + { 0x6F0D8086, "BDX Xeon E5 V4 Non-Transparent Bridge B2B", NTB_XEON, + NTB_SDOORBELL_LOCKUP | NTB_B2BDOORBELL_BIT14 | + NTB_SB01BASE_LOCKUP }, + + { 0x00000000, NULL, NTB_ATOM, 0 } +}; + +static const struct ntb_reg atom_reg = { + .ntb_ctl = ATOM_NTBCNTL_OFFSET, + .lnk_sta = ATOM_LINK_STATUS_OFFSET, + .db_size = sizeof(uint64_t), + .mw_bar = { NTB_B2B_BAR_1, NTB_B2B_BAR_2 }, +}; + +static const struct ntb_alt_reg atom_pri_reg = { + .db_bell = ATOM_PDOORBELL_OFFSET, *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201709111848.v8BIm9xr075694>