Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 30 Sep 2016 16:24:17 +0800
From:      Sepherosa Ziehau <sepherosa@gmail.com>
To:        Hans Petter Selasky <hselasky@freebsd.org>
Cc:        src-committers@freebsd.org, svn-src-all@freebsd.org,  svn-src-head@freebsd.org
Subject:   Re: svn commit: r306486 - in head: share/man/man4 sys/amd64/conf sys/conf sys/dev/mlx4 sys/dev/mlx4/mlx4_core sys/dev/mlx4/mlx4_en sys/dev/mlx4/mlx4_ib sys/i386/conf sys/modules sys/modules/mlx4 sys/mo...
Message-ID:  <CAMOc5cytXAmC2NnWWnbQt7kL8yf3PqiO_9N3KAxoiihbQPjrvA@mail.gmail.com>
In-Reply-To: <201609300823.u8U8N7ff043558@repo.freebsd.org>
References:  <201609300823.u8U8N7ff043558@repo.freebsd.org>

next in thread | previous in thread | raw e-mail | index | archive | help
Nice! :)

On Fri, Sep 30, 2016 at 4:23 PM, Hans Petter Selasky
<hselasky@freebsd.org> wrote:
> Author: hselasky
> Date: Fri Sep 30 08:23:06 2016
> New Revision: 306486
> URL: https://svnweb.freebsd.org/changeset/base/306486
>
> Log:
>   Move the ConnectX-3 and ConnectX-2 driver from sys/ofed into sys/dev/mlx4
>   like other PCI network drivers. The sys/ofed directory is now mainly
>   reserved for generic infiniband code, with exception of the mthca driver.
>
>   - Add new manual page, mlx4en(4), describing how to configure and load
>   mlx4en.
>
>   - All relevant driver C-files are now prefixed mlx4, mlx4_en and
>   mlx4_ib respectivly to avoid object filename collisions when compiling
>   the kernel. This also fixes an issue with proper dependency file
>   generation for the C-files in question.
>
>   - Device mlxen is now device mlx4en and depends on device mlx4, see
>   mlx4en(4). Only the network device name remains unchanged.
>
>   - The mlx4 and mlx4en modules are now built by default on i386 and
>   amd64 targets. Only building the mlx4ib module depends on
>   WITH_OFED=YES .
>
>   Sponsored by: Mellanox Technologies
>
> Added:
>   head/share/man/man4/mlx4en.4   (contents, props changed)
>   head/sys/dev/mlx4/
>   head/sys/dev/mlx4/cmd.h
>      - copied unchanged from r306485, head/sys/ofed/include/linux/mlx4/cmd.h
>   head/sys/dev/mlx4/cq.h
>      - copied, changed from r306485, head/sys/ofed/include/linux/mlx4/cq.h
>   head/sys/dev/mlx4/device.h
>      - copied unchanged from r306485, head/sys/ofed/include/linux/mlx4/device.h
>   head/sys/dev/mlx4/doorbell.h
>      - copied unchanged from r306485, head/sys/ofed/include/linux/mlx4/doorbell.h
>   head/sys/dev/mlx4/driver.h
>      - copied, changed from r306485, head/sys/ofed/include/linux/mlx4/driver.h
>   head/sys/dev/mlx4/mlx4_core/
>   head/sys/dev/mlx4/mlx4_core/fw.h
>      - copied unchanged from r306485, head/sys/ofed/drivers/net/mlx4/fw.h
>   head/sys/dev/mlx4/mlx4_core/icm.h
>      - copied unchanged from r306485, head/sys/ofed/drivers/net/mlx4/icm.h
>   head/sys/dev/mlx4/mlx4_core/mlx4.h
>      - copied, changed from r306485, head/sys/ofed/drivers/net/mlx4/mlx4.h
>   head/sys/dev/mlx4/mlx4_core/mlx4_alloc.c
>      - copied unchanged from r306485, head/sys/ofed/drivers/net/mlx4/alloc.c
>   head/sys/dev/mlx4/mlx4_core/mlx4_catas.c
>      - copied unchanged from r306485, head/sys/ofed/drivers/net/mlx4/catas.c
>   head/sys/dev/mlx4/mlx4_core/mlx4_cmd.c
>      - copied, changed from r306485, head/sys/ofed/drivers/net/mlx4/cmd.c
>   head/sys/dev/mlx4/mlx4_core/mlx4_cq.c
>      - copied, changed from r306485, head/sys/ofed/drivers/net/mlx4/cq.c
>   head/sys/dev/mlx4/mlx4_core/mlx4_eq.c
>      - copied, changed from r306485, head/sys/ofed/drivers/net/mlx4/eq.c
>   head/sys/dev/mlx4/mlx4_core/mlx4_fw.c
>      - copied, changed from r306485, head/sys/ofed/drivers/net/mlx4/fw.c
>   head/sys/dev/mlx4/mlx4_core/mlx4_icm.c
>      - copied, changed from r306485, head/sys/ofed/drivers/net/mlx4/icm.c
>   head/sys/dev/mlx4/mlx4_core/mlx4_intf.c
>      - copied unchanged from r306485, head/sys/ofed/drivers/net/mlx4/intf.c
>   head/sys/dev/mlx4/mlx4_core/mlx4_main.c
>      - copied, changed from r306485, head/sys/ofed/drivers/net/mlx4/main.c
>   head/sys/dev/mlx4/mlx4_core/mlx4_mcg.c
>      - copied, changed from r306485, head/sys/ofed/drivers/net/mlx4/mcg.c
>   head/sys/dev/mlx4/mlx4_core/mlx4_mr.c
>      - copied, changed from r306485, head/sys/ofed/drivers/net/mlx4/mr.c
>   head/sys/dev/mlx4/mlx4_core/mlx4_pd.c
>      - copied unchanged from r306485, head/sys/ofed/drivers/net/mlx4/pd.c
>   head/sys/dev/mlx4/mlx4_core/mlx4_port.c
>      - copied, changed from r306485, head/sys/ofed/drivers/net/mlx4/port.c
>   head/sys/dev/mlx4/mlx4_core/mlx4_profile.c
>      - copied unchanged from r306485, head/sys/ofed/drivers/net/mlx4/profile.c
>   head/sys/dev/mlx4/mlx4_core/mlx4_qp.c
>      - copied, changed from r306485, head/sys/ofed/drivers/net/mlx4/qp.c
>   head/sys/dev/mlx4/mlx4_core/mlx4_reset.c
>      - copied unchanged from r306485, head/sys/ofed/drivers/net/mlx4/reset.c
>   head/sys/dev/mlx4/mlx4_core/mlx4_resource_tracker.c
>      - copied, changed from r306485, head/sys/ofed/drivers/net/mlx4/resource_tracker.c
>   head/sys/dev/mlx4/mlx4_core/mlx4_sense.c
>      - copied, changed from r306485, head/sys/ofed/drivers/net/mlx4/sense.c
>   head/sys/dev/mlx4/mlx4_core/mlx4_srq.c
>      - copied, changed from r306485, head/sys/ofed/drivers/net/mlx4/srq.c
>   head/sys/dev/mlx4/mlx4_core/mlx4_sys_tune.c
>      - copied unchanged from r306485, head/sys/ofed/drivers/net/mlx4/sys_tune.c
>   head/sys/dev/mlx4/mlx4_en/
>   head/sys/dev/mlx4/mlx4_en/en.h
>      - copied, changed from r306485, head/sys/ofed/drivers/net/mlx4/mlx4_en.h
>   head/sys/dev/mlx4/mlx4_en/en_port.h
>      - copied unchanged from r306485, head/sys/ofed/drivers/net/mlx4/en_port.h
>   head/sys/dev/mlx4/mlx4_en/mlx4_en_cq.c
>      - copied, changed from r306485, head/sys/ofed/drivers/net/mlx4/en_cq.c
>   head/sys/dev/mlx4/mlx4_en/mlx4_en_main.c
>      - copied, changed from r306485, head/sys/ofed/drivers/net/mlx4/en_main.c
>   head/sys/dev/mlx4/mlx4_en/mlx4_en_netdev.c
>      - copied, changed from r306485, head/sys/ofed/drivers/net/mlx4/en_netdev.c
>   head/sys/dev/mlx4/mlx4_en/mlx4_en_port.c
>      - copied, changed from r306485, head/sys/ofed/drivers/net/mlx4/en_port.c
>   head/sys/dev/mlx4/mlx4_en/mlx4_en_resources.c
>      - copied, changed from r306485, head/sys/ofed/drivers/net/mlx4/en_resources.c
>   head/sys/dev/mlx4/mlx4_en/mlx4_en_rx.c
>      - copied, changed from r306485, head/sys/ofed/drivers/net/mlx4/en_rx.c
>   head/sys/dev/mlx4/mlx4_en/mlx4_en_tx.c
>      - copied, changed from r306485, head/sys/ofed/drivers/net/mlx4/en_tx.c
>   head/sys/dev/mlx4/mlx4_ib/
>   head/sys/dev/mlx4/mlx4_ib/mlx4_exp.h
>      - copied unchanged from r306485, head/sys/ofed/drivers/infiniband/hw/mlx4/mlx4_exp.h
>   head/sys/dev/mlx4/mlx4_ib/mlx4_ib.h
>      - copied, changed from r306485, head/sys/ofed/drivers/infiniband/hw/mlx4/mlx4_ib.h
>   head/sys/dev/mlx4/mlx4_ib/mlx4_ib_ah.c
>      - copied unchanged from r306485, head/sys/ofed/drivers/infiniband/hw/mlx4/ah.c
>   head/sys/dev/mlx4/mlx4_ib/mlx4_ib_alias_GUID.c
>      - copied, changed from r306485, head/sys/ofed/drivers/infiniband/hw/mlx4/alias_GUID.c
>   head/sys/dev/mlx4/mlx4_ib/mlx4_ib_cm.c
>      - copied, changed from r306485, head/sys/ofed/drivers/infiniband/hw/mlx4/cm.c
>   head/sys/dev/mlx4/mlx4_ib/mlx4_ib_cq.c
>      - copied, changed from r306485, head/sys/ofed/drivers/infiniband/hw/mlx4/cq.c
>   head/sys/dev/mlx4/mlx4_ib/mlx4_ib_doorbell.c
>      - copied unchanged from r306485, head/sys/ofed/drivers/infiniband/hw/mlx4/doorbell.c
>   head/sys/dev/mlx4/mlx4_ib/mlx4_ib_exp.c
>      - copied, changed from r306485, head/sys/ofed/drivers/infiniband/hw/mlx4/mlx4_exp.c
>   head/sys/dev/mlx4/mlx4_ib/mlx4_ib_mad.c
>      - copied, changed from r306485, head/sys/ofed/drivers/infiniband/hw/mlx4/mad.c
>   head/sys/dev/mlx4/mlx4_ib/mlx4_ib_main.c
>      - copied, changed from r306485, head/sys/ofed/drivers/infiniband/hw/mlx4/main.c
>   head/sys/dev/mlx4/mlx4_ib/mlx4_ib_mcg.c
>      - copied, changed from r306485, head/sys/ofed/drivers/infiniband/hw/mlx4/mcg.c
>   head/sys/dev/mlx4/mlx4_ib/mlx4_ib_mr.c
>      - copied unchanged from r306485, head/sys/ofed/drivers/infiniband/hw/mlx4/mr.c
>   head/sys/dev/mlx4/mlx4_ib/mlx4_ib_qp.c
>      - copied, changed from r306485, head/sys/ofed/drivers/infiniband/hw/mlx4/qp.c
>   head/sys/dev/mlx4/mlx4_ib/mlx4_ib_srq.c
>      - copied, changed from r306485, head/sys/ofed/drivers/infiniband/hw/mlx4/srq.c
>   head/sys/dev/mlx4/mlx4_ib/mlx4_ib_sysfs.c
>      - copied unchanged from r306485, head/sys/ofed/drivers/infiniband/hw/mlx4/sysfs.c
>   head/sys/dev/mlx4/mlx4_ib/mlx4_ib_wc.c
>      - copied unchanged from r306485, head/sys/ofed/drivers/infiniband/hw/mlx4/wc.c
>   head/sys/dev/mlx4/mlx4_ib/user.h
>      - copied unchanged from r306485, head/sys/ofed/drivers/infiniband/hw/mlx4/user.h
>   head/sys/dev/mlx4/mlx4_ib/wc.h
>      - copied unchanged from r306485, head/sys/ofed/drivers/infiniband/hw/mlx4/wc.h
>   head/sys/dev/mlx4/qp.h
>      - copied, changed from r306485, head/sys/ofed/include/linux/mlx4/qp.h
>   head/sys/dev/mlx4/srq.h
>      - copied unchanged from r306485, head/sys/ofed/include/linux/mlx4/srq.h
>   head/sys/dev/mlx4/stats.h
>      - copied unchanged from r306485, head/sys/ofed/drivers/net/mlx4/mlx4_stats.h
>   head/sys/modules/mlx4en/
>      - copied from r306485, head/sys/modules/mlxen/
> Deleted:
>   head/sys/modules/mlxen/
>   head/sys/ofed/drivers/infiniband/hw/mlx4/
>   head/sys/ofed/drivers/net/
>   head/sys/ofed/include/linux/
> Modified:
>   head/share/man/man4/Makefile
>   head/sys/amd64/conf/NOTES
>   head/sys/conf/files
>   head/sys/i386/conf/NOTES
>   head/sys/modules/Makefile
>   head/sys/modules/mlx4/Makefile
>   head/sys/modules/mlx4en/Makefile
>   head/sys/modules/mlx4ib/Makefile
>
> Modified: head/share/man/man4/Makefile
> ==============================================================================
> --- head/share/man/man4/Makefile        Fri Sep 30 06:58:45 2016        (r306485)
> +++ head/share/man/man4/Makefile        Fri Sep 30 08:23:06 2016        (r306486)
> @@ -279,6 +279,7 @@ MAN=        aac.4 \
>         mk48txx.4 \
>         mld.4 \
>         mlx.4 \
> +       mlx4en.4 \
>         mlx5en.4 \
>         mly.4 \
>         mmc.4 \
>
> Added: head/share/man/man4/mlx4en.4
> ==============================================================================
> --- /dev/null   00:00:00 1970   (empty, because file is newly added)
> +++ head/share/man/man4/mlx4en.4        Fri Sep 30 08:23:06 2016        (r306486)
> @@ -0,0 +1,95 @@
> +.\" Copyright (c) 2016 Mellanox Technologies
> +.\" All rights reserved.
> +.\"
> +.\" Redistribution and use in source and binary forms, with or without
> +.\" modification, are permitted provided that the following conditions
> +.\" are met:
> +.\" 1. Redistributions of source code must retain the above copyright
> +.\"    notice, this list of conditions and the following disclaimer.
> +.\" 2. Redistributions in binary form must reproduce the above copyright
> +.\"    notice, this list of conditions and the following disclaimer in the
> +.\"    documentation and/or other materials provided with the distribution.
> +.\"
> +.\" THIS SOFTWARE IS PROVIDED BY AUTHOR AND CONTRIBUTORS `AS IS' AND
> +.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
> +.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
> +.\" ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
> +.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
> +.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
> +.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
> +.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
> +.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> +.\" SUCH DAMAGE.
> +.\"
> +.\" $FreeBSD$
> +.\"
> +.Dd September 30, 2016
> +.Dt MLX4EN 4
> +.Os
> +.Sh NAME
> +.Nm mlx4en
> +.Nd "Mellanox ConnectX-3 10GbE/40GbE network adapter driver"
> +.Sh SYNOPSIS
> +To compile this driver into the kernel,
> +place the following lines in your
> +kernel configuration file:
> +.Bd -ragged -offset indent
> +.Cd "options COMPAT_LINUXKPI"
> +.Cd "device mlx4"
> +.Cd "device mlx4en"
> +.Ed
> +.Pp
> +To load the driver as a module at run-time,
> +run the following command as root:
> +.Bd -literal -offset indent
> +kldload mlx4en
> +.Ed
> +.Pp
> +To load the driver as a
> +module at boot time, place the following lines in
> +.Xr loader.conf 5 :
> +.Bd -literal -offset indent
> +mlx4en_load="YES"
> +.Ed
> +.Sh DESCRIPTION
> +Mellanox ConnectX adapter cards with Virtual Protocol Interconnect
> +(VPI) provide the highest performing and most flexible interconnect
> +solution for Enterprise Data Centers, High-Performance Computing, and
> +Embedded environments.
> +Clustered data bases, parallelized applications, transactional
> +services and high-performance embedded I/O applications will achieve
> +significant performance improvements resulting in reduced completion
> +time and lower cost per operation.
> +.Sh HARDWARE
> +The
> +.Nm
> +driver supports the following network adapters:
> +.Pp
> +.Bl -bullet -compact
> +.It
> +Mellanox ConnectX-2 (ETH)
> +.It
> +Mellanox ConnectX-3 (ETH)
> +.El
> +.Sh SUPPORT
> +For general information and support,
> +go to the Mellanox support website at:
> +.Pa http://www.mellanox.com/ .
> +.Pp
> +If an issue is identified with this driver with a supported adapter,
> +email all the specific information related to the issue to
> +.Aq Mt freebsd-drivers@mellanox.com .
> +.Sh SEE ALSO
> +.Xr ifconfig 8
> +.Sh HISTORY
> +The
> +.Nm
> +device driver first appeared in
> +.Fx 9.x .
> +.Sh AUTHORS
> +.An -nosplit
> +The
> +.Nm
> +driver was written by
> +.An Mellanox Technologies <freebsd-drivers@mellanox.com> .
>
> Modified: head/sys/amd64/conf/NOTES
> ==============================================================================
> --- head/sys/amd64/conf/NOTES   Fri Sep 30 06:58:45 2016        (r306485)
> +++ head/sys/amd64/conf/NOTES   Fri Sep 30 08:23:06 2016        (r306486)
> @@ -318,7 +318,7 @@ options     DRM_DEBUG       # Include debug print
>  # ixl: Intel XL710 40Gbe PCIE Ethernet
>  # ixlv:        Intel XL710 40Gbe VF PCIE Ethernet
>  # mlx4ib: Mellanox ConnectX HCA InfiniBand
> -# mlxen: Mellanox ConnectX HCA Ethernet
> +# mlx4en: Mellanox ConnectX HCA Ethernet
>  # mthca: Mellanox HCA InfiniBand
>  # nfe: nVidia nForce MCP on-board Ethernet Networking (BSD open source)
>  # sfxge: Solarflare SFC9000 family 10Gb Ethernet adapters
> @@ -336,8 +336,9 @@ device              iwi             # Intel 2200BG/2225BG/2915A
>  device         iwn             # Intel 4965/1000/5000/6000 wireless NICs.
>  device         ixl             # Intel XL710 40Gbe PCIE Ethernet
>  device         ixlv            # Intel XL710 40Gbe VF PCIE Ethernet
> +device         mlx4            # Shared code module between IB and Ethernet
>  device         mlx4ib          # Mellanox ConnectX HCA InfiniBand
> -device         mlxen           # Mellanox ConnectX HCA Ethernet
> +device         mlx4en          # Mellanox ConnectX HCA Ethernet
>  device         mthca           # Mellanox HCA InfiniBand
>  device         nfe             # nVidia nForce MCP on-board Ethernet
>  device         sfxge           # Solarflare SFC9000 10Gb Ethernet
>
> Modified: head/sys/conf/files
> ==============================================================================
> --- head/sys/conf/files Fri Sep 30 06:58:45 2016        (r306485)
> +++ head/sys/conf/files Fri Sep 30 08:23:06 2016        (r306486)
> @@ -4036,131 +4036,90 @@ ofed/drivers/infiniband/ulp/sdp/sdp_cma.
>  ofed/drivers/infiniband/ulp/sdp/sdp_tx.c       optional sdp inet       \
>         compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/ulp/sdp/"
>
> -ofed/drivers/infiniband/hw/mlx4/alias_GUID.c    optional mlx4ib         \
> -        no-depend obj-prefix "mlx4ib_"                                  \
> -        compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/"
> -ofed/drivers/infiniband/hw/mlx4/mcg.c           optional mlx4ib         \
> -        no-depend obj-prefix "mlx4ib_"                                  \
> -        compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/"
> -ofed/drivers/infiniband/hw/mlx4/sysfs.c         optional mlx4ib         \
> -        no-depend obj-prefix "mlx4ib_"                                  \
> -        compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/"
> -ofed/drivers/infiniband/hw/mlx4/cm.c            optional mlx4ib         \
> -        no-depend obj-prefix "mlx4ib_"                                  \
> -        compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/"
> -ofed/drivers/infiniband/hw/mlx4/ah.c           optional mlx4ib         \
> -       no-depend obj-prefix "mlx4ib_"                                  \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/"
> -ofed/drivers/infiniband/hw/mlx4/cq.c           optional mlx4ib         \
> -       no-depend obj-prefix "mlx4ib_"                                  \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/"
> -ofed/drivers/infiniband/hw/mlx4/doorbell.c     optional mlx4ib         \
> -       no-depend obj-prefix "mlx4ib_"                                  \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/"
> -ofed/drivers/infiniband/hw/mlx4/mad.c          optional mlx4ib         \
> -       no-depend obj-prefix "mlx4ib_"                                  \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/"
> -ofed/drivers/infiniband/hw/mlx4/main.c         optional mlx4ib         \
> -       no-depend obj-prefix "mlx4ib_"                                  \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/"
> -ofed/drivers/infiniband/hw/mlx4/mlx4_exp.c     optional mlx4ib         \
> -       no-depend obj-prefix "mlx4ib_"                                  \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/"
> -ofed/drivers/infiniband/hw/mlx4/mr.c           optional mlx4ib         \
> -       no-depend obj-prefix "mlx4ib_"                                  \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/"
> -ofed/drivers/infiniband/hw/mlx4/qp.c           optional mlx4ib         \
> -       no-depend obj-prefix "mlx4ib_"                                  \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/"
> -ofed/drivers/infiniband/hw/mlx4/srq.c          optional mlx4ib         \
> -       no-depend obj-prefix "mlx4ib_"                                  \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/"
> -ofed/drivers/infiniband/hw/mlx4/wc.c           optional mlx4ib         \
> -       no-depend obj-prefix "mlx4ib_"                                  \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/"
> +dev/mlx4/mlx4_ib/mlx4_ib_alias_GUID.c          optional mlx4ib pci ofed \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_ib/mlx4_ib_mcg.c                 optional mlx4ib pci ofed \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_ib/mlx4_ib_sysfs.c               optional mlx4ib pci ofed \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_ib/mlx4_ib_cm.c                  optional mlx4ib pci ofed \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_ib/mlx4_ib_ah.c                  optional mlx4ib pci ofed \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_ib/mlx4_ib_cq.c                  optional mlx4ib pci ofed \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_ib/mlx4_ib_doorbell.c            optional mlx4ib pci ofed \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_ib/mlx4_ib_mad.c                 optional mlx4ib pci ofed \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_ib/mlx4_ib_main.c                        optional mlx4ib pci ofed \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_ib/mlx4_ib_exp.c                 optional mlx4ib pci ofed \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_ib/mlx4_ib_mr.c                  optional mlx4ib pci ofed \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_ib/mlx4_ib_qp.c                  optional mlx4ib pci ofed \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_ib/mlx4_ib_srq.c                 optional mlx4ib pci ofed \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_ib/mlx4_ib_wc.c                  optional mlx4ib pci ofed \
> +       compile-with "${OFED_C}"
>
> -ofed/drivers/net/mlx4/alloc.c                  optional mlx4ib | mlxen \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/catas.c                  optional mlx4ib | mlxen \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/cmd.c                    optional mlx4ib | mlxen \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/cq.c                     optional mlx4ib | mlxen \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/eq.c                     optional mlx4ib | mlxen \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/fw.c                     optional mlx4ib | mlxen \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/icm.c                    optional mlx4ib | mlxen \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/intf.c                   optional mlx4ib | mlxen \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/main.c                   optional mlx4ib | mlxen \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/mcg.c                    optional mlx4ib | mlxen \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/ -Wno-unused"
> -ofed/drivers/net/mlx4/mr.c                     optional mlx4ib | mlxen \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/pd.c                     optional mlx4ib | mlxen \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/port.c                   optional mlx4ib | mlxen \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/profile.c                        optional mlx4ib | mlxen \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/qp.c                     optional mlx4ib | mlxen \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/reset.c                  optional mlx4ib | mlxen \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/sense.c                  optional mlx4ib | mlxen \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/srq.c                    optional mlx4ib | mlxen \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/resource_tracker.c        optional mlx4ib | mlxen        \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/sys_tune.c               optional mlx4ib | mlxen \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> +dev/mlx4/mlx4_core/mlx4_alloc.c                        optional mlx4 pci \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_core/mlx4_catas.c                        optional mlx4 pci \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_core/mlx4_cmd.c                  optional mlx4 pci \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_core/mlx4_cq.c                   optional mlx4 pci \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_core/mlx4_eq.c                   optional mlx4 pci \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_core/mlx4_fw.c                   optional mlx4 pci \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_core/mlx4_icm.c                  optional mlx4 pci \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_core/mlx4_intf.c                 optional mlx4 pci \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_core/mlx4_main.c                 optional mlx4 pci \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_core/mlx4_mcg.c                  optional mlx4 pci \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_core/mlx4_mr.c                   optional mlx4 pci \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_core/mlx4_pd.c                   optional mlx4 pci \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_core/mlx4_port.c                 optional mlx4 pci \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_core/mlx4_profile.c              optional mlx4 pci \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_core/mlx4_qp.c                   optional mlx4 pci \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_core/mlx4_reset.c                        optional mlx4 pci \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_core/mlx4_sense.c                        optional mlx4 pci \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_core/mlx4_srq.c                  optional mlx4 pci \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_core/mlx4_resource_tracker.c     optional mlx4 pci \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_core/mlx4_sys_tune.c             optional mlx4 pci \
> +       compile-with "${OFED_C}"
>
> -ofed/drivers/net/mlx4/en_cq.c                  optional mlxen          \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/en_main.c                        optional mlxen          \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/en_netdev.c              optional mlxen          \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/en_port.c                        optional mlxen          \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/en_resources.c           optional mlxen          \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/en_rx.c                  optional mlxen          \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> -ofed/drivers/net/mlx4/en_tx.c                  optional mlxen          \
> -       no-depend obj-prefix "mlx4_"                                    \
> -       compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/"
> +dev/mlx4/mlx4_en/mlx4_en_cq.c                  optional mlx4en pci inet inet6  \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_en/mlx4_en_main.c                        optional mlx4en pci inet inet6  \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_en/mlx4_en_netdev.c              optional mlx4en pci inet inet6  \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_en/mlx4_en_port.c                        optional mlx4en pci inet inet6  \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_en/mlx4_en_resources.c           optional mlx4en pci inet inet6  \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_en/mlx4_en_rx.c                  optional mlx4en pci inet inet6  \
> +       compile-with "${OFED_C}"
> +dev/mlx4/mlx4_en/mlx4_en_tx.c                  optional mlx4en pci inet inet6  \
> +       compile-with "${OFED_C}"
>
>  dev/mlx5/mlx5_core/mlx5_alloc.c                        optional mlx5 pci       \
>         compile-with "${OFED_C}"
>
> Copied: head/sys/dev/mlx4/cmd.h (from r306485, head/sys/ofed/include/linux/mlx4/cmd.h)
> ==============================================================================
> --- /dev/null   00:00:00 1970   (empty, because file is newly added)
> +++ head/sys/dev/mlx4/cmd.h     Fri Sep 30 08:23:06 2016        (r306486, copy of r306485, head/sys/ofed/include/linux/mlx4/cmd.h)
> @@ -0,0 +1,258 @@
> +/*
> + * Copyright (c) 2006 Cisco Systems, Inc.  All rights reserved.
> + *
> + * This software is available to you under a choice of one of two
> + * licenses.  You may choose to be licensed under the terms of the GNU
> + * General Public License (GPL) Version 2, available from the file
> + * COPYING in the main directory of this source tree, or the
> + * OpenIB.org BSD license below:
> + *
> + *     Redistribution and use in source and binary forms, with or
> + *     without modification, are permitted provided that the following
> + *     conditions are met:
> + *
> + *      - Redistributions of source code must retain the above
> + *        copyright notice, this list of conditions and the following
> + *        disclaimer.
> + *
> + *      - Redistributions in binary form must reproduce the above
> + *        copyright notice, this list of conditions and the following
> + *        disclaimer in the documentation and/or other materials
> + *        provided with the distribution.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
> + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
> + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
> + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
> + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
> + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
> + * SOFTWARE.
> + */
> +
> +#ifndef MLX4_CMD_H
> +#define MLX4_CMD_H
> +
> +#include <linux/dma-mapping.h>
> +#include <linux/types.h>
> +
> +enum {
> +       /* initialization and general commands */
> +       MLX4_CMD_SYS_EN          = 0x1,
> +       MLX4_CMD_SYS_DIS         = 0x2,
> +       MLX4_CMD_MAP_FA          = 0xfff,
> +       MLX4_CMD_UNMAP_FA        = 0xffe,
> +       MLX4_CMD_RUN_FW          = 0xff6,
> +       MLX4_CMD_MOD_STAT_CFG    = 0x34,
> +       MLX4_CMD_QUERY_DEV_CAP   = 0x3,
> +       MLX4_CMD_QUERY_FW        = 0x4,
> +       MLX4_CMD_ENABLE_LAM      = 0xff8,
> +       MLX4_CMD_DISABLE_LAM     = 0xff7,
> +       MLX4_CMD_QUERY_DDR       = 0x5,
> +       MLX4_CMD_QUERY_ADAPTER   = 0x6,
> +       MLX4_CMD_INIT_HCA        = 0x7,
> +       MLX4_CMD_CLOSE_HCA       = 0x8,
> +       MLX4_CMD_INIT_PORT       = 0x9,
> +       MLX4_CMD_CLOSE_PORT      = 0xa,
> +       MLX4_CMD_QUERY_HCA       = 0xb,
> +       MLX4_CMD_QUERY_PORT      = 0x43,
> +       MLX4_CMD_SENSE_PORT      = 0x4d,
> +       MLX4_CMD_HW_HEALTH_CHECK = 0x50,
> +       MLX4_CMD_SET_PORT        = 0xc,
> +       MLX4_CMD_SET_NODE        = 0x5a,
> +       MLX4_CMD_QUERY_FUNC      = 0x56,
> +       MLX4_CMD_ACCESS_DDR      = 0x2e,
> +       MLX4_CMD_MAP_ICM         = 0xffa,
> +       MLX4_CMD_UNMAP_ICM       = 0xff9,
> +       MLX4_CMD_MAP_ICM_AUX     = 0xffc,
> +       MLX4_CMD_UNMAP_ICM_AUX   = 0xffb,
> +       MLX4_CMD_SET_ICM_SIZE    = 0xffd,
> +       /*master notify fw on finish for slave's flr*/
> +       MLX4_CMD_INFORM_FLR_DONE = 0x5b,
> +       MLX4_CMD_GET_OP_REQ      = 0x59,
> +
> +       /* TPT commands */
> +       MLX4_CMD_SW2HW_MPT       = 0xd,
> +       MLX4_CMD_QUERY_MPT       = 0xe,
> +       MLX4_CMD_HW2SW_MPT       = 0xf,
> +       MLX4_CMD_READ_MTT        = 0x10,
> +       MLX4_CMD_WRITE_MTT       = 0x11,
> +       MLX4_CMD_SYNC_TPT        = 0x2f,
> +
> +       /* EQ commands */
> +       MLX4_CMD_MAP_EQ          = 0x12,
> +       MLX4_CMD_SW2HW_EQ        = 0x13,
> +       MLX4_CMD_HW2SW_EQ        = 0x14,
> +       MLX4_CMD_QUERY_EQ        = 0x15,
> +
> +       /* CQ commands */
> +       MLX4_CMD_SW2HW_CQ        = 0x16,
> +       MLX4_CMD_HW2SW_CQ        = 0x17,
> +       MLX4_CMD_QUERY_CQ        = 0x18,
> +       MLX4_CMD_MODIFY_CQ       = 0x2c,
> +
> +       /* SRQ commands */
> +       MLX4_CMD_SW2HW_SRQ       = 0x35,
> +       MLX4_CMD_HW2SW_SRQ       = 0x36,
> +       MLX4_CMD_QUERY_SRQ       = 0x37,
> +       MLX4_CMD_ARM_SRQ         = 0x40,
> +
> +       /* QP/EE commands */
> +       MLX4_CMD_RST2INIT_QP     = 0x19,
> +       MLX4_CMD_INIT2RTR_QP     = 0x1a,
> +       MLX4_CMD_RTR2RTS_QP      = 0x1b,
> +       MLX4_CMD_RTS2RTS_QP      = 0x1c,
> +       MLX4_CMD_SQERR2RTS_QP    = 0x1d,
> +       MLX4_CMD_2ERR_QP         = 0x1e,
> +       MLX4_CMD_RTS2SQD_QP      = 0x1f,
> +       MLX4_CMD_SQD2SQD_QP      = 0x38,
> +       MLX4_CMD_SQD2RTS_QP      = 0x20,
> +       MLX4_CMD_2RST_QP         = 0x21,
> +       MLX4_CMD_QUERY_QP        = 0x22,
> +       MLX4_CMD_INIT2INIT_QP    = 0x2d,
> +       MLX4_CMD_SUSPEND_QP      = 0x32,
> +       MLX4_CMD_UNSUSPEND_QP    = 0x33,
> +       MLX4_CMD_UPDATE_QP       = 0x61,
> +       /* special QP and management commands */
> +       MLX4_CMD_CONF_SPECIAL_QP = 0x23,
> +       MLX4_CMD_MAD_IFC         = 0x24,
> +
> +       /* multicast commands */
> +       MLX4_CMD_READ_MCG        = 0x25,
> +       MLX4_CMD_WRITE_MCG       = 0x26,
> +       MLX4_CMD_MGID_HASH       = 0x27,
> +
> +       /* miscellaneous commands */
> +       MLX4_CMD_DIAG_RPRT       = 0x30,
> +       MLX4_CMD_NOP             = 0x31,
> +       MLX4_CMD_ACCESS_MEM      = 0x2e,
> +       MLX4_CMD_SET_VEP         = 0x52,
> +
> +       /* Ethernet specific commands */
> +       MLX4_CMD_SET_VLAN_FLTR   = 0x47,
> +       MLX4_CMD_SET_MCAST_FLTR  = 0x48,
> +       MLX4_CMD_DUMP_ETH_STATS  = 0x49,
> +
> +       /* Communication channel commands */
> +       MLX4_CMD_ARM_COMM_CHANNEL = 0x57,
> +       MLX4_CMD_GEN_EQE         = 0x58,
> +
> +       /* virtual commands */
> +       MLX4_CMD_ALLOC_RES       = 0xf00,
> +       MLX4_CMD_FREE_RES        = 0xf01,
> +       MLX4_CMD_MCAST_ATTACH    = 0xf05,
> +       MLX4_CMD_UCAST_ATTACH    = 0xf06,
> +       MLX4_CMD_PROMISC         = 0xf08,
> +       MLX4_CMD_QUERY_FUNC_CAP  = 0xf0a,
> +       MLX4_CMD_QP_ATTACH       = 0xf0b,
> +
> +       /* debug commands */
> +       MLX4_CMD_QUERY_DEBUG_MSG = 0x2a,
> +       MLX4_CMD_SET_DEBUG_MSG   = 0x2b,
> +
> +       /* statistics commands */
> +       MLX4_CMD_QUERY_IF_STAT   = 0X54,
> +       MLX4_CMD_SET_IF_STAT     = 0X55,
> +
> +       /* register/delete flow steering network rules */
> +       MLX4_QP_FLOW_STEERING_ATTACH = 0x65,
> +       MLX4_QP_FLOW_STEERING_DETACH = 0x66,
> +       MLX4_FLOW_STEERING_IB_UC_QP_RANGE = 0x64,
> +};
> +
> +enum {
> +       MLX4_CMD_TIME_CLASS_A   = 60000,
> +       MLX4_CMD_TIME_CLASS_B   = 60000,
> +       MLX4_CMD_TIME_CLASS_C   = 60000,
> +};
> +
> +enum {
> +       MLX4_MAILBOX_SIZE       = 4096,
> +       MLX4_ACCESS_MEM_ALIGN   = 256,
> +};
> +
> +enum {
> +       /* set port opcode modifiers */
> +       MLX4_SET_PORT_GENERAL           = 0x0,
> +       MLX4_SET_PORT_RQP_CALC          = 0x1,
> +       MLX4_SET_PORT_MAC_TABLE         = 0x2,
> +       MLX4_SET_PORT_VLAN_TABLE        = 0x3,
> +       MLX4_SET_PORT_PRIO_MAP          = 0x4,
> +       MLX4_SET_PORT_GID_TABLE         = 0x5,
> +       MLX4_SET_PORT_PRIO2TC           = 0x8,
> +       MLX4_SET_PORT_SCHEDULER         = 0x9
> +};
> +
> +enum {
> +       MLX4_CMD_WRAPPED,
> +       MLX4_CMD_NATIVE
> +};
> +
> +struct mlx4_dev;
> +
> +struct mlx4_cmd_mailbox {
> +       void                   *buf;
> +       dma_addr_t              dma;
> +};
> +
> +int __mlx4_cmd(struct mlx4_dev *dev, u64 in_param, u64 *out_param,
> +              int out_is_imm, u32 in_modifier, u8 op_modifier,
> +              u16 op, unsigned long timeout, int native);
> +
> +/* Invoke a command with no output parameter */
> +static inline int mlx4_cmd(struct mlx4_dev *dev, u64 in_param, u32 in_modifier,
> +                          u8 op_modifier, u16 op, unsigned long timeout,
> +                          int native)
> +{
> +       return __mlx4_cmd(dev, in_param, NULL, 0, in_modifier,
> +                         op_modifier, op, timeout, native);
> +}
> +
> +/* Invoke a command with an output mailbox */
> +static inline int mlx4_cmd_box(struct mlx4_dev *dev, u64 in_param, u64 out_param,
> +                              u32 in_modifier, u8 op_modifier, u16 op,
> +                              unsigned long timeout, int native)
> +{
> +       return __mlx4_cmd(dev, in_param, &out_param, 0, in_modifier,
> +                         op_modifier, op, timeout, native);
> +}
> +
> +/*
> + * Invoke a command with an immediate output parameter (and copy the
> + * output into the caller's out_param pointer after the command
> + * executes).
> + */
> +static inline int mlx4_cmd_imm(struct mlx4_dev *dev, u64 in_param, u64 *out_param,
> +                              u32 in_modifier, u8 op_modifier, u16 op,
> +                              unsigned long timeout, int native)
> +{
> +       return __mlx4_cmd(dev, in_param, out_param, 1, in_modifier,
> +                         op_modifier, op, timeout, native);
> +}
> +
> +struct mlx4_cmd_mailbox *mlx4_alloc_cmd_mailbox(struct mlx4_dev *dev);
> +void mlx4_free_cmd_mailbox(struct mlx4_dev *dev, struct mlx4_cmd_mailbox *mailbox);
> +
> +u32 mlx4_comm_get_version(void);
> +int mlx4_set_vf_mac(struct mlx4_dev *dev, int port, int vf, u8 *mac);
> +int mlx4_set_vf_vlan(struct mlx4_dev *dev, int port, int vf, u16 vlan, u8 qos);
> +int mlx4_set_vf_spoofchk(struct mlx4_dev *dev, int port, int vf, bool setting);
> +int mlx4_set_vf_link_state(struct mlx4_dev *dev, int port, int vf, int link_state);
> +int mlx4_get_vf_link_state(struct mlx4_dev *dev, int port, int vf);
> +/*
> + * mlx4_get_slave_default_vlan -
> + * retrun true if VST ( default vlan)
> + * if VST will fill vlan & qos (if not NULL)
> + */
> +bool mlx4_get_slave_default_vlan(struct mlx4_dev *dev, int port, int slave, u16 *vlan, u8 *qos);
> +
> +enum {
> +       IFLA_VF_LINK_STATE_AUTO,        /* link state of the uplink */
> +       IFLA_VF_LINK_STATE_ENABLE,      /* link always up */
> +       IFLA_VF_LINK_STATE_DISABLE,     /* link always down */
> +       __IFLA_VF_LINK_STATE_MAX,
> +};
> +
> +#define MLX4_COMM_GET_IF_REV(cmd_chan_ver) (u8)((cmd_chan_ver) >> 8)
> +
> +#endif /* MLX4_CMD_H */
>
> Copied and modified: head/sys/dev/mlx4/cq.h (from r306485, head/sys/ofed/include/linux/mlx4/cq.h)
> ==============================================================================
> --- head/sys/ofed/include/linux/mlx4/cq.h       Fri Sep 30 06:58:45 2016        (r306485, copy source)
> +++ head/sys/dev/mlx4/cq.h      Fri Sep 30 08:23:06 2016        (r306486)
> @@ -35,8 +35,8 @@
>
>  #include <linux/types.h>
>
> -#include <linux/mlx4/device.h>
> -#include <linux/mlx4/doorbell.h>
> +#include <dev/mlx4/device.h>
> +#include <dev/mlx4/doorbell.h>
>
>  struct mlx4_cqe {
>         __be32                  vlan_my_qpn;
>
> Copied: head/sys/dev/mlx4/device.h (from r306485, head/sys/ofed/include/linux/mlx4/device.h)
> ==============================================================================
> --- /dev/null   00:00:00 1970   (empty, because file is newly added)
> +++ head/sys/dev/mlx4/device.h  Fri Sep 30 08:23:06 2016        (r306486, copy of r306485, head/sys/ofed/include/linux/mlx4/device.h)
> @@ -0,0 +1,1327 @@
> +/*
> + * Copyright (c) 2006, 2007 Cisco Systems, Inc.  All rights reserved.
> + *
> + * This software is available to you under a choice of one of two
> + * licenses.  You may choose to be licensed under the terms of the GNU
> + * General Public License (GPL) Version 2, available from the file
> + * COPYING in the main directory of this source tree, or the
> + * OpenIB.org BSD license below:
> + *
> + *     Redistribution and use in source and binary forms, with or
> + *     without modification, are permitted provided that the following
> + *     conditions are met:
> + *
> + *     - Redistributions of source code must retain the above
> + *       copyright notice, this list of conditions and the following
> + *       disclaimer.
> + *
> + *     - Redistributions in binary form must reproduce the above
> + *       copyright notice, this list of conditions and the following
> + *       disclaimer in the documentation and/or other materials
> + *       provided with the distribution.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
> + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
> + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
> + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
> + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
> + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
> + * SOFTWARE.
> + */
> +
> +#ifndef MLX4_DEVICE_H
> +#define MLX4_DEVICE_H
> +
> +#include <linux/pci.h>
> +#include <linux/completion.h>
> +#include <linux/radix-tree.h>
> +#include <linux/types.h>
> +#include <linux/bitops.h>
> +#include <linux/workqueue.h>
> +#include <asm/atomic.h>
> +
> +#include <linux/clocksource.h>
> +
> +#define MAX_MSIX_P_PORT                17
> +#define MAX_MSIX               64
> +#define MSIX_LEGACY_SZ         4
> +#define MIN_MSIX_P_PORT                5
> +
> +#define MLX4_ROCE_MAX_GIDS     128
> +#define MLX4_ROCE_PF_GIDS      16
> +
> +#define MLX4_NUM_UP                    8
> +#define MLX4_NUM_TC                    8
> +#define MLX4_MAX_100M_UNITS_VAL                255     /*
> +                                                * work around: can't set values
> +                                                * greater then this value when
> +                                                * using 100 Mbps units.
> +                                                */
> +#define MLX4_RATELIMIT_100M_UNITS      3       /* 100 Mbps */
> +#define MLX4_RATELIMIT_1G_UNITS                4       /* 1 Gbps */
> +#define MLX4_RATELIMIT_DEFAULT         0x00ff
> +
> +#define CORE_CLOCK_MASK 0xffffffffffffULL
> +
> +enum {
> +       MLX4_FLAG_MSI_X         = 1 << 0,
> +       MLX4_FLAG_OLD_PORT_CMDS = 1 << 1,
> +       MLX4_FLAG_MASTER        = 1 << 2,
> +       MLX4_FLAG_SLAVE         = 1 << 3,
> +       MLX4_FLAG_SRIOV         = 1 << 4,
> +       MLX4_FLAG_DEV_NUM_STR   = 1 << 5,
> +       MLX4_FLAG_OLD_REG_MAC   = 1 << 6,
> +};
> +
> +enum {
> +       MLX4_PORT_CAP_IS_SM     = 1 << 1,
> +       MLX4_PORT_CAP_DEV_MGMT_SUP = 1 << 19,
> +};
> +
> +enum {
> +       MLX4_MAX_PORTS          = 2,
> +       MLX4_MAX_PORT_PKEYS     = 128
> +};
> +
> +/* base qkey for use in sriov tunnel-qp/proxy-qp communication.
> + * These qkeys must not be allowed for general use. This is a 64k range,
> + * and to test for violation, we use the mask (protect against future chg).
> + */
> +#define MLX4_RESERVED_QKEY_BASE  (0xFFFF0000)
> +#define MLX4_RESERVED_QKEY_MASK  (0xFFFF0000)
> +
> +enum {
> +       MLX4_BOARD_ID_LEN = 64,
> +       MLX4_VSD_LEN = 208
> +};
> +
> +enum {
> +       MLX4_MAX_NUM_PF         = 16,
> +       MLX4_MAX_NUM_VF         = 64,
> +       MLX4_MFUNC_MAX          = 80,
> +       MLX4_MAX_EQ_NUM         = 1024,
> +       MLX4_MFUNC_EQ_NUM       = 4,
> +       MLX4_MFUNC_MAX_EQES     = 8,
> +       MLX4_MFUNC_EQE_MASK     = (MLX4_MFUNC_MAX_EQES - 1)
> +};
> +
> +/* Driver supports 3 different device methods to manage traffic steering:
> + *     -device managed - High level API for ib and eth flow steering. FW is
> + *                       managing flow steering tables.
> + *     - B0 steering mode - Common low level API for ib and (if supported) eth.
> + *     - A0 steering mode - Limited low level API for eth. In case of IB,
> + *                          B0 mode is in use.
> + */
> +enum {
> +       MLX4_STEERING_MODE_A0,
> +       MLX4_STEERING_MODE_B0,
> +       MLX4_STEERING_MODE_DEVICE_MANAGED
> +};
> +
> +static inline const char *mlx4_steering_mode_str(int steering_mode)
> +{
> +       switch (steering_mode) {
> +       case MLX4_STEERING_MODE_A0:
> +               return "A0 steering";
> +
> +       case MLX4_STEERING_MODE_B0:
> +               return "B0 steering";
> +
> +       case MLX4_STEERING_MODE_DEVICE_MANAGED:
> +               return "Device managed flow steering";
> +
> +       default:
> +               return "Unrecognize steering mode";
> +       }
> +}
> +
> +enum {
> +       MLX4_DEV_CAP_FLAG_RC            = 1LL <<  0,
> +       MLX4_DEV_CAP_FLAG_UC            = 1LL <<  1,
> +       MLX4_DEV_CAP_FLAG_UD            = 1LL <<  2,
> +       MLX4_DEV_CAP_FLAG_XRC           = 1LL <<  3,
> +       MLX4_DEV_CAP_FLAG_SRQ           = 1LL <<  6,
> +       MLX4_DEV_CAP_FLAG_IPOIB_CSUM    = 1LL <<  7,
> +       MLX4_DEV_CAP_FLAG_BAD_PKEY_CNTR = 1LL <<  8,
> +       MLX4_DEV_CAP_FLAG_BAD_QKEY_CNTR = 1LL <<  9,
> +       MLX4_DEV_CAP_FLAG_DPDP          = 1LL << 12,
> +       MLX4_DEV_CAP_FLAG_BLH           = 1LL << 15,
> +       MLX4_DEV_CAP_FLAG_MEM_WINDOW    = 1LL << 16,
> +       MLX4_DEV_CAP_FLAG_APM           = 1LL << 17,
> +       MLX4_DEV_CAP_FLAG_ATOMIC        = 1LL << 18,
> +       MLX4_DEV_CAP_FLAG_RAW_MCAST     = 1LL << 19,
> +       MLX4_DEV_CAP_FLAG_UD_AV_PORT    = 1LL << 20,
> +       MLX4_DEV_CAP_FLAG_UD_MCAST      = 1LL << 21,
> +       MLX4_DEV_CAP_FLAG_IBOE          = 1LL << 30,
> +       MLX4_DEV_CAP_FLAG_UC_LOOPBACK   = 1LL << 32,
> +       MLX4_DEV_CAP_FLAG_FCS_KEEP      = 1LL << 34,
> +       MLX4_DEV_CAP_FLAG_WOL_PORT1     = 1LL << 37,
> +       MLX4_DEV_CAP_FLAG_WOL_PORT2     = 1LL << 38,
> +       MLX4_DEV_CAP_FLAG_UDP_RSS       = 1LL << 40,
> +       MLX4_DEV_CAP_FLAG_VEP_UC_STEER  = 1LL << 41,
> +       MLX4_DEV_CAP_FLAG_VEP_MC_STEER  = 1LL << 42,
> +       MLX4_DEV_CAP_FLAG_CROSS_CHANNEL = 1LL << 44,
> +       MLX4_DEV_CAP_FLAG_COUNTERS      = 1LL << 48,
> +       MLX4_DEV_CAP_FLAG_COUNTERS_EXT  = 1LL << 49,
> +       MLX4_DEV_CAP_FLAG_SET_PORT_ETH_SCHED = 1LL << 53,
> +       MLX4_DEV_CAP_FLAG_SENSE_SUPPORT = 1LL << 55,
> +       MLX4_DEV_CAP_FLAG_FAST_DROP     = 1LL << 57,
> +       MLX4_DEV_CAP_FLAG_PORT_MNG_CHG_EV = 1LL << 59,
> +       MLX4_DEV_CAP_FLAG_64B_EQE       = 1LL << 61,
> +       MLX4_DEV_CAP_FLAG_64B_CQE       = 1LL << 62
> +};
> +
> +enum {
> +       MLX4_DEV_CAP_FLAG2_RSS                  = 1LL <<  0,
> +       MLX4_DEV_CAP_FLAG2_RSS_TOP              = 1LL <<  1,
> +       MLX4_DEV_CAP_FLAG2_RSS_XOR              = 1LL <<  2,
> +       MLX4_DEV_CAP_FLAG2_FS_EN                = 1LL <<  3,
> +       MLX4_DEV_CAP_FLAG2_FSM                  = 1LL <<  4,
> +       MLX4_DEV_CAP_FLAG2_VLAN_CONTROL         = 1LL <<  5,
> +       MLX4_DEV_CAP_FLAG2_UPDATE_QP            = 1LL <<  6,
> +       MLX4_DEV_CAP_FLAG2_LB_SRC_CHK           = 1LL <<  7,
> +       MLX4_DEV_CAP_FLAG2_DMFS_IPOIB           = 1LL <<  8,
> +       MLX4_DEV_CAP_FLAG2_ETS_CFG              = 1LL <<  9,
> +       MLX4_DEV_CAP_FLAG2_ETH_BACKPL_AN_REP    = 1LL <<  10,
> +       MLX4_DEV_CAP_FLAG2_FLOWSTATS_EN         = 1LL <<  11,
> +       MLX4_DEV_CAP_FLAG2_RECOVERABLE_ERROR_EVENT = 1LL << 12,
> +       MLX4_DEV_CAP_FLAG2_TS                   = 1LL <<  13,
> +       MLX4_DEV_CAP_FLAG2_DRIVER_VERSION_TO_FW    = 1LL <<  14,
> +       MLX4_DEV_CAP_FLAG2_REASSIGN_MAC_EN      = 1LL <<  15,
> +       MLX4_DEV_CAP_FLAG2_VXLAN_OFFLOADS       = 1LL <<  16,
> +       MLX4_DEV_CAP_FLAG2_FS_EN_NCSI           = 1LL <<  17,
> +       MLX4_DEV_CAP_FLAG2_80_VFS               = 1LL <<  18,
> +       MLX4_DEV_CAP_FLAG2_DMFS_TAG_MODE        = 1LL <<  19,
> +       MLX4_DEV_CAP_FLAG2_ROCEV2               = 1LL <<  20,
> +       MLX4_DEV_CAP_FLAG2_ETH_PROT_CTRL        = 1LL <<  21,
> +       MLX4_DEV_CAP_FLAG2_CQE_STRIDE           = 1LL <<  22,
> +       MLX4_DEV_CAP_FLAG2_EQE_STRIDE           = 1LL <<  23,
> +       MLX4_DEV_CAP_FLAG2_UPDATE_QP_SRC_CHECK_LB = 1LL << 24,
> +       MLX4_DEV_CAP_FLAG2_RX_CSUM_MODE         = 1LL <<  25,
> +};
> +
> +/* bit enums for an 8-bit flags field indicating special use
> + * QPs which require special handling in qp_reserve_range.
> + * Currently, this only includes QPs used by the ETH interface,
> + * where we expect to use blueflame.  These QPs must not have
> + * bits 6 and 7 set in their qp number.
> + *
> + * This enum may use only bits 0..7.
> + */
> +enum {
> +       MLX4_RESERVE_BF_QP      = 1 << 7,
> +};
> +
> +enum {
> +       MLX4_DEV_CAP_CQ_FLAG_IO                 = 1 <<  0
> +};
> +
> +enum {
> +       MLX4_DEV_CAP_64B_EQE_ENABLED    = 1LL << 0,
> +       MLX4_DEV_CAP_64B_CQE_ENABLED    = 1LL << 1
> +};
> +
> +enum {
> +       MLX4_USER_DEV_CAP_64B_CQE       = 1L << 0
> +};
> +
> +enum {
> +       MLX4_FUNC_CAP_64B_EQE_CQE       = 1L << 0
> +};
> +
> +
> +#define MLX4_ATTR_EXTENDED_PORT_INFO   cpu_to_be16(0xff90)
> +
> +enum {
> +       MLX4_BMME_FLAG_WIN_TYPE_2B      = 1 << 1,
> +       MLX4_BMME_FLAG_LOCAL_INV        = 1 <<  6,
> +       MLX4_BMME_FLAG_REMOTE_INV       = 1 <<  7,
> +       MLX4_BMME_FLAG_TYPE_2_WIN       = 1 <<  9,
> +       MLX4_BMME_FLAG_RESERVED_LKEY    = 1 << 10,
> +       MLX4_BMME_FLAG_FAST_REG_WR      = 1 << 11,
> +};
> +
> +enum mlx4_event {
> +       MLX4_EVENT_TYPE_COMP               = 0x00,
> +       MLX4_EVENT_TYPE_PATH_MIG           = 0x01,
> +       MLX4_EVENT_TYPE_COMM_EST           = 0x02,
> +       MLX4_EVENT_TYPE_SQ_DRAINED         = 0x03,
> +       MLX4_EVENT_TYPE_SRQ_QP_LAST_WQE    = 0x13,
> +       MLX4_EVENT_TYPE_SRQ_LIMIT          = 0x14,
> +       MLX4_EVENT_TYPE_CQ_ERROR           = 0x04,
> +       MLX4_EVENT_TYPE_WQ_CATAS_ERROR     = 0x05,
> +       MLX4_EVENT_TYPE_EEC_CATAS_ERROR    = 0x06,
> +       MLX4_EVENT_TYPE_PATH_MIG_FAILED    = 0x07,
> +       MLX4_EVENT_TYPE_WQ_INVAL_REQ_ERROR = 0x10,
> +       MLX4_EVENT_TYPE_WQ_ACCESS_ERROR    = 0x11,
> +       MLX4_EVENT_TYPE_SRQ_CATAS_ERROR    = 0x12,
> +       MLX4_EVENT_TYPE_LOCAL_CATAS_ERROR  = 0x08,
> +       MLX4_EVENT_TYPE_PORT_CHANGE        = 0x09,
> +       MLX4_EVENT_TYPE_EQ_OVERFLOW        = 0x0f,
> +       MLX4_EVENT_TYPE_ECC_DETECT         = 0x0e,
> +       MLX4_EVENT_TYPE_CMD                = 0x0a,
> +       MLX4_EVENT_TYPE_VEP_UPDATE         = 0x19,
> +       MLX4_EVENT_TYPE_COMM_CHANNEL       = 0x18,
> +       MLX4_EVENT_TYPE_OP_REQUIRED        = 0x1a,
> +       MLX4_EVENT_TYPE_FATAL_WARNING      = 0x1b,
> +       MLX4_EVENT_TYPE_FLR_EVENT          = 0x1c,
> +       MLX4_EVENT_TYPE_PORT_MNG_CHG_EVENT = 0x1d,
> +       MLX4_EVENT_TYPE_RECOVERABLE_ERROR_EVENT  = 0x3e,
> +       MLX4_EVENT_TYPE_NONE               = 0xff,
> +};
> +
> +enum {
> +       MLX4_PORT_CHANGE_SUBTYPE_DOWN   = 1,
> +       MLX4_PORT_CHANGE_SUBTYPE_ACTIVE = 4
> +};
> +
> +enum {
> +       MLX4_RECOVERABLE_ERROR_EVENT_SUBTYPE_BAD_CABLE          = 1,
> +       MLX4_RECOVERABLE_ERROR_EVENT_SUBTYPE_UNSUPPORTED_CABLE  = 2,
> +};
> +
> +enum {
> +       MLX4_FATAL_WARNING_SUBTYPE_WARMING = 0,
> +};
> +
> +enum slave_port_state {
> +       SLAVE_PORT_DOWN = 0,
> +       SLAVE_PENDING_UP,
> +       SLAVE_PORT_UP,
> +};
> +
> +enum slave_port_gen_event {
> +       SLAVE_PORT_GEN_EVENT_DOWN = 0,
> +       SLAVE_PORT_GEN_EVENT_UP,
> +       SLAVE_PORT_GEN_EVENT_NONE,
> +};
> +
> +enum slave_port_state_event {
> +       MLX4_PORT_STATE_DEV_EVENT_PORT_DOWN,
> +       MLX4_PORT_STATE_DEV_EVENT_PORT_UP,
> +       MLX4_PORT_STATE_IB_PORT_STATE_EVENT_GID_VALID,
> +       MLX4_PORT_STATE_IB_EVENT_GID_INVALID,
> +};
> +
> +enum {
> +       MLX4_PERM_LOCAL_READ    = 1 << 10,
> +       MLX4_PERM_LOCAL_WRITE   = 1 << 11,
> +       MLX4_PERM_REMOTE_READ   = 1 << 12,
> +       MLX4_PERM_REMOTE_WRITE  = 1 << 13,
> +       MLX4_PERM_ATOMIC        = 1 << 14,
> +       MLX4_PERM_BIND_MW       = 1 << 15,
> +};
> +
> +enum {
> +       MLX4_OPCODE_NOP                 = 0x00,
> +       MLX4_OPCODE_SEND_INVAL          = 0x01,
> +       MLX4_OPCODE_RDMA_WRITE          = 0x08,
> +       MLX4_OPCODE_RDMA_WRITE_IMM      = 0x09,
> +       MLX4_OPCODE_SEND                = 0x0a,
> +       MLX4_OPCODE_SEND_IMM            = 0x0b,
> +       MLX4_OPCODE_LSO                 = 0x0e,
> +       MLX4_OPCODE_RDMA_READ           = 0x10,
> +       MLX4_OPCODE_ATOMIC_CS           = 0x11,
> +       MLX4_OPCODE_ATOMIC_FA           = 0x12,
> +       MLX4_OPCODE_MASKED_ATOMIC_CS    = 0x14,
> +       MLX4_OPCODE_MASKED_ATOMIC_FA    = 0x15,
> +       MLX4_OPCODE_BIND_MW             = 0x18,
> +       MLX4_OPCODE_FMR                 = 0x19,
> +       MLX4_OPCODE_LOCAL_INVAL         = 0x1b,
> +       MLX4_OPCODE_CONFIG_CMD          = 0x1f,
> +
> +       MLX4_RECV_OPCODE_RDMA_WRITE_IMM = 0x00,
> +       MLX4_RECV_OPCODE_SEND           = 0x01,
> +       MLX4_RECV_OPCODE_SEND_IMM       = 0x02,
> +       MLX4_RECV_OPCODE_SEND_INVAL     = 0x03,
> +
> +       MLX4_CQE_OPCODE_ERROR           = 0x1e,
> +       MLX4_CQE_OPCODE_RESIZE          = 0x16,
> +};
> +
> +enum {
> +       MLX4_STAT_RATE_OFFSET   = 5
> +};
> +
> +enum mlx4_protocol {
> +       MLX4_PROT_IB_IPV6 = 0,
> +       MLX4_PROT_ETH,
> +       MLX4_PROT_IB_IPV4,
> +       MLX4_PROT_FCOE
> +};
> +
> +enum {
> +       MLX4_MTT_FLAG_PRESENT           = 1
> +};
> +
> +enum {
> +       MLX4_MAX_MTT_SHIFT              = 31
> +};
> +
> +enum mlx4_qp_region {
> +       MLX4_QP_REGION_FW = 0,
> +       MLX4_QP_REGION_ETH_ADDR,
> +       MLX4_QP_REGION_FC_ADDR,
> +       MLX4_QP_REGION_FC_EXCH,
> +       MLX4_NUM_QP_REGION
> +};
> +
> +enum mlx4_port_type {
> +       MLX4_PORT_TYPE_NONE     = 0,
> +       MLX4_PORT_TYPE_IB       = 1,
> +       MLX4_PORT_TYPE_ETH      = 2,
> +       MLX4_PORT_TYPE_AUTO     = 3,
> +       MLX4_PORT_TYPE_NA       = 4
> +};
> +
>
> *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***
>



-- 
Tomorrow Will Never Die



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAMOc5cytXAmC2NnWWnbQt7kL8yf3PqiO_9N3KAxoiihbQPjrvA>