Date: Mon, 3 Oct 2016 22:42:23 +0000 (UTC) From: John Baldwin <jhb@FreeBSD.org> To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-11@freebsd.org Subject: svn commit: r306660 - in stable/11/sys: conf dev/cxgbe modules/cxgbe/if_cxgbe Message-ID: <201610032242.u93MgNSZ046098@repo.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: jhb Date: Mon Oct 3 22:42:23 2016 New Revision: 306660 URL: https://svnweb.freebsd.org/changeset/base/306660 Log: MFC 303205,303722,305032,305752: Create VF devices on Chelsio T4/T5 NICs. 303205: Add a driver to create VF devices on Chelsio T4/T5 NICs. Chelsio NICs are a bit unique compared to some other NICs in that they expose different functionality on different physical functions. In particular, PF4 is used to manage the NIC interfaces ('t4nex' and 't5nex'). However, PF4 is not able to create VF devices. Instead, VFs are only supported by physical functions 0 through 3. This commit adds 't4iov' and 't5iov' drivers that attach to PF0-3. One extra wrinkle is that the iov devices cannot enable SR-IOV until the firwmare has been initialized by the main PF4 driver. To handle this case, a new t4_if kobj interface has been added to permit cross-calls between the PF drivers. The PF4 driver notifies sibling drivers when it is fully attached. It also requests sibling drivers to detach before it detaches. Sibling drivers query the PF4 driver during their attach routine to see if it is attached. If not, the sibling drivers defer their attach actions until the PF4 driver informs them it is attached. VF devices are associated with a single port on the NIC. VF devices created from PF0 are associated with the first port on the NIC, VFs from PF1 are associated with the second port, etc. VF devices can only be created from a PF device that has an associated port. Thus, on a 2-port card, VFs are only supported on PF0 and PF1. 303722: Use the port device name for the iov device for Chelsio T4/T5 cards. Chelsio T4/T5 adapters are multifunction cards. The main driver uses physical function 4 (PF4). However, VF devices for SR-IOV are only supported on physical functions 0 through 3, where PF0 creates VFs tied to port 0, etc. The t4iov/t5iov driver was previously added to create VF devices for ports that are present on each adapter. This change uses the recently added pci_iov_attach_name() function to name the character device in /dev/iov after the associated port on the card (e.g. /dev/iov/cxl0 is used to create VFs that share the cxl0 port). With this in place, mark the t4iov/t5iov devices quiet to prevent them from cluttering dmesg. 305032: Use device_verbose() to undo device_quiet() when detaching from t[45]iovX. The device quiet flag is not automatically reset on detach, so it is inherited by other device drivers (e.g. when switching a device driver over to ppt for PCI pass through). Cope with this behavior by explicitly marking the device verbose during detach so that the next driver can make its own decision. 305752: Remove explicit device_verbose() from the t4iov driver detach routine now that this case is handled generically. Sponsored by: Chelsio Communications Added: stable/11/sys/dev/cxgbe/t4_if.m - copied, changed from r303205, head/sys/dev/cxgbe/t4_if.m stable/11/sys/dev/cxgbe/t4_iov.c - copied, changed from r303205, head/sys/dev/cxgbe/t4_iov.c Modified: stable/11/sys/conf/files stable/11/sys/dev/cxgbe/t4_main.c stable/11/sys/modules/cxgbe/if_cxgbe/Makefile Directory Properties: stable/11/ (props changed) Modified: stable/11/sys/conf/files ============================================================================== --- stable/11/sys/conf/files Mon Oct 3 22:11:45 2016 (r306659) +++ stable/11/sys/conf/files Mon Oct 3 22:42:23 2016 (r306660) @@ -1244,6 +1244,8 @@ dev/cxgb/sys/uipc_mvec.c optional cxgb p compile-with "${NORMAL_C} -I$S/dev/cxgb" dev/cxgb/cxgb_t3fw.c optional cxgb cxgb_t3fw \ compile-with "${NORMAL_C} -I$S/dev/cxgb" +dev/cxgbe/t4_iov.c optional cxgbe pci \ + compile-with "${NORMAL_C} -I$S/dev/cxgbe" dev/cxgbe/t4_mp_ring.c optional cxgbe pci \ compile-with "${NORMAL_C} -I$S/dev/cxgbe" dev/cxgbe/t4_main.c optional cxgbe pci \ Copied and modified: stable/11/sys/dev/cxgbe/t4_if.m (from r303205, head/sys/dev/cxgbe/t4_if.m) ============================================================================== --- head/sys/dev/cxgbe/t4_if.m Fri Jul 22 22:46:41 2016 (r303205, copy source) +++ stable/11/sys/dev/cxgbe/t4_if.m Mon Oct 3 22:42:23 2016 (r306660) @@ -55,11 +55,11 @@ METHOD int detach_child { device_t dev; }; -# Called by a driver to query the PF4 driver for the unit number to use -# for a given port. If the port is not enabled on the adapter, this -# will fail. -METHOD int read_port_unit { +# Called by a driver to query the PF4 driver for the child device +# associated with a given port. If the port is not enabled on the adapter, +# this will fail. +METHOD int read_port_device { device_t dev; int port; - int *unit; + device_t *child; }; Copied and modified: stable/11/sys/dev/cxgbe/t4_iov.c (from r303205, head/sys/dev/cxgbe/t4_iov.c) ============================================================================== --- head/sys/dev/cxgbe/t4_iov.c Fri Jul 22 22:46:41 2016 (r303205, copy source) +++ stable/11/sys/dev/cxgbe/t4_iov.c Mon Oct 3 22:42:23 2016 (r306660) @@ -104,6 +104,7 @@ t4iov_probe(device_t dev) for (i = 0; i < nitems(t4iov_pciids); i++) { if (d == t4iov_pciids[i].device) { device_set_desc(dev, t4iov_pciids[i].desc); + device_quiet(dev); return (BUS_PROBE_DEFAULT); } } @@ -120,6 +121,7 @@ t5iov_probe(device_t dev) for (i = 0; i < nitems(t5iov_pciids); i++) { if (d == t5iov_pciids[i].device) { device_set_desc(dev, t5iov_pciids[i].desc); + device_quiet(dev); return (BUS_PROBE_DEFAULT); } } @@ -148,25 +150,27 @@ t4iov_attach_child(device_t dev) #ifdef PCI_IOV nvlist_t *pf_schema, *vf_schema; #endif - int error, unit; + device_t pdev; + int error; sc = device_get_softc(dev); MPASS(!sc->sc_attached); /* * PF0-3 are associated with a specific port on the NIC (PF0 - * with port 0, etc.). Ask the PF4 driver for the unit number - * for this function's associated port to determine if the port - * is present. + * with port 0, etc.). Ask the PF4 driver for the device for + * this function's associated port to determine if the port is + * present. */ - error = T4_READ_PORT_UNIT(sc->sc_main, pci_get_function(dev), &unit); + error = T4_READ_PORT_DEVICE(sc->sc_main, pci_get_function(dev), &pdev); if (error) return (0); #ifdef PCI_IOV pf_schema = pci_iov_schema_alloc_node(); vf_schema = pci_iov_schema_alloc_node(); - error = pci_iov_attach(dev, pf_schema, vf_schema); + error = pci_iov_attach_name(dev, pf_schema, vf_schema, "%s", + device_get_nameunit(pdev)); if (error) { device_printf(dev, "Failed to initialize SR-IOV: %d\n", error); return (0); @@ -205,10 +209,14 @@ static int t4iov_detach(device_t dev) { struct t4iov_softc *sc; + int error; sc = device_get_softc(dev); - if (sc->sc_attached) - return (t4iov_detach_child(dev)); + if (sc->sc_attached) { + error = t4iov_detach_child(dev); + if (error) + return (error); + } return (0); } Modified: stable/11/sys/dev/cxgbe/t4_main.c ============================================================================== --- stable/11/sys/dev/cxgbe/t4_main.c Mon Oct 3 22:11:45 2016 (r306659) +++ stable/11/sys/dev/cxgbe/t4_main.c Mon Oct 3 22:42:23 2016 (r306660) @@ -76,16 +76,22 @@ __FBSDID("$FreeBSD$"); #include "t4_ioctl.h" #include "t4_l2t.h" #include "t4_mp_ring.h" +#include "t4_if.h" /* T4 bus driver interface */ static int t4_probe(device_t); static int t4_attach(device_t); static int t4_detach(device_t); +static int t4_ready(device_t); +static int t4_read_port_device(device_t, int, device_t *); static device_method_t t4_methods[] = { DEVMETHOD(device_probe, t4_probe), DEVMETHOD(device_attach, t4_attach), DEVMETHOD(device_detach, t4_detach), + DEVMETHOD(t4_is_main_ready, t4_ready), + DEVMETHOD(t4_read_port_device, t4_read_port_device), + DEVMETHOD_END }; static driver_t t4_driver = { @@ -147,6 +153,9 @@ static device_method_t t5_methods[] = { DEVMETHOD(device_attach, t4_attach), DEVMETHOD(device_detach, t4_detach), + DEVMETHOD(t4_is_main_ready, t4_ready), + DEVMETHOD(t4_read_port_device, t4_read_port_device), + DEVMETHOD_END }; static driver_t t5_driver = { @@ -533,6 +542,7 @@ static int set_sched_queue(struct adapte static int toe_capability(struct vi_info *, int); #endif static int mod_event(module_t, int, void *); +static int notify_siblings(device_t, int); struct { uint16_t device; @@ -1062,6 +1072,8 @@ t4_attach(device_t dev) t4_set_desc(sc); + notify_siblings(dev, 0); + done: if (rc != 0 && sc->cdev) { /* cdev was created and so cxgbetool works; recover that way. */ @@ -1078,6 +1090,57 @@ done: return (rc); } +static int +t4_ready(device_t dev) +{ + struct adapter *sc; + + sc = device_get_softc(dev); + if (sc->flags & FW_OK) + return (0); + return (ENXIO); +} + +static int +t4_read_port_device(device_t dev, int port, device_t *child) +{ + struct adapter *sc; + struct port_info *pi; + + sc = device_get_softc(dev); + if (port < 0 || port >= MAX_NPORTS) + return (EINVAL); + pi = sc->port[port]; + if (pi == NULL || pi->dev == NULL) + return (ENXIO); + *child = pi->dev; + return (0); +} + +static int +notify_siblings(device_t dev, int detaching) +{ + device_t sibling; + int error, i; + + error = 0; + for (i = 0; i < PCI_FUNCMAX; i++) { + if (i == pci_get_function(dev)) + continue; + sibling = pci_find_dbsf(pci_get_domain(dev), pci_get_bus(dev), + pci_get_slot(dev), i); + if (sibling == NULL || !device_is_attached(sibling)) + continue; + if (detaching) + error = T4_DETACH_CHILD(sibling); + else + (void)T4_ATTACH_CHILD(sibling); + if (error) + break; + } + return (error); +} + /* * Idempotent */ @@ -1090,6 +1153,13 @@ t4_detach(device_t dev) sc = device_get_softc(dev); + rc = notify_siblings(dev, 1); + if (rc) { + device_printf(dev, + "failed to detach sibling devices: %d\n", rc); + return (rc); + } + if (sc->flags & FULL_INIT_DONE) t4_intr_disable(sc); Modified: stable/11/sys/modules/cxgbe/if_cxgbe/Makefile ============================================================================== --- stable/11/sys/modules/cxgbe/if_cxgbe/Makefile Mon Oct 3 22:11:45 2016 (r306659) +++ stable/11/sys/modules/cxgbe/if_cxgbe/Makefile Mon Oct 3 22:42:23 2016 (r306660) @@ -13,8 +13,10 @@ SRCS+= opt_inet.h SRCS+= opt_inet6.h SRCS+= opt_ofed.h SRCS+= opt_rss.h -SRCS+= pci_if.h +SRCS+= pci_if.h pci_iov_if.h +SRCS+= t4_if.c t4_if.h SRCS+= t4_hw.c +SRCS+= t4_iov.c SRCS+= t4_l2t.c SRCS+= t4_main.c SRCS+= t4_mp_ring.c
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201610032242.u93MgNSZ046098>