From owner-svn-src-user@freebsd.org Wed Dec 30 09:02:05 2015 Return-Path: Delivered-To: svn-src-user@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B5B66A55FE1 for ; Wed, 30 Dec 2015 09:02:05 +0000 (UTC) (envelope-from ngie@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7DF671AD1; Wed, 30 Dec 2015 09:02:05 +0000 (UTC) (envelope-from ngie@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id tBU924h8098505; Wed, 30 Dec 2015 09:02:04 GMT (envelope-from ngie@FreeBSD.org) Received: (from ngie@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id tBU924Pw098498; Wed, 30 Dec 2015 09:02:04 GMT (envelope-from ngie@FreeBSD.org) Message-Id: <201512300902.tBU924Pw098498@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: ngie set sender to ngie@FreeBSD.org using -f From: Garrett Cooper Date: Wed, 30 Dec 2015 09:02:04 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org Subject: svn commit: r292910 - in user/ngie/stable-10-libnv: share/man/man9 sys/dev/pci X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Dec 2015 09:02:05 -0000 Author: ngie Date: Wed Dec 30 09:02:03 2015 New Revision: 292910 URL: https://svnweb.freebsd.org/changeset/base/292910 Log: MFC r279442,r279446: r279442 (by rstone): Refactor PCI resource allocation Refactor PCI resource allocation code to allow a request for a memory-mapped I/O window that is a multiple of a requested size. This is needed by the SR-IOV code because the VF BARs are all allocated contiguously. We can't just allocate a resource that is a multiple of a single VF BAR because the size of an allocation implies its alignment requirement. Differential Revision: https://reviews.freebsd.org/D71 Reviewed by: jhb Sponsored by: Sandvine Inc. r279446 (by rstone): Add manpages for SR-IOV enable/disable driver interface Differential Revision: https://reviews.freebsd.org/D75 Reviewed by: wblock Sponsored by: Sandvine Inc. Added: user/ngie/stable-10-libnv/share/man/man9/PCI_ADD_VF.9 - copied unchanged from r279446, head/share/man/man9/PCI_ADD_VF.9 user/ngie/stable-10-libnv/share/man/man9/PCI_INIT_IOV.9 - copied unchanged from r279446, head/share/man/man9/PCI_INIT_IOV.9 user/ngie/stable-10-libnv/share/man/man9/PCI_UNINIT_IOV.9 - copied unchanged from r279446, head/share/man/man9/PCI_UNINIT_IOV.9 Modified: user/ngie/stable-10-libnv/share/man/man9/Makefile user/ngie/stable-10-libnv/sys/dev/pci/pci.c user/ngie/stable-10-libnv/sys/dev/pci/pci_private.h Directory Properties: user/ngie/stable-10-libnv/ (props changed) Modified: user/ngie/stable-10-libnv/share/man/man9/Makefile ============================================================================== --- user/ngie/stable-10-libnv/share/man/man9/Makefile Wed Dec 30 08:55:13 2015 (r292909) +++ user/ngie/stable-10-libnv/share/man/man9/Makefile Wed Dec 30 09:02:03 2015 (r292910) @@ -194,6 +194,9 @@ MAN= accept_filter.9 \ p_candebug.9 \ p_cansee.9 \ pci.9 \ + PCI_ADD_VF.9 \ + PCI_INIT_IOV.9 \ + PCI_UNINIT_IOV.9 \ pfil.9 \ pfind.9 \ pget.9 \ Copied: user/ngie/stable-10-libnv/share/man/man9/PCI_ADD_VF.9 (from r279446, head/share/man/man9/PCI_ADD_VF.9) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ user/ngie/stable-10-libnv/share/man/man9/PCI_ADD_VF.9 Wed Dec 30 09:02:03 2015 (r292910, copy of r279446, head/share/man/man9/PCI_ADD_VF.9) @@ -0,0 +1,113 @@ +.\" +.\" Copyright (c) 2014 Sandvine Inc. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +.\" SUCH DAMAGE. +.\" +.\" $FreeBSD$ +.\" +.Dd May 24, 2014 +.Dt PCI_ADD_VF 9 +.Os +.Sh NAME +.Nm PCI_ADD_VF +.Nd inform a PF driver that a VF is being created +.Sh SYNOPSIS +.In sys/bus.h +.In machine/stdarg.h +.In sys/nv.h +.In dev/pci/pcireg.h +.In dev/pci/pcivar.h +.Ft int +.Fn PCI_ADD_VF "device_t dev" "uint16_t vfnum" "const nvlist_t *vf_config" +.Sh DESCRIPTION +The +.Fn PCI_ADD_VF +method is called by the PCI Single-Root I/O Virtualization +.Pq SR-IOV +infrastructure when it is initializating a new Virtual Function (VF) as a child +of the given Physical Function (PF) device. +This method will not be called until a successful call to +.Xr PCI_INIT_IOV 9 +has been made. +It is not guaranteed that this method will be called following a successful call +to +.Xr PCI_INIT_IOV 9 . +If the infrastructure encounters a failure to allocate resources following the +call to +.Xr PCI_INIT_IOV 9 , +the VF creation will be aborted and +.Xr PCI_UNINIT_IOV 9 +will be called immediately without any preceding calls to +.Nm . +.Pp +The index of the VF being initialized is passed in the +.Fa vfnum +argument. +VFs are always numbered sequentially starting at 0. +.Pp +If the driver requested device-specific configuration parameters via a VF schema +in its call to +.Xr pci_iov_attach 9 , +those parameters will be contained in the +.Pa vf_config +argument. +All configuration parameters that were either set as required parameters or that +had a default value set in the VF schema are guaranteed to be present in +.Fa vf_config . +Configuration parameters that were neither set as required nor were given a +default value are optional and may or may not be present in +.Fa vf_config . +.Fa vf_config +will not contain any configuration parameters that were not specified in the VF +schema. +All configuration parameters will have the correct type and will be in the range +of valid values specified in the schema. +.Pp +Note that it is possible for the user to set different configuration values on +different VF devices that are children of the same PF. +The PF driver must not cache configuration parameters passed in previous calls +to +.Fn PCI_ADD_VF +for other VFs and apply those parameters to the current VF. +.Pp +This function will not be called twice for the same +.Fa vf_num +on the same PF device without +.Xr PCI_UNINIT_IOV 9 +and +.Xr PCI_INIT_IOV 9 +first being called, in that order. +.Sh RETURN VALUES +This method returns 0 on success, otherwise an appropriate error is returned. +If this method returns an error then the current VF device will be destroyed +but the rest of the VF devices will be created and SR-IOV will be enabled on +the PF. +.Sh SEE ALSO +.Xr nv 9 , +.Xr pci 9 , +.Xr pci_iov_schema 9 , +.Xr PCI_INIT_IOV 9 , +.Xr PCI_UNINIT_IOV 9 +.Sh AUTHORS +This manual page was written by +.An Ryan Stone Aq Mt rstone@FreeBSD.org . Copied: user/ngie/stable-10-libnv/share/man/man9/PCI_INIT_IOV.9 (from r279446, head/share/man/man9/PCI_INIT_IOV.9) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ user/ngie/stable-10-libnv/share/man/man9/PCI_INIT_IOV.9 Wed Dec 30 09:02:03 2015 (r292910, copy of r279446, head/share/man/man9/PCI_INIT_IOV.9) @@ -0,0 +1,86 @@ +.\" +.\" Copyright (c) 2014 Sandvine Inc. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +.\" SUCH DAMAGE. +.\" +.\" $FreeBSD$ +.\" +.Dd May 24, 2014 +.Dt PCI_INIT_IOV 9 +.Os +.Sh NAME +.Nm PCI_INIT_IOV +.Nd enable SR-IOV on a PF device +.Sh SYNOPSIS +.In sys/bus.h +.In machine/stdarg.h +.In sys/nv.h +.In dev/pci/pcireg.h +.In dev/pci/pcivar.h +.Ft int +.Fn PCI_INIT_IOV "device_t dev" "uint16_t num_vfs" "const nvlist_t *pf_config" +.Sh DESCRIPTION +The +.Fn PCI_INIT_IOV +method is called by the PCI Single-Root I/O Virtualization (SR-IOV) +infrastucture when the user requests that SR-IOV be enabled on a Physical +Function (PF). +The number of Virtual Functions (VFs) that will be created is passed to this +method in the +.Fa num_vfs +argument. +.Pp +If the driver requested device-specific PF configuration parameters via a PF +schema in its call to +.Xr pci_iov_attach 9 , +those parameters will be available in the +.Fa pf_config +argument. +All configuration parameters that were either set as required parameters or that +had a default value set in the PF schema are guaranteed to be present in +.Fa pf_config . +Configuration parameters that were neither set as required nor were given a +default value are optional and may or may not be present in +.Fa pf_config . +.Fa pf_config +will not contain any configuration parameters that were not specified in the PF +schema. +All configuration parameters will have the correct type and are in the range of +valid values specified in the schema. +.Pp +If this method returns successfully, then this method will not be called again +on the same device until after a call to +.Xr PCI_UNINIT_IOV . +.Sh RETURN VALUES +Returns 0 on success, otherwise an appropriate error is returned. +If this method returns an error then the SR-IOV configuration will be aborted +and no VFs will be created. +.Sh SEE ALSO +.Xr nv 9 , +.Xr pci 9 , +.Xr pci_iov_schema 9 , +.Xr PCI_ADD_VF 9 , +.Xr PCI_UNINIT_IOV 9 +.Sh AUTHORS +This manual page was written by +.An Ryan Stone Aq Mt rstone@FreeBSD.org . Copied: user/ngie/stable-10-libnv/share/man/man9/PCI_UNINIT_IOV.9 (from r279446, head/share/man/man9/PCI_UNINIT_IOV.9) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ user/ngie/stable-10-libnv/share/man/man9/PCI_UNINIT_IOV.9 Wed Dec 30 09:02:03 2015 (r292910, copy of r279446, head/share/man/man9/PCI_UNINIT_IOV.9) @@ -0,0 +1,64 @@ +.\" +.\" Copyright (c) 2014 Sandvine Inc. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +.\" SUCH DAMAGE. +.\" +.\" $FreeBSD$ +.\" +.Dd May 24, 2014 +.Dt PCI_UNINIT_IOV 9 +.Os +.Sh NAME +.Nm PCI_UNINIT_IOV +.Nd disable SR-IOV on a PF device +.Sh SYNOPSIS +.In sys/bus.h +.In dev/pci/pcireg.h +.In dev/pci/pcivar.h +.Ft void +.Fn PCI_UNINIT_IOV "device_t dev" +.Sh DESCRIPTION +The +.Fn PCI_UNINIT_IOV +method is called by the PCI Single-Root I/O Virtualization (SR-IOV) +infrastructure when the user requests that SR-IOV be disabled on a Physical +Function (PF). +When this method is called, the PF driver must release any SR-IOV-related +resources that it has allocated and disable any device-specific SR-IOV +configuration in the device. +.Pp +This method will only be called following a successful call to +.Xr PCI_INIT_IOV . +It is not guaranteed that +.Xr PCI_ADD_VF +will have been called for any Virtual Function (VF) after the call to +.Xr PCI_INIT_IOV +and before the call to +.Nm . +.Sh SEE ALSO +.Xr pci 9 , +.Xr PCI_ADD_VF 9 , +.Xr PCI_INIT_IOV 9 +.Sh AUTHORS +This manual page was written by +.An Ryan Stone Aq Mt rstone@FreeBSD.org . Modified: user/ngie/stable-10-libnv/sys/dev/pci/pci.c ============================================================================== --- user/ngie/stable-10-libnv/sys/dev/pci/pci.c Wed Dec 30 08:55:13 2015 (r292909) +++ user/ngie/stable-10-libnv/sys/dev/pci/pci.c Wed Dec 30 09:02:03 2015 (r292910) @@ -77,7 +77,6 @@ __FBSDID("$FreeBSD$"); static int pci_has_quirk(uint32_t devid, int quirk); static pci_addr_t pci_mapbase(uint64_t mapreg); static const char *pci_maptype(uint64_t mapreg); -static int pci_mapsize(uint64_t testval); static int pci_maprange(uint64_t mapreg); static pci_addr_t pci_rombase(uint64_t mapreg); static int pci_romsize(uint64_t testval); @@ -492,7 +491,7 @@ pci_maptype(uint64_t mapreg) /* return log2 of map size decoded for memory or port map */ -static int +int pci_mapsize(uint64_t testval) { int ln2size; @@ -2674,8 +2673,9 @@ pci_memen(device_t dev) return (pci_read_config(dev, PCIR_COMMAND, 2) & PCIM_CMD_MEMEN) != 0; } -static void -pci_read_bar(device_t dev, int reg, pci_addr_t *mapp, pci_addr_t *testvalp) +void +pci_read_bar(device_t dev, int reg, pci_addr_t *mapp, pci_addr_t *testvalp, + int *bar64) { struct pci_devinfo *dinfo; pci_addr_t map, testval; @@ -2695,6 +2695,8 @@ pci_read_bar(device_t dev, int reg, pci_ pci_write_config(dev, reg, map, 4); *mapp = map; *testvalp = testval; + if (bar64 != NULL) + *bar64 = 0; return; } @@ -2736,6 +2738,8 @@ pci_read_bar(device_t dev, int reg, pci_ *mapp = map; *testvalp = testval; + if (bar64 != NULL) + *bar64 = (ln2range == 64); } static void @@ -2790,7 +2794,7 @@ pci_bar_enabled(device_t dev, struct pci return ((cmd & PCIM_CMD_PORTEN) != 0); } -static struct pci_map * +struct pci_map * pci_add_bar(device_t dev, int reg, pci_addr_t value, pci_addr_t size) { struct pci_devinfo *dinfo; @@ -2861,7 +2865,7 @@ pci_add_map(device_t bus, device_t dev, return (barlen); } - pci_read_bar(dev, reg, &map, &testval); + pci_read_bar(dev, reg, &map, &testval, NULL); if (PCI_BAR_MEM(map)) { type = SYS_RES_MEMORY; if (map & PCIM_BAR_MEM_PREFETCH) @@ -4531,7 +4535,7 @@ DB_SHOW_COMMAND(pciregs, db_pci_dump) static struct resource * pci_reserve_map(device_t dev, device_t child, int type, int *rid, - u_long start, u_long end, u_long count, u_int flags) + u_long start, u_long end, u_long count, u_int num, u_int flags) { struct pci_devinfo *dinfo = device_get_ivars(child); struct resource_list *rl = &dinfo->resources; @@ -4555,7 +4559,7 @@ pci_reserve_map(device_t dev, device_t c * have a atapci device in legacy mode and it fails * here, that other code is broken. */ - pci_read_bar(child, *rid, &map, &testval); + pci_read_bar(child, *rid, &map, &testval, NULL); /* * Determine the size of the BAR and ignore BARs with a size @@ -4597,7 +4601,7 @@ pci_reserve_map(device_t dev, device_t c * situation where we might allocate the excess to * another driver, which won't work. */ - count = (pci_addr_t)1 << mapsize; + count = ((pci_addr_t)1 << mapsize) * num; if (RF_ALIGNMENT(flags) < mapsize) flags = (flags & ~RF_ALIGNMENT_MASK) | RF_ALIGNMENT_LOG2(mapsize); if (PCI_BAR_MEM(map) && (map & PCIM_BAR_MEM_PREFETCH)) @@ -4628,8 +4632,8 @@ out: } struct resource * -pci_alloc_resource(device_t dev, device_t child, int type, int *rid, - u_long start, u_long end, u_long count, u_int flags) +pci_alloc_multi_resource(device_t dev, device_t child, int type, int *rid, + u_long start, u_long end, u_long count, u_long num, u_int flags) { struct pci_devinfo *dinfo; struct resource_list *rl; @@ -4637,10 +4641,6 @@ pci_alloc_resource(device_t dev, device_ struct resource *res; pcicfgregs *cfg; - if (device_get_parent(child) != dev) - return (BUS_ALLOC_RESOURCE(device_get_parent(dev), child, - type, rid, start, end, count, flags)); - /* * Perform lazy resource allocation */ @@ -4697,7 +4697,7 @@ pci_alloc_resource(device_t dev, device_ rle = resource_list_find(rl, type, *rid); if (rle == NULL) { res = pci_reserve_map(dev, child, type, rid, start, end, - count, flags); + count, num, flags); if (res == NULL) return (NULL); } @@ -4706,6 +4706,19 @@ pci_alloc_resource(device_t dev, device_ start, end, count, flags)); } +struct resource * +pci_alloc_resource(device_t dev, device_t child, int type, int *rid, + u_long start, u_long end, u_long count, u_int flags) +{ + + if (device_get_parent(child) != dev) + return (BUS_ALLOC_RESOURCE(device_get_parent(dev), child, + type, rid, start, end, count, flags)); + + return (pci_alloc_multi_resource(dev, child, type, rid, start, end, + count, 1, flags)); +} + int pci_release_resource(device_t dev, device_t child, int type, int rid, struct resource *r) Modified: user/ngie/stable-10-libnv/sys/dev/pci/pci_private.h ============================================================================== --- user/ngie/stable-10-libnv/sys/dev/pci/pci_private.h Wed Dec 30 08:55:13 2015 (r292909) +++ user/ngie/stable-10-libnv/sys/dev/pci/pci_private.h Wed Dec 30 09:02:03 2015 (r292910) @@ -139,4 +139,14 @@ void pci_cfg_restore(device_t, struct p */ void pci_cfg_save(device_t, struct pci_devinfo *, int); +int pci_mapsize(uint64_t testval); +void pci_read_bar(device_t dev, int reg, pci_addr_t *mapp, + pci_addr_t *testvalp, int *bar64); +struct pci_map *pci_add_bar(device_t dev, int reg, pci_addr_t value, + pci_addr_t size); + +struct resource *pci_alloc_multi_resource(device_t dev, device_t child, + int type, int *rid, u_long start, u_long end, u_long count, + u_long num, u_int flags); + #endif /* _PCI_PRIVATE_H_ */