From owner-freebsd-hackers@FreeBSD.ORG Thu Mar 20 16:47:05 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 15EA11FB for ; Thu, 20 Mar 2014 16:47:05 +0000 (UTC) Received: from mail-qc0-x22c.google.com (mail-qc0-x22c.google.com [IPv6:2607:f8b0:400d:c01::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id C877CEE2 for ; Thu, 20 Mar 2014 16:47:04 +0000 (UTC) Received: by mail-qc0-f172.google.com with SMTP id i8so1338685qcq.31 for ; Thu, 20 Mar 2014 09:47:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=lY2yj8BRTnjQ2243AsIZVDxaFEO1XMrinHNJMZFFmB0=; b=A+ltjvPkpXE7RHcnGxCMnRszLd9ADAraBvhiAXBzWr+2W1GX9MtEgVYN1MP7vTodOZ zk58RAwqHEVFmRQ/B8ZEmQZrWmXcaKRnKfDvVJuwNp/aUHLmimuclbOcKizntjDfednK QQvHP6b/yDexizGfpROjs7aAoVxfHA+QqfNNNoDokViyoiwhyEYOLKqRoPgbuSHWhIMy uoJAAYWsCyoeAOhuf0iiiKLtrR5Ack9nRWMr0JfDE24YwJJ0LVRsiofk6PJYabccpiLM +vwCKspqYCRtdL335dv7TO1Yp5uHYZecsfcWKA8yBfX3dWZwReqz5tE/ZZ5XOQVvonLS BDKg== MIME-Version: 1.0 X-Received: by 10.140.23.52 with SMTP id 49mr49302831qgo.17.1395334024012; Thu, 20 Mar 2014 09:47:04 -0700 (PDT) Received: by 10.140.87.74 with HTTP; Thu, 20 Mar 2014 09:47:03 -0700 (PDT) In-Reply-To: References: <20140316141216.GA21331@kib.kiev.ua> Date: Thu, 20 Mar 2014 09:47:03 -0700 Message-ID: Subject: Re: [PATCH] Support PCIe Alternative RID Interpretation (ARI) From: Neel Natu To: Ryan Stone Content-Type: text/plain; charset=ISO-8859-1 Cc: Konstantin Belousov , "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Mar 2014 16:47:05 -0000 Hi Ryan, On Tue, Mar 18, 2014 at 10:51 AM, Ryan Stone wrote: > On Mon, Mar 17, 2014 at 5:20 PM, Neel Natu wrote: >> Hi Ryan, >> In any case it seems that the VT-d implementation in bhyve will work >> fine with ARI enabled devices. > > There was an assert that would trip for the function number being > greater than 8. > > > I've put together the following patch series: > > http://people.freebsd.org/~rstone/patches/ari/0001-Add-a-method-to-get-the-PCI-RID-for-a-device.patch > > This adds a method to get the RID that will be consumed by the VT-d > drivers. This patch is non-ARI only. > > > http://people.freebsd.org/~rstone/patches/ari/0002-Re-implement-the-DMAR-I-O-MMU-code-in-terms-of-PCI-R.patch > > This reworks the busdma DMAR code to work in terms of RIDs where > necessary. This should be a no-op. I tested this with > hw.dmar.enable=1 on a Nehalem with the em driver and a Sandy Bridge > with the igb and ixgbe driver and was able to pass traffic. > > > http://people.freebsd.org/~rstone/patches/ari/0003-Re-write-bhyve-s-I-O-MMU-handling-in-terms-of-PCI-RI.patch > > Same thing, but for bhyve. I'm not sure that passing the rid down to > the CPU-dependent layer is the right thing to do, because I'm not sure > what the AMD VT-d equivalent requires. Should I just pass down the > entire device_t and let the CPU layer deal with it? I tested loading > vmm.ko with a device assigned to the ppt driver but I didn't go as far > as starting a VM and using PCI passthrough. > The bhyve portion of the patch looks fine to me. A cursory read of the AMD IOMMU spec suggests that the translation table is indexed using the 16-bit RID so passing it into the CPU-dependent layer will work fine. best Neel > (Also, as you'd probably expected doing this with hw.dmar.enable=1 > causes all hell to break loose). > > > http://people.freebsd.org/~rstone/patches/ari/0004-Add-support-for-PCIe-ARI.patch > > This is a slightly reworked version of the previous patch. The main > difference are that there is a new implementation of pcib_get_rid that > understands ARI RIDs. I also fixed a bug where the default > implementation of pcib_numslots was not actually being used because I > misspelled DEFAULT as default in pcib_if.m. > > > http://people.freebsd.org/~rstone/patches/ari/0005-Print-status-of-ARI-capability-in-pciconf-c.patch > > This makes pciconf -c dump the status of ARI on PCIe downstream ports.