Date: Mon, 13 Nov 2017 20:22:06 -0800 From: Anish <akgupt3@gmail.com> To: Andriy Gapon <avg@freebsd.org> Cc: "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org> Subject: Re: problem with pass-through on amd Message-ID: <CALnRwMTq7pD0SBj3D0MRp7mxQ9nK5d1vk_XLA8uCnaYchCf18A@mail.gmail.com> In-Reply-To: <e26820b8-048b-9684-f810-2cba364d0474@FreeBSD.org> References: <cee09b94-3a1d-ac36-c99a-99196faa75a9@FreeBSD.org> <CALnRwMShTha5aKfY6EbB=n2JiiK1z_Y--39WoWGgbUUufafnmw@mail.gmail.com> <e26820b8-048b-9684-f810-2cba364d0474@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Andriy, Newer AMD IOMMU has EFR capability which is missing here, something like: ivhd0: PCI cap 0x190b640f@0x40 feature:19<IOTLB,EFR> Is this an old chipset? Also ivhd has fault interrupt enabled which is very helpful in debugging: [root@ryzen /home/anish/FreeBSD/head]# vmstat -ia |grep ivh irq256: ivhd0:fault 0 0 irq257: ivhd1:fault 0 0 Regards, Anish On Mon, Nov 13, 2017 at 12:55 PM, Andriy Gapon <avg@freebsd.org> wrote: > On 13/11/2017 22:41, Anish wrote: > > Hi Andriy, > > > >>What's suspicious is that there are no interrupts either for the > > ppt device on the host or the ahci device in the guest. > > > > I think most likely AMD IOMMU/amdvi is not enabled, by default it is > disabled > > and can be confirmed by sysctl. > > > > [root@ryzen /usr/home/anish]# sysctl hw.vmm.amdvi > > hw.vmm.amdvi.domain_id: 3 > > hw.vmm.amdvi.disable_io_fault: 0 > > hw.vmm.amdvi.ptp_level: 4 > > hw.vmm.amdvi.host_ptp: 1 > > hw.vmm.amdvi.enable: 1 > > hw.vmm.amdvi.count: 2 > > > > If hw.vmm.amdvi.enable is 0, set it and reload vmm and things should > work. > > > I followed your how-to, so I didn't make this mistake :-) > > $ sysctl hw.vmm.amdvi > hw.vmm.amdvi.domain_id: 3 > hw.vmm.amdvi.disable_io_fault: 0 > hw.vmm.amdvi.ptp_level: 4 > hw.vmm.amdvi.host_ptp: 1 > hw.vmm.amdvi.enable: 1 > hw.vmm.amdvi.count: 1 > > Could it be that AMD-Vi support is broken in the hardware or BIOS? > Are there any checks that I could do? > > Here is what I see in dmesg, but it is not a full dump of IVRS, of course: > $ dmesg | egrep -i 'ivrs|ivh|amd-vi|mmu' > Table 'IVRS' at 0xbdafaf80 > ACPI: IVRS 0x00000000BDAFAF80 0000F8 (v01 AMD RD890S 00202031 AMD > 00000000) > AMD-Vi IVRS VAsize = 64 PAsize = 52 GVAsize = 0 flags:0 > ivhd0: <AMD-Vi/IOMMU or ivhd> on acpi0 > ivhd0: Unknown dev entry:0xff > ivhd0: Flag:3e<PassPW,ResPassPW,Isoc,IotlbSup,Coherent> > ivhd0: max supported paging level:7 restricting to: 4 > ivhd0: device supported range [0x0 - 0xb01] > ivhd0: device [0xa0 - 0xa0]config:d7<INIT,ExtInt,NMI,LINT0> > ivhd0: PCI cap 0x10b540f@0x40 feature:1<IOTLB> > pci0: <base peripheral, IOMMU> at device 0.2 (no driver attached) > > > On Mon, Nov 13, 2017 at 11:02 AM, Andriy Gapon <avg@freebsd.org > > <mailto:avg@freebsd.org>> wrote: > > > > > > I have a FreeBSD guest on a FreeBSD host, both amd64 head. > > I decided to experiment with PCI pass-through and used an AHCI > controller for > > the experiment. It seems like that partially works. The guest > detects the > > controller and attaches a driver to it, the messages (reported > capabilities, > > etc) look identical to those on the host. But the guest can not > detect any > > disks behind the controller. From what I can tell, the guest has no > problems > > accessing PCI configuration registers and a memory mapped BAR. Not > sure about > > the I/O ports. What's suspicious is that there are no interrupts > either for the > > ppt device on the host or the ahci device in the guest. Maybe that > is what > > causes the failure to see the disks. > > > > When the guest driver attaches to the controller I see these > messages on the > > host: > > ppt0: attempting to allocate 1 MSI vectors (1 supported) > > ppt0: using IRQ 265 for MSI > > and in the guest: > > ahci0: attempting to allocate 1 MSI vectors (1 supported) > > ahci0: using IRQ 279 for MSI > > > > But vmstat -i does not report any interrupts. > > > > Just in case, the controller has some complex topology where the > AHCI PCI device > > is behind two PCI-PCI bridges from the main bus. But I think that > that should > > not matter and the guest needs to see only the AHCI device and the > host handles > > the bridges. > > > > Here are some more messages from the guest: > > ahci0: <HighPoint RocketRAID 640 AHCI SATA controller> port > > 0x21c0-0x21c7,0x21c8-0x21cb,0x21d0-0x21d7,0x21d8-0x21db,0x21e0-0x21ef > mem > > 0xc000e000-0xc000efff irq 17 at device 9.0 on pci0 > > ahci0: attempting to allocate 1 MSI vectors (1 supported) > > ahci0: using IRQ 279 for MSI > > ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with > FBS > > ahci0: Caps: 64bit NCQ SNTF AL 6Gbps PM FBS PMD SSC PSC 32cmd eSATA > 2ports > > ahcich0: <AHCI channel> at channel 0 on ahci0 > > ahcich0: Caps: CPD ESP FBSCP > > ahcich1: <AHCI channel> at channel 1 on ahci0 > > ahcich1: Caps: CPD ESP FBSCP > > ahcich0: AHCI reset... > > ahcich0: SATA connect time=100us status=00000123 > > ahcich0: AHCI reset: device found > > ahcich1: AHCI reset... > > ahcich1: SATA connect time=100us status=00000123 > > ahcich1: AHCI reset: device found > > ahcich1: AHCI reset: device ready after 100ms > > ahcich1: Poll timeout on slot 1 port 15 > > ahcich1: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 150 > serr 00000000 > > cmd 00318017 > > ahcich0: AHCI reset: device ready after 200ms > > ahcich0: Poll timeout on slot 1 port 15 > > ahcich0: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 150 > serr 00000000 > > cmd 00318017 > > And some more... > > > > Any ideas or suggestions? > > Or maybe some additional information from me? > > > > Thank you! > > > > -- > > Andriy Gapon > > _______________________________________________ > > freebsd-virtualization@freebsd.org > > <mailto:freebsd-virtualization@freebsd.org> mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > > <https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization> > > To unsubscribe, send any mail to > > "freebsd-virtualization-unsubscribe@freebsd.org > > <mailto:freebsd-virtualization-unsubscribe@freebsd.org>" > > > > > > > -- > Andriy Gapon >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CALnRwMTq7pD0SBj3D0MRp7mxQ9nK5d1vk_XLA8uCnaYchCf18A>