Date: Mon, 13 Nov 2017 22:55:46 +0200 From: Andriy Gapon <avg@FreeBSD.org> To: Anish <akgupt3@gmail.com> Cc: "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org> Subject: Re: problem with pass-through on amd Message-ID: <e26820b8-048b-9684-f810-2cba364d0474@FreeBSD.org> In-Reply-To: <CALnRwMShTha5aKfY6EbB=n2JiiK1z_Y--39WoWGgbUUufafnmw@mail.gmail.com> References: <cee09b94-3a1d-ac36-c99a-99196faa75a9@FreeBSD.org> <CALnRwMShTha5aKfY6EbB=n2JiiK1z_Y--39WoWGgbUUufafnmw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 13/11/2017 22:41, Anish wrote: > Hi Andriy, > >>What's suspicious is that there are no interrupts either for the > ppt device on the host or the ahci device in the guest. > > I think most likely AMD IOMMU/amdvi is not enabled, by default it is disabled > and can be confirmed by sysctl. > > [root@ryzen /usr/home/anish]# sysctl hw.vmm.amdvi > hw.vmm.amdvi.domain_id: 3 > hw.vmm.amdvi.disable_io_fault: 0 > hw.vmm.amdvi.ptp_level: 4 > hw.vmm.amdvi.host_ptp: 1 > hw.vmm.amdvi.enable: 1 > hw.vmm.amdvi.count: 2 > > If hw.vmm.amdvi.enable is 0, set it and reload vmm and things should work. I followed your how-to, so I didn't make this mistake :-) $ sysctl hw.vmm.amdvi hw.vmm.amdvi.domain_id: 3 hw.vmm.amdvi.disable_io_fault: 0 hw.vmm.amdvi.ptp_level: 4 hw.vmm.amdvi.host_ptp: 1 hw.vmm.amdvi.enable: 1 hw.vmm.amdvi.count: 1 Could it be that AMD-Vi support is broken in the hardware or BIOS? Are there any checks that I could do? Here is what I see in dmesg, but it is not a full dump of IVRS, of course: $ dmesg | egrep -i 'ivrs|ivh|amd-vi|mmu' Table 'IVRS' at 0xbdafaf80 ACPI: IVRS 0x00000000BDAFAF80 0000F8 (v01 AMD RD890S 00202031 AMD 00000000) AMD-Vi IVRS VAsize = 64 PAsize = 52 GVAsize = 0 flags:0 ivhd0: <AMD-Vi/IOMMU or ivhd> on acpi0 ivhd0: Unknown dev entry:0xff ivhd0: Flag:3e<PassPW,ResPassPW,Isoc,IotlbSup,Coherent> ivhd0: max supported paging level:7 restricting to: 4 ivhd0: device supported range [0x0 - 0xb01] ivhd0: device [0xa0 - 0xa0]config:d7<INIT,ExtInt,NMI,LINT0> ivhd0: PCI cap 0x10b540f@0x40 feature:1<IOTLB> pci0: <base peripheral, IOMMU> at device 0.2 (no driver attached) > On Mon, Nov 13, 2017 at 11:02 AM, Andriy Gapon <avg@freebsd.org > <mailto:avg@freebsd.org>> wrote: > > > I have a FreeBSD guest on a FreeBSD host, both amd64 head. > I decided to experiment with PCI pass-through and used an AHCI controller for > the experiment. It seems like that partially works. The guest detects the > controller and attaches a driver to it, the messages (reported capabilities, > etc) look identical to those on the host. But the guest can not detect any > disks behind the controller. From what I can tell, the guest has no problems > accessing PCI configuration registers and a memory mapped BAR. Not sure about > the I/O ports. What's suspicious is that there are no interrupts either for the > ppt device on the host or the ahci device in the guest. Maybe that is what > causes the failure to see the disks. > > When the guest driver attaches to the controller I see these messages on the > host: > ppt0: attempting to allocate 1 MSI vectors (1 supported) > ppt0: using IRQ 265 for MSI > and in the guest: > ahci0: attempting to allocate 1 MSI vectors (1 supported) > ahci0: using IRQ 279 for MSI > > But vmstat -i does not report any interrupts. > > Just in case, the controller has some complex topology where the AHCI PCI device > is behind two PCI-PCI bridges from the main bus. But I think that that should > not matter and the guest needs to see only the AHCI device and the host handles > the bridges. > > Here are some more messages from the guest: > ahci0: <HighPoint RocketRAID 640 AHCI SATA controller> port > 0x21c0-0x21c7,0x21c8-0x21cb,0x21d0-0x21d7,0x21d8-0x21db,0x21e0-0x21ef mem > 0xc000e000-0xc000efff irq 17 at device 9.0 on pci0 > ahci0: attempting to allocate 1 MSI vectors (1 supported) > ahci0: using IRQ 279 for MSI > ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS > ahci0: Caps: 64bit NCQ SNTF AL 6Gbps PM FBS PMD SSC PSC 32cmd eSATA 2ports > ahcich0: <AHCI channel> at channel 0 on ahci0 > ahcich0: Caps: CPD ESP FBSCP > ahcich1: <AHCI channel> at channel 1 on ahci0 > ahcich1: Caps: CPD ESP FBSCP > ahcich0: AHCI reset... > ahcich0: SATA connect time=100us status=00000123 > ahcich0: AHCI reset: device found > ahcich1: AHCI reset... > ahcich1: SATA connect time=100us status=00000123 > ahcich1: AHCI reset: device found > ahcich1: AHCI reset: device ready after 100ms > ahcich1: Poll timeout on slot 1 port 15 > ahcich1: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 150 serr 00000000 > cmd 00318017 > ahcich0: AHCI reset: device ready after 200ms > ahcich0: Poll timeout on slot 1 port 15 > ahcich0: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 150 serr 00000000 > cmd 00318017 > And some more... > > Any ideas or suggestions? > Or maybe some additional information from me? > > Thank you! > > -- > Andriy Gapon > _______________________________________________ > freebsd-virtualization@freebsd.org > <mailto:freebsd-virtualization@freebsd.org> mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > <https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization> > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscribe@freebsd.org > <mailto:freebsd-virtualization-unsubscribe@freebsd.org>" > > -- Andriy Gapon
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?e26820b8-048b-9684-f810-2cba364d0474>