Date: Mon, 13 Nov 2017 12:41:10 -0800 From: Anish <akgupt3@gmail.com> To: Andriy Gapon <avg@freebsd.org> Cc: "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org> Subject: Re: problem with pass-through on amd Message-ID: <CALnRwMShTha5aKfY6EbB=n2JiiK1z_Y--39WoWGgbUUufafnmw@mail.gmail.com> In-Reply-To: <cee09b94-3a1d-ac36-c99a-99196faa75a9@FreeBSD.org> References: <cee09b94-3a1d-ac36-c99a-99196faa75a9@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Andriy, >What's suspicious is that there are no interrupts either for the ppt device on the host or the ahci device in the guest. I think most likely AMD IOMMU/amdvi is not enabled, by default it is disabled and can be confirmed by sysctl. [root@ryzen /usr/home/anish]# sysctl hw.vmm.amdvi hw.vmm.amdvi.domain_id: 3 hw.vmm.amdvi.disable_io_fault: 0 hw.vmm.amdvi.ptp_level: 4 hw.vmm.amdvi.host_ptp: 1 hw.vmm.amdvi.enable: 1 hw.vmm.amdvi.count: 2 If hw.vmm.amdvi.enable is 0, set it and reload vmm and things should work. Regards, Anish On Mon, Nov 13, 2017 at 11:02 AM, Andriy Gapon <avg@freebsd.org> wrote: > > I have a FreeBSD guest on a FreeBSD host, both amd64 head. > I decided to experiment with PCI pass-through and used an AHCI controller > for > the experiment. It seems like that partially works. The guest detects the > controller and attaches a driver to it, the messages (reported > capabilities, > etc) look identical to those on the host. But the guest can not detect any > disks behind the controller. From what I can tell, the guest has no > problems > accessing PCI configuration registers and a memory mapped BAR. Not sure > about > the I/O ports. What's suspicious is that there are no interrupts either > for the > ppt device on the host or the ahci device in the guest. Maybe that is what > causes the failure to see the disks. > > When the guest driver attaches to the controller I see these messages on > the host: > ppt0: attempting to allocate 1 MSI vectors (1 supported) > ppt0: using IRQ 265 for MSI > and in the guest: > ahci0: attempting to allocate 1 MSI vectors (1 supported) > ahci0: using IRQ 279 for MSI > > But vmstat -i does not report any interrupts. > > Just in case, the controller has some complex topology where the AHCI PCI > device > is behind two PCI-PCI bridges from the main bus. But I think that that > should > not matter and the guest needs to see only the AHCI device and the host > handles > the bridges. > > Here are some more messages from the guest: > ahci0: <HighPoint RocketRAID 640 AHCI SATA controller> port > 0x21c0-0x21c7,0x21c8-0x21cb,0x21d0-0x21d7,0x21d8-0x21db,0x21e0-0x21ef mem > 0xc000e000-0xc000efff irq 17 at device 9.0 on pci0 > ahci0: attempting to allocate 1 MSI vectors (1 supported) > ahci0: using IRQ 279 for MSI > ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS > ahci0: Caps: 64bit NCQ SNTF AL 6Gbps PM FBS PMD SSC PSC 32cmd eSATA 2ports > ahcich0: <AHCI channel> at channel 0 on ahci0 > ahcich0: Caps: CPD ESP FBSCP > ahcich1: <AHCI channel> at channel 1 on ahci0 > ahcich1: Caps: CPD ESP FBSCP > ahcich0: AHCI reset... > ahcich0: SATA connect time=100us status=00000123 > ahcich0: AHCI reset: device found > ahcich1: AHCI reset... > ahcich1: SATA connect time=100us status=00000123 > ahcich1: AHCI reset: device found > ahcich1: AHCI reset: device ready after 100ms > ahcich1: Poll timeout on slot 1 port 15 > ahcich1: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 150 serr > 00000000 > cmd 00318017 > ahcich0: AHCI reset: device ready after 200ms > ahcich0: Poll timeout on slot 1 port 15 > ahcich0: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 150 serr > 00000000 > cmd 00318017 > And some more... > > Any ideas or suggestions? > Or maybe some additional information from me? > > Thank you! > > -- > Andriy Gapon > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization- > unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CALnRwMShTha5aKfY6EbB=n2JiiK1z_Y--39WoWGgbUUufafnmw>