Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 16 Nov 2017 19:33:44 -0800
From:      Anish <akgupt3@gmail.com>
To:        Andriy Gapon <avg@freebsd.org>
Cc:        "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org>
Subject:   Re: problem with pass-through on amd
Message-ID:  <CALnRwMSajH4%2BBkVEZwLEZqoBEDUm8Xsoy-yWXeS06Ydy%2BEVgeQ@mail.gmail.com>
In-Reply-To: <c73ab383-4dd4-5aaf-1682-830fc625a80b@FreeBSD.org>
References:  <cee09b94-3a1d-ac36-c99a-99196faa75a9@FreeBSD.org> <CALnRwMShTha5aKfY6EbB=n2JiiK1z_Y--39WoWGgbUUufafnmw@mail.gmail.com> <e26820b8-048b-9684-f810-2cba364d0474@FreeBSD.org> <CALnRwMTq7pD0SBj3D0MRp7mxQ9nK5d1vk_XLA8uCnaYchCf18A@mail.gmail.com> <c73ab383-4dd4-5aaf-1682-830fc625a80b@FreeBSD.org>

next in thread | previous in thread | raw e-mail | index | archive | help
Hi Andriy,
>After I fixed the MSI setup problem that I described in the previous email
I was
finally able to see the real problem.

Awesome! fault interrupt handler has been very helpful in debugging AMD-Vi.

>I've passed both of the PCI device to bhyve and everything started working.
Thanks!
Bhyve uses single domain for all the passthrough devices of a guest, that's
why even with aliasing it worked.

>But it appears to me that FreeBSD currently ignores those aliases.
Also, it looks like we are not making any use of IVHD entries beyond
printing
them.

We make use of of very few IVRS entries, see ivhd_dev_parse() but not much
for aliasing entries. Found out AMD IOMMU spec has some details on it:

"An Alias device type entry is used for each peripheral that does not use
its own DeviceID information

in bus transactions. For example, peripherals downstream of a bridge device
that use the DeviceID of

the bridge must have a corresponding Alias Select or Alias Start of Range
entry to inform system

software which IOMMU Device Table entry will be used for translation
information."


Thanks a lot for trying out AMD-Vi, let me know if you have any other
feedback.

Thanks and regards,

Anish





On Thu, Nov 16, 2017 at 12:51 PM, Andriy Gapon <avg@freebsd.org> wrote:

> On 14/11/2017 06:22, Anish wrote:
> > [root@ryzen /home/anish/FreeBSD/head]# vmstat -ia |grep ivh
> > irq256: ivhd0:fault                    0          0
> > irq257: ivhd1:fault                    0          0
>
> After I fixed the MSI setup problem that I described in the previous email
> I was
> finally able to see the real problem.
> Now I have:
>
> irq263: ivhd0:fault                    9          0
>
> dev.ivhd.0.event_tail: 240
> dev.ivhd.0.event_head: 240
> dev.ivhd.0.event_intr_count: 9
>
>
> I've got a bunch of messages like these in the log:
> [IO_PAGE_FAULT EVT: devId:0xa01 DomId:0x2 Addr:0x70f70400x30<PR,RW>]
> [IO_PAGE_FAULT EVT: devId:0xa01 DomId:0x2 Addr:0x2fdf0400x30<PR,RW>]
> (BTW, there seems to be a missing space before 0x30)
>
> Now it's obvious what the problem is.
> The controller has a RID of 0xa00 (its PCI locator is 10:0:0) but the
> requests
> are coming from 0xa01.  The card actually has another PCI device at that
> location (10:0:1), it's a vestigial IDE controller (in a sense that it is
> not
> connected to any ports, so it cannot really provide any functionality).
>
> I've passed both of the PCI device to bhyve and everything started working.
> Thanks!
>
> I've googled a little bit and it seems that this is not an uncommon
> problem.
> Linux has a bunch of quirks for this and similar kinds of problems (they
> call it
> PCI or DMA aliasing):
> http://elixir.free-electrons.com/linux/v4.8.16/source/
> drivers/pci/quirks.c#L3701
> The comments say that sometimes requests come from devices that do not
> exist at
> all (or hidden beyond non-transparent bridges).
>
> An interesting new world for me :-)
>
> By the way, my understanding is that the alias IVHD entries in IVRS are
> designed
> to report such aliasing issues when they are known to the platform.
> But it appears to me that FreeBSD currently ignores those aliases.
> Also, it looks like we are not making any use of IVHD entries beyond
> printing
> them.  Based on the specification I think that we should use IVHD flags to
> set
> up appropriate bits in the corresponding device table entries.  But we are
> not
> doing that.  That's probably okay since we do not support the pass-though
> for
> devices with complex properties anyway.
>
> --
> Andriy Gapon
>



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CALnRwMSajH4%2BBkVEZwLEZqoBEDUm8Xsoy-yWXeS06Ydy%2BEVgeQ>