Date: Tue, 5 May 2015 19:15:21 -0400 From: Ryan Stone <rysto32@gmail.com> To: Eric Badger <eric_badger@dell.com> Cc: "current@freebsd.org" <current@freebsd.org> Subject: Re: PCI PF memory decode disable when sizing VF BARs Message-ID: <CAFMmRNyYJPOyvRrFU_pR8dJBsWGCykL1CiPFBHEDCUFte1dSYQ@mail.gmail.com> In-Reply-To: <cb2ce4e0ff294bcd982c6db87ae64dfb@mspexmb1.Beer.Town> References: <cb2ce4e0ff294bcd982c6db87ae64dfb@mspexmb1.Beer.Town>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, May 5, 2015 at 9:17 AM, Eric Badger <eric_badger@dell.com> wrote: > Hi Ryan and -current, > > During IOV config, when setting up VF bars, several calls are made to > 'pci_read_bar' (in sys/dev/pci/pci.c) in order to size VF BARs, which > causes memory decoding to be turned off temporarily for the PF associated > with those VFs. I'm finding that this can interfere with an already running > PF. > Ouch. That's a nasty bug. Thanks for tracking this down. > 1. Check the value of the 'reg' arg to 'pci_read_bar' and, if it is > outside a standard BAR range, don't disable memory decoding. This is > simple, but feels a little hackish and may have consequences I'm missing. > 2. Pass some flag/context through such that pci_read_bar knows it is > configuring VF BARs (we might instead disable VF MSE in this case, if it is > enabled). It would be necessary to carry this flag/context through several > function calls before reaching pci_read_bar, which might end up being ugly. > 3. Rearrange the calls so that VF BARs are sized when the PF is not yet > running, and that info saved until VFs are created. Probably it would be > done when the PF BARs are sized for any device supporting IOV, even if that > device never creates VFs. > I don't think that #3 is possible. Unfortunately the BAR is sized again when the BAR is allocated so it's difficult so it wouldn't be enough to just size the BAR during attach. I would have to reserve the memory space during attach, but that might reserve physical address space that other devices need to function. Actually, that problem will prevent #2 from being easy to implement too. We'd have to add an extra flag to pci_alloc_multi_resource. I think that #1 is the best option. There's already a precedent for something similar has it has a special base for the device ROM BAR. I haven't had a chance to test this yet, but I believe that this patch will solve the problem: https://people.freebsd.org/~rstone/patches/iov_bar.diff
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFMmRNyYJPOyvRrFU_pR8dJBsWGCykL1CiPFBHEDCUFte1dSYQ>