Skip site navigation (1)Skip section navigation (2) 
Date:      Thu, 25 Oct 2018 08:58:40 -0700
From:      John Baldwin <jhb@FreeBSD.org>
To:        Peter Jeremy <peter@rulingia.com>
Cc:        Greg V <greg@unrelenting.technology>, freebsd-arm@freebsd.org
Subject:   Re: Poor virtio performance on Scaleway ARM systems
Message-ID:  <f232c2cd-eecc-91e9-eb44-dbeb5ccefd0a@FreeBSD.org>
In-Reply-To: <20181023184949.GB20902@server.rulingia.com>
References:  <20180916184657.GB24416@server.rulingia.com> <1540309565.1771.1@smtp.migadu.com> <966ed99b-4709-5693-dca2-77e3e6d7d830@FreeBSD.org> <20181023184949.GB20902@server.rulingia.com>
next in thread | previous in thread | raw e-mail | index | archive | help 
On 10/23/18 11:49 AM, Peter Jeremy wrote:
> On 2018-Oct-23 10:04:04 -0700, John Baldwin <jhb@FreeBSD.org> wrote:
>> On 10/23/18 8:46 AM, Greg V wrote:
>>> On Sun, Sep 16, 2018 at 9:46 PM, Peter Jeremy <peter@rulingia.com> 
>>> wrote:
>>>> I have been playing with the 4-core ARM64 VPSs on 
>>>> https://www.scaleway.com
>>>> and notice that the disk I/O performance (using virtio_blk) is 
>>>> abyssmal.
>>>>
>>>> The only oddity I've found is that FreeBSD is reporting very high 
>>>> interrupt
>>>> rates on gic0,p11, gic0,s4 and gic0,s5 whilst disk I/O is occurring.
>>>> Unfortunately, I can't tell what is attached to those interrupts (it's
>>>> not obvious from the dmesg and reported as "+").
>>> gic0,s4 correlates to network activity.
>>> gic0,s5 correlates to disk activity.
>>>
>>> echo 'hw.pci.honor_msi_blacklist="0"' >> /boot/loader.conf
> 
>> Can you make 'pciconf -lc' output available?  There is a "generic"
>> blacklist that assumes we will see a host bridge device with PCI-e
>> or PCI-X and if neither of those is present we also blacklist MSI.
>> However, we can certainly figure out how to make this work out of
>> the box.
> 
> $ sudo pciconf -lc
> hostb0@pci0:0:0:0:      class=0x060000 card=0x11001af4 chip=0x00081b36 rev=0x00 hdr=0x00
> virtio_pci0@pci0:0:1:0: class=0x020000 card=0x00011af4 chip=0x10001af4 rev=0x00 hdr=0x00
>     cap 11[98] = MSI-X supports 3 messages
>                  Table in map 0x14[0x0], PBA in map 0x14[0x800]
>     cap 09[84] = vendor (length 20)
>     cap 09[70] = vendor (length 20)
>     cap 09[60] = vendor (length 16)
>     cap 09[50] = vendor (length 16)
>     cap 09[40] = vendor (length 16)
> virtio_pci1@pci0:0:2:0: class=0x010000 card=0x00021af4 chip=0x10011af4 rev=0x00 hdr=0x00
>     cap 11[98] = MSI-X supports 2 messages
>                  Table in map 0x14[0x0], PBA in map 0x14[0x800]
>     cap 09[84] = vendor (length 20)
>     cap 09[70] = vendor (length 20)
>     cap 09[60] = vendor (length 16)
>     cap 09[50] = vendor (length 16)
>     cap 09[40] = vendor (length 16)

The relevant code is here in sys/dev/pci/pci.c:

/*
 * Determine if MSI is blacklisted globally on this system.  Currently,
 * we just check for blacklisted chipsets as represented by the
 * host-PCI bridge at device 0:0:0.  In the future, it may become
 * necessary to check other system attributes, such as the kenv values
 * that give the motherboard manufacturer and model number.
 */
static int
pci_msi_blacklisted(void)
{
        device_t dev;

        if (!pci_honor_msi_blacklist)
                return (0);

        /* Blacklist all non-PCI-express and non-PCI-X chipsets. */
        if (!(pcie_chipset || pcix_chipset)) {
                if (vm_guest != VM_GUEST_NO) {
                        /*
                         * Whitelist older chipsets in virtual
                         * machines known to support MSI.
                         */
                        dev = pci_find_bsf(0, 0, 0);
                        if (dev != NULL)
                                return (!pci_has_quirk(pci_get_devid(dev),
                                        PCI_QUIRK_ENABLE_MSI_VM));
                }
                return (1);
        }

        dev = pci_find_bsf(0, 0, 0);
        if (dev != NULL)
                return (pci_msi_device_blacklisted(dev));
        return (0);
}

In this case no devices have PCI-express or PCI-X capabilities.  We
could perhaps whitelist your hostb device, but I think a more general
solution might be to whitelist MSI if there are any virtio devices as
virtio devices can always do MSI.  I can work on a patch unless someone
else beats me to it, but for the virtio approach what I would do is
have a global 'virtio_seen' similar to 'pcie_chipset' that gets set to
true if we find any virtio devices during the initial bus scans and
then add '|| virtio_seen' to the second 'if' there.

-- 
John Baldwin

Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?f232c2cd-eecc-91e9-eb44-dbeb5ccefd0a>