Date: Tue, 23 Oct 2018 18:46:05 +0300 From: Greg V <greg@unrelenting.technology> To: Peter Jeremy <peter@rulingia.com> Cc: freebsd-arm@freebsd.org Subject: Re: Poor virtio performance on Scaleway ARM systems Message-ID: <1540309565.1771.1@smtp.migadu.com> In-Reply-To: <20180916184657.GB24416@server.rulingia.com> References: <20180916184657.GB24416@server.rulingia.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Sep 16, 2018 at 9:46 PM, Peter Jeremy <peter@rulingia.com> wrote: > I have been playing with the 4-core ARM64 VPSs on > https://www.scaleway.com > and notice that the disk I/O performance (using virtio_blk) is > abyssmal. > Using "dd if=/dev/{vda|vtbd0} of=/dev/null bs=256k count=4096", I get > 400-500MBps, whilst under FreeBSD-12, I get about 5MBps. I've > checked on a > couple of instances and both Linux & FreeBSD on the same instance and > get > similar results. Linux & FreeBSD are both using a virtio block device > attached to the PCI bus. Rebuilding FreeBSD to turn off all the > debugging > has no effect. > > The only oddity I've found is that FreeBSD is reporting very high > interrupt > rates on gic0,p11, gic0,s4 and gic0,s5 whilst disk I/O is occurring. > Unfortunately, I can't tell what is attached to those interrupts (it's > not obvious from the dmesg and reported as "+"). gic0,s4 correlates to network activity. gic0,s5 correlates to disk activity. > I've done some searching and have only found general FUD ("FreeBSD > virtio > isn't any good") and nothing specifically related to Scaleway. > > Can anyone suggest where to look for a solution? I FIGURED IT OUT!! echo 'hw.pci.honor_msi_blacklist="0"' >> /boot/loader.conf SOMEHOW FreeBSD ended up using legacy interrupts instead of MSI-X. I have no idea how. Looking at pci_quirks, MSI disable is for old Intel stuff, some old VMWare stuff... NOTHING about QEMU/KVM or Cavium! With the fix, we're getting ~1000 interrupts on 'its0,4: virtio_pci1' (instead of ~30000 on gic), latency is no longer awful, and bandwidth is about 220 MB/s. YAY!!
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1540309565.1771.1>