From owner-freebsd-arm@freebsd.org Tue Oct 23 15:46:16 2018 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B157FFEA759 for ; Tue, 23 Oct 2018 15:46:16 +0000 (UTC) (envelope-from greg@unrelenting.technology) Received: from out.migadu.com (out.migadu.com [91.121.223.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.migadu.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 309D782EA2 for ; Tue, 23 Oct 2018 15:46:15 +0000 (UTC) (envelope-from greg@unrelenting.technology) Received: (Migadu outbound); Tue, 23 Oct 2018 15:46:07 +0000 Authentication-Results: out.migadu.com; auth=pass (plain) Received: from [192.168.1.141] ([62.122.208.146]) by out.migadu.com (Haraka/2.8.16) with ESMTPSA id A944EB7C-E6C0-41C4-8E68-13F9CB93A68F.1 envelope-from (authenticated bits=0) (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 verify=FAIL); Tue, 23 Oct 2018 15:46:07 +0000 Date: Tue, 23 Oct 2018 18:46:05 +0300 From: Greg V Subject: Re: Poor virtio performance on Scaleway ARM systems To: Peter Jeremy Cc: freebsd-arm@freebsd.org Message-Id: <1540309565.1771.1@smtp.migadu.com> In-Reply-To: <20180916184657.GB24416@server.rulingia.com> References: <20180916184657.GB24416@server.rulingia.com> X-Mailer: geary/0.12.4 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed DKIM-Signature: v=1; a=rsa-sha256; bh=wcIJGX08xd7okFOrML54+qceg3m1WIv0JTICn8cfBgQ=; c=relaxed/simple; d=unrelenting.technology; h=from:subject:date:to; s=default; b=TkUuif791mDcdEsrF6VU5QlBxNmTVeOC3X+t5bymb3j32dz6vjSsIHAgpeBruSAmRmI21yIX9JCThQiuw0SzY3NorDSWMMdDfwHPFfIbsIP9bwdSHEwqmLCeQOfnMaG1a1pAee52Spc6cv3VrFejegbfE1WXx7lCTjupiLnwmBk= X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2018 15:46:16 -0000 On Sun, Sep 16, 2018 at 9:46 PM, Peter Jeremy wrote: > I have been playing with the 4-core ARM64 VPSs on > https://www.scaleway.com > and notice that the disk I/O performance (using virtio_blk) is > abyssmal. > Using "dd if=/dev/{vda|vtbd0} of=/dev/null bs=256k count=4096", I get > 400-500MBps, whilst under FreeBSD-12, I get about 5MBps. I've > checked on a > couple of instances and both Linux & FreeBSD on the same instance and > get > similar results. Linux & FreeBSD are both using a virtio block device > attached to the PCI bus. Rebuilding FreeBSD to turn off all the > debugging > has no effect. > > The only oddity I've found is that FreeBSD is reporting very high > interrupt > rates on gic0,p11, gic0,s4 and gic0,s5 whilst disk I/O is occurring. > Unfortunately, I can't tell what is attached to those interrupts (it's > not obvious from the dmesg and reported as "+"). gic0,s4 correlates to network activity. gic0,s5 correlates to disk activity. > I've done some searching and have only found general FUD ("FreeBSD > virtio > isn't any good") and nothing specifically related to Scaleway. > > Can anyone suggest where to look for a solution? I FIGURED IT OUT!! echo 'hw.pci.honor_msi_blacklist="0"' >> /boot/loader.conf SOMEHOW FreeBSD ended up using legacy interrupts instead of MSI-X. I have no idea how. Looking at pci_quirks, MSI disable is for old Intel stuff, some old VMWare stuff... NOTHING about QEMU/KVM or Cavium! With the fix, we're getting ~1000 interrupts on 'its0,4: virtio_pci1' (instead of ~30000 on gic), latency is no longer awful, and bandwidth is about 220 MB/s. YAY!!