Date: Tue, 4 Feb 2014 00:23:11 -0600 From: Bryan Venteicher <bryanv@freebsd.org> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: Bryan Venteicher <bryanv@freebsd.org>, Alfred Perlstein <alfred@freebsd.org>, freebsd-stable@freebsd.org Subject: Re: Major performance/stability regression in virtio network drivers between 9.2-RELEASE and 10.0-RC5 Message-ID: <CAGaYwLc-111jEF2moA8da0Nzwno=HVebcbRZ%2B2bBFFSGOkWEoQ@mail.gmail.com> In-Reply-To: <449096679.2303214.1391470484784.JavaMail.root@uoguelph.ca> References: <52DBE19C.4040101@freebsd.org> <449096679.2303214.1391470484784.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
--001a11c2e19085471b04f18eaebd
Content-Type: text/plain; charset=ISO-8859-1
On Mon, Feb 3, 2014 at 5:34 PM, Rick Macklem <rmacklem@uoguelph.ca> wrote:
> Alfred Perlstein wrote:
> > What does vmstat say about memory zones? I think vmstat -m/vmstat -z
> > and also netstat -m?
> >
> > Are you set with the same mbufs on both 9 and 10?
> >
> > -Alfred
> >
> >
> > On 1/18/14 1:32 PM, Eric Dombroski wrote:
> > > Adrian:
> > >
> > > Yes, no change.
> > >
> > > -Eric
> > >
> > >
> > >
> > >
> > >
> > > On Sat, Jan 18, 2014 at 4:06 PM, Adrian Chadd
> > > <adrian.chadd@gmail.com>wrote:
> > >
> > >> Hi,
> > >>
> > >> Have you tried disabling tso?
> > >>
> > >> Adrian
> > >> On Jan 18, 2014 1:52 PM, "Eric Dombroski" <eric@edombroski.com>
> > >> wrote:
> > >>
> > >>> Hello:
> > >>>
> > >>> I believe there is a major performance regression between FreeBSD
> > >>> 9.2-RELEASE and 10.0-RC5 involving the virtio network drivers
> > >>> (vtnet) and
> > >>> handling incoming traffic. Below are the results of some iperf
> > >>> tests and
> > >>> large dd operations over NFS. Write throughput goes from ~40Gbps
> > >>> to
> > >>> ~2.4Gbps from 9.2 to 10.0RC5, and over time the connection
> > >>> becomes
> > >>> unstable
> > >>> ("no buffer space available"), requiring the interface to be
> > >>> taken
> > >>> down/up.
> > >>>
> > >>>
> > >>> These results are on fresh installs of 9.2 and 10.0RC5, no sysctl
> > >>> tweaks
> > >>> on
> > >>> either system.
> > >>>
> > >>> I can't reproduce this using an Intel 1Gbps ethernet through PCIe
> > >>> passthrough, although I suspect the problem manifests itself over
> > >>> 1Gbps
> > >>> speeds anyway.
> > >>>
> > >>> Tests:
> > >>>
> > >>> Client (host):
> > >>> root@gogo:~# uname -a
> > >>> Linux gogo 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64
> > >>> GNU/Linux
> > >>> root@gogo:~# kvm -version
> > >>> QEMU emulator version 1.1.2 (qemu-kvm-1.1.2+dfsg-6, Debian),
> > >>> Copyright
> > >>> (c) 2003-2008 Fabrice Bellard
> > >>> root@gogo:~# lsmod | grep vhost
> > >>> vhost_net 27436 3
> > >>> tun 18337 8 vhost_net
> > >>> macvtap 17633 1 vhost_net
> > >>>
> > >>>
> > >>> Command: iperf -c 192.168.100.x -t 60
> > >>>
> > >>>
> > >>> Server (FreeBSD 9.2 VM):
> > >>>
> > >>> root@umarotest:~ # uname -a
> > >>> FreeBSD umarotest 9.2-RELEASE-p3 FreeBSD 9.2-RELEASE-p3
> > >>> #0: Sat Jan
> > >>> 11 03:25:02 UTC 2014
> > >>> root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
> > >>> amd64
> > >>> root@umarotest:~ # iperf -s
> > >>> ------------------------------------------------------------
> > >>> Server listening on TCP port 5001
> > >>> TCP window size: 64.0 KByte (default)
> > >>> ------------------------------------------------------------
> > >>> [ 4] local 192.168.100.44 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port 58996
> > >>> [ ID] Interval Transfer Bandwidth
> > >>> [ 4] 0.0-60.0 sec 293 GBytes 41.9 Gbits/sec
> > >>> [ 5] local 192.168.100.44 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port 58997
> > >>> [ 5] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec
> > >>> [ 4] local 192.168.100.44 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port 58998
> > >>> [ 4] 0.0-60.0 sec 291 GBytes 41.6 Gbits/sec
> > >>> [ 5] local 192.168.100.44 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port 58999
> > >>> [ 5] 0.0-60.0 sec 297 GBytes 42.6 Gbits/sec
> > >>> [ 4] local 192.168.100.44 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port 59000
> > >>> [ 4] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec
> > >>>
> > >>> While pinging out from the server to the client, I do not
> > >>> get any
> > >>> errors.
> > >>>
> > >>>
> > >>> root@umaro:~ # uname -a FreeBSD umaro 10.0-RC5 FreeBSD
> > >>> 10.0-RC5 #0
> > >>> r260430: Wed Jan 8 05:10:04 UTC 2014
> > >>> root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC
> > >>> amd64
> > >>> root@umaro:~ # iperf -s
> > >>> ------------------------------------------------------------
> > >>> Server listening on TCP port 5001
> > >>> TCP window size: 64.0 KByte (default)
> > >>> ------------------------------------------------------------
> > >>> [ 4] local 192.168.100.5 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port
> > >>> 50264
> > >>> [ ID] Interval Transfer Bandwidth
> > >>> [ 4] 0.0-60.0 sec 16.7 GBytes 2.39 Gbits/sec
> > >>> [ 5] local 192.168.100.5 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port
> > >>> 50265
> > >>> [ 5] 0.0-60.0 sec 18.3 GBytes 2.62 Gbits/sec
> > >>> [ 4] local 192.168.100.5 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port
> > >>> 50266
> > >>> [ 4] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec
> > >>> [ 5] local 192.168.100.5 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port
> > >>> 50267
> > >>> [ 5] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec
> > >>> [ 4] local 192.168.100.5 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port
> > >>> 50268
> > >>> [ 4] 0.0-60.0 sec 16.8 GBytes 2.41 Gbits/sec
> > >>>
> > >>> *** While pinging out from the server to client, frequent
> > >>> "ping:
> > >>> sendto: No space left on device" errors ***
> > >>>
> > >>>
> > >>> After a while, I can also reliably re-produce more
> > >>> egregious "ping:
> > >>> sendto: No buffer space available" errors after doing a large
> > >>> sequential
> > >>> write over NFS:
> > >>>
> > >>> mount -t nfs -o rsize=65536,wsize=65536 192.168.100.5:
> > >>> /storage/shared
> > >>> /mnt/nfs
> > >>> dd if=/dev/zero of=/mnt/nfs/testfile bs=1M count=30000
> > >>>
> > >>> I am going to file a freebsd bug report as well.
> > >>>
> Are you able to test the driver from an up to date head?
> Bryan Venteicher just committed some changes to the driver
> that increase the size of the segments list (# of mbufs in chain)
> and also switches it from using m_collapse() to m_defrag().
> I have no idea if these might affect what you are seeing?
>
>
Yes, please try HEAD if you can, but I think you can take the driver from
HEAD on 10 without too much work. I'll MFC my recent changes to 10 in a
week or so.
As for the "No buffer space available" error, can you try the attached
patch [1]? It is from HEAD, but should apply to 10 as well. I have not been
able to recreate this but I suspect it might fix it.
[1] - http://people.freebsd.org/~bryanv/patches/vtnet_watchdog.patch
rick
>
> > >>> Thanks,
> > >>> Eric
> > >>> _______________________________________________
> > >>> freebsd-stable@freebsd.org mailing list
> > >>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> > >>> To unsubscribe, send any mail to
> > >>> "freebsd-stable-unsubscribe@freebsd.org"
> > >>>
> > > _______________________________________________
> > > freebsd-stable@freebsd.org mailing list
> > > http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> > > To unsubscribe, send any mail to
> > > "freebsd-stable-unsubscribe@freebsd.org"
> > >
> >
> > _______________________________________________
> > freebsd-stable@freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> > To unsubscribe, send any mail to
> > "freebsd-stable-unsubscribe@freebsd.org"
> >
>
--001a11c2e19085471b04f18eaebd
Content-Type: text/x-patch; charset=US-ASCII; name="vtnet_watchdog.patch"
Content-Disposition: attachment; filename="vtnet_watchdog.patch"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_hr8s11pf0
Y29tbWl0IDNmMjU3NTM4Nzk4YjQ4ZGQ3MjdlNzljOGFmNmIyYTExYmQ1MDJlNWIKQXV0aG9yOiBC
cnlhbiBWZW50ZWljaGVyIDxicnlhbnZAZGFlbW9uaW50aGVjbG9zZXQub3JnPgpEYXRlOiAgIFR1
ZSBGZWIgNCAwMDowODo1MiAyMDE0IC0wNjAwCgogICAgU2ltdWxhdGUgYW4gaW50ZXJydXB0IGlm
IHRoZSB3YXRjaGRvZyBkcmFpbmVkIHRoZSBUeCBjb21wbGV0ZSBxdWV1ZQoKZGlmZiAtLWdpdCBh
L3N5cy9kZXYvdmlydGlvL25ldHdvcmsvaWZfdnRuZXQuYyBiL3N5cy9kZXYvdmlydGlvL25ldHdv
cmsvaWZfdnRuZXQuYwppbmRleCBjYzRhZGViLi45M2RlNTA0IDEwMDY0NAotLS0gYS9zeXMvZGV2
L3ZpcnRpby9uZXR3b3JrL2lmX3Z0bmV0LmMKKysrIGIvc3lzL2Rldi92aXJ0aW8vbmV0d29yay9p
Zl92dG5ldC5jCkBAIC0xNDksNyArMTQ5LDcgQEAgc3RhdGljIHZvaWQJdnRuZXRfdHhxX3RxX2Rl
ZmVycmVkKHZvaWQgKiwgaW50KTsKICNlbmRpZgogc3RhdGljIHZvaWQJdnRuZXRfdHhxX3N0YXJ0
KHN0cnVjdCB2dG5ldF90eHEgKik7CiBzdGF0aWMgdm9pZAl2dG5ldF90eHFfdHFfaW50cih2b2lk
ICosIGludCk7Ci1zdGF0aWMgdm9pZAl2dG5ldF90eHFfZW9mKHN0cnVjdCB2dG5ldF90eHEgKik7
CitzdGF0aWMgaW50CXZ0bmV0X3R4cV9lb2Yoc3RydWN0IHZ0bmV0X3R4cSAqKTsKIHN0YXRpYyB2
b2lkCXZ0bmV0X3R4X3ZxX2ludHIodm9pZCAqKTsKIHN0YXRpYyB2b2lkCXZ0bmV0X3R4X3N0YXJ0
X2FsbChzdHJ1Y3QgdnRuZXRfc29mdGMgKik7CiAKQEAgLTIzODUsMTggKzIzODUsMjEgQEAgdnRu
ZXRfdHhxX3RxX2ludHIodm9pZCAqeHR4cSwgaW50IHBlbmRpbmcpCiAJVlRORVRfVFhRX1VOTE9D
Syh0eHEpOwogfQogCi1zdGF0aWMgdm9pZAorc3RhdGljIGludAogdnRuZXRfdHhxX2VvZihzdHJ1
Y3QgdnRuZXRfdHhxICp0eHEpCiB7CiAJc3RydWN0IHZpcnRxdWV1ZSAqdnE7CiAJc3RydWN0IHZ0
bmV0X3R4X2hlYWRlciAqdHhoZHI7CiAJc3RydWN0IG1idWYgKm07CisJaW50IGRlcTsKIAogCXZx
ID0gdHhxLT52dG50eF92cTsKKwlkZXEgPSAwOwogCVZUTkVUX1RYUV9MT0NLX0FTU0VSVCh0eHEp
OwogCiAJd2hpbGUgKCh0eGhkciA9IHZpcnRxdWV1ZV9kZXF1ZXVlKHZxLCBOVUxMKSkgIT0gTlVM
TCkgewogCQltID0gdHhoZHItPnZ0aF9tYnVmOworCQlkZXErKzsKIAogCQl0eHEtPnZ0bnR4X3N0
YXRzLnZ0eHNfb3BhY2tldHMrKzsKIAkJdHhxLT52dG50eF9zdGF0cy52dHhzX29ieXRlcyArPSBt
LT5tX3BrdGhkci5sZW47CkBAIC0yNDA5LDYgKzI0MTIsOCBAQCB2dG5ldF90eHFfZW9mKHN0cnVj
dCB2dG5ldF90eHEgKnR4cSkKIAogCWlmICh2aXJ0cXVldWVfZW1wdHkodnEpKQogCQl0eHEtPnZ0
bnR4X3dhdGNoZG9nID0gMDsKKworCXJldHVybiAoZGVxKTsKIH0KIAogc3RhdGljIHZvaWQKQEAg
LTI1MTIsOCArMjUxNywxMyBAQCB2dG5ldF93YXRjaGRvZyhzdHJ1Y3QgdnRuZXRfdHhxICp0eHEp
CiAJc2MgPSB0eHEtPnZ0bnR4X3NjOwogCiAJVlRORVRfVFhRX0xPQ0sodHhxKTsKLQlpZiAoc2Mt
PnZ0bmV0X2ZsYWdzICYgVlRORVRfRkxBR19FVkVOVF9JRFgpCi0JCXZ0bmV0X3R4cV9lb2YodHhx
KTsKKwlpZiAoc2MtPnZ0bmV0X2ZsYWdzICYgVlRORVRfRkxBR19FVkVOVF9JRFgpIHsKKwkJaWYg
KHR4cS0+dnRudHhfd2F0Y2hkb2cgIT0gMCAmJiB2dG5ldF90eHFfZW9mKHR4cSkgIT0gMCkgewor
CQkJdGFza3F1ZXVlX2VucXVldWUodHhxLT52dG50eF90cSwgJnR4cS0+dnRudHhfaW50cnRhc2sp
OworCQkJVlRORVRfVFhRX1VOTE9DSyh0eHEpOworCQkJcmV0dXJuICgwKTsKKwkJfQorCX0KIAlp
ZiAodHhxLT52dG50eF93YXRjaGRvZyA9PSAwIHx8IC0tdHhxLT52dG50eF93YXRjaGRvZykgewog
CQlWVE5FVF9UWFFfVU5MT0NLKHR4cSk7CiAJCXJldHVybiAoMCk7Cg==
--001a11c2e19085471b04f18eaebd--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGaYwLc-111jEF2moA8da0Nzwno=HVebcbRZ%2B2bBFFSGOkWEoQ>