From owner-freebsd-stable@FreeBSD.ORG Mon Feb 3 23:34:46 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ACFE497C; Mon, 3 Feb 2014 23:34:46 +0000 (UTC) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id A7B4813DA; Mon, 3 Feb 2014 23:34:45 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: X-IronPort-AV: E=Sophos;i="4.95,775,1384318800"; d="scan'208";a="93330176" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 03 Feb 2014 18:34:44 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id C171DB3F45; Mon, 3 Feb 2014 18:34:44 -0500 (EST) Date: Mon, 3 Feb 2014 18:34:44 -0500 (EST) From: Rick Macklem To: Alfred Perlstein Message-ID: <449096679.2303214.1391470484784.JavaMail.root@uoguelph.ca> In-Reply-To: <52DBE19C.4040101@freebsd.org> Subject: Re: Major performance/stability regression in virtio network drivers between 9.2-RELEASE and 10.0-RC5 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.209] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: Bryan Venteicher , freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Feb 2014 23:34:46 -0000 Alfred Perlstein wrote: > What does vmstat say about memory zones? I think vmstat -m/vmstat -z > and also netstat -m? > > Are you set with the same mbufs on both 9 and 10? > > -Alfred > > > On 1/18/14 1:32 PM, Eric Dombroski wrote: > > Adrian: > > > > Yes, no change. > > > > -Eric > > > > > > > > > > > > On Sat, Jan 18, 2014 at 4:06 PM, Adrian Chadd > > wrote: > > > >> Hi, > >> > >> Have you tried disabling tso? > >> > >> Adrian > >> On Jan 18, 2014 1:52 PM, "Eric Dombroski" > >> wrote: > >> > >>> Hello: > >>> > >>> I believe there is a major performance regression between FreeBSD > >>> 9.2-RELEASE and 10.0-RC5 involving the virtio network drivers > >>> (vtnet) and > >>> handling incoming traffic. Below are the results of some iperf > >>> tests and > >>> large dd operations over NFS. Write throughput goes from ~40Gbps > >>> to > >>> ~2.4Gbps from 9.2 to 10.0RC5, and over time the connection > >>> becomes > >>> unstable > >>> ("no buffer space available"), requiring the interface to be > >>> taken > >>> down/up. > >>> > >>> > >>> These results are on fresh installs of 9.2 and 10.0RC5, no sysctl > >>> tweaks > >>> on > >>> either system. > >>> > >>> I can't reproduce this using an Intel 1Gbps ethernet through PCIe > >>> passthrough, although I suspect the problem manifests itself over > >>> 1Gbps > >>> speeds anyway. > >>> > >>> Tests: > >>> > >>> Client (host): > >>> root@gogo:~# uname -a > >>> Linux gogo 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 > >>> GNU/Linux > >>> root@gogo:~# kvm -version > >>> QEMU emulator version 1.1.2 (qemu-kvm-1.1.2+dfsg-6, Debian), > >>> Copyright > >>> (c) 2003-2008 Fabrice Bellard > >>> root@gogo:~# lsmod | grep vhost > >>> vhost_net 27436 3 > >>> tun 18337 8 vhost_net > >>> macvtap 17633 1 vhost_net > >>> > >>> > >>> Command: iperf -c 192.168.100.x -t 60 > >>> > >>> > >>> Server (FreeBSD 9.2 VM): > >>> > >>> root@umarotest:~ # uname -a > >>> FreeBSD umarotest 9.2-RELEASE-p3 FreeBSD 9.2-RELEASE-p3 > >>> #0: Sat Jan > >>> 11 03:25:02 UTC 2014 > >>> root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC > >>> amd64 > >>> root@umarotest:~ # iperf -s > >>> ------------------------------------------------------------ > >>> Server listening on TCP port 5001 > >>> TCP window size: 64.0 KByte (default) > >>> ------------------------------------------------------------ > >>> [ 4] local 192.168.100.44 port 5001 connected with > >>> 192.168.100.1 > >>> port 58996 > >>> [ ID] Interval Transfer Bandwidth > >>> [ 4] 0.0-60.0 sec 293 GBytes 41.9 Gbits/sec > >>> [ 5] local 192.168.100.44 port 5001 connected with > >>> 192.168.100.1 > >>> port 58997 > >>> [ 5] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec > >>> [ 4] local 192.168.100.44 port 5001 connected with > >>> 192.168.100.1 > >>> port 58998 > >>> [ 4] 0.0-60.0 sec 291 GBytes 41.6 Gbits/sec > >>> [ 5] local 192.168.100.44 port 5001 connected with > >>> 192.168.100.1 > >>> port 58999 > >>> [ 5] 0.0-60.0 sec 297 GBytes 42.6 Gbits/sec > >>> [ 4] local 192.168.100.44 port 5001 connected with > >>> 192.168.100.1 > >>> port 59000 > >>> [ 4] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec > >>> > >>> While pinging out from the server to the client, I do not > >>> get any > >>> errors. > >>> > >>> > >>> root@umaro:~ # uname -a FreeBSD umaro 10.0-RC5 FreeBSD > >>> 10.0-RC5 #0 > >>> r260430: Wed Jan 8 05:10:04 UTC 2014 > >>> root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC > >>> amd64 > >>> root@umaro:~ # iperf -s > >>> ------------------------------------------------------------ > >>> Server listening on TCP port 5001 > >>> TCP window size: 64.0 KByte (default) > >>> ------------------------------------------------------------ > >>> [ 4] local 192.168.100.5 port 5001 connected with > >>> 192.168.100.1 > >>> port > >>> 50264 > >>> [ ID] Interval Transfer Bandwidth > >>> [ 4] 0.0-60.0 sec 16.7 GBytes 2.39 Gbits/sec > >>> [ 5] local 192.168.100.5 port 5001 connected with > >>> 192.168.100.1 > >>> port > >>> 50265 > >>> [ 5] 0.0-60.0 sec 18.3 GBytes 2.62 Gbits/sec > >>> [ 4] local 192.168.100.5 port 5001 connected with > >>> 192.168.100.1 > >>> port > >>> 50266 > >>> [ 4] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec > >>> [ 5] local 192.168.100.5 port 5001 connected with > >>> 192.168.100.1 > >>> port > >>> 50267 > >>> [ 5] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec > >>> [ 4] local 192.168.100.5 port 5001 connected with > >>> 192.168.100.1 > >>> port > >>> 50268 > >>> [ 4] 0.0-60.0 sec 16.8 GBytes 2.41 Gbits/sec > >>> > >>> *** While pinging out from the server to client, frequent > >>> "ping: > >>> sendto: No space left on device" errors *** > >>> > >>> > >>> After a while, I can also reliably re-produce more > >>> egregious "ping: > >>> sendto: No buffer space available" errors after doing a large > >>> sequential > >>> write over NFS: > >>> > >>> mount -t nfs -o rsize=65536,wsize=65536 192.168.100.5: > >>> /storage/shared > >>> /mnt/nfs > >>> dd if=/dev/zero of=/mnt/nfs/testfile bs=1M count=30000 > >>> > >>> I am going to file a freebsd bug report as well. > >>> Are you able to test the driver from an up to date head? Bryan Venteicher just committed some changes to the driver that increase the size of the segments list (# of mbufs in chain) and also switches it from using m_collapse() to m_defrag(). I have no idea if these might affect what you are seeing? rick > >>> Thanks, > >>> Eric > >>> _______________________________________________ > >>> freebsd-stable@freebsd.org mailing list > >>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable > >>> To unsubscribe, send any mail to > >>> "freebsd-stable-unsubscribe@freebsd.org" > >>> > > _______________________________________________ > > freebsd-stable@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > > To unsubscribe, send any mail to > > "freebsd-stable-unsubscribe@freebsd.org" > > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to > "freebsd-stable-unsubscribe@freebsd.org" >