Date: Mon, 27 Jan 2014 18:14:50 +0800 From: Wang Weidong <wangweidong1@huawei.com> To: Vincenzo Maffione <v.maffione@gmail.com> Cc: =?ISO-8859-1?Q?facolt=E0?= <giuseppe.lettieri73@gmail.com>, Giuseppe Lettieri <g.lettieri@iet.unipi.it>, Luigi Rizzo <rizzo@iet.unipi.it>, net@freebsd.org Subject: Re: netmap: I got some troubles with netmap Message-ID: <52E6319A.8070601@huawei.com> In-Reply-To: <CA%2B_eA9j0thuO6sALz%2Ben=xVUj%2BL=UaBkEvYjQ3THCLYHMQsOjA@mail.gmail.com> References: <52D74E15.1040909@huawei.com> <CA%2BhQ2%2BjBhSyHwFsFo%2BzH-EuJEkKEcyc6YBH%2BfnEHi=Y27FyWyQ@mail.gmail.com> <92C7725B-B30A-4A19-925A-A93A2489A525@iet.unipi.it> <52D8A5E1.9020408@huawei.com> <52DD1914.7090506@iet.unipi.it> <52E1E272.8060009@huawei.com> <CA%2B_eA9hOzQiOWKvHOiKjY4kjxmerMWp=MhtF_vbr8t-q4V732g@mail.gmail.com> <52E4BC38.7040407@huawei.com> <CA%2B_eA9j0thuO6sALz%2Ben=xVUj%2BL=UaBkEvYjQ3THCLYHMQsOjA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2014/1/27 17:37, Vincenzo Maffione wrote: > > > > 2014/1/26 Wang Weidong <wangweidong1@huawei.com <mailto:wangweidong1@huawei.com>> > > On 2014/1/24 22:56, Vincenzo Maffione wrote: > > > > > > > > 2014/1/24 Wang Weidong <wangweidong1@huawei.com <mailto:wangweidong1@huawei.com> <mailto:wangweidong1@huawei.com <mailto:wangweidong1@huawei.com>>> > > > > On 2014/1/20 20:39, Giuseppe Lettieri wrote: > > > Hi Wang, > > > > > > OK, you are using the netmap support in the upstream qemu git. That does not yet include all our modifications, some of which are very important for high throughput with VALE. In particular, the upstream qemu does not include the batching improvements in the frontend/backend interface, and it does not include the "map ring" optimization of the e1000 frontend. Please find attached a gzipped patch that contains all of our qemu code. The patch is against the latest upstream master (commit 1cf892ca). > > > > > > Please ./configure the patched qemu with the following option, in addition to any other option you may need: > > > > > > --enable-e1000-paravirt --enable-netmap \ > > > --extra-cflags=-I/path/to/netmap/sys/directory > > > > > > Note that --enable-e1000-paravirt is needed to enable the "map ring" optimization in the e1000 frontend, even if you are not going to use the e1000-paravirt device. > > > > > > Now you should be able to rerun your tests. I am also attaching a README file that describes some more tests you may want to run. > > > > > > > Hello, > > > > > > Yes, I patch the qemu-netmap-bc767e701.patch to the qemu, download the 20131019-tinycore-netmap.hdd. > > And I do some test that: > > > > 1. I use the bridge below: > > qemu-system-x86_64 -m 2048 -boot c -net nic -net bridge,br=br1 -hda /home/wwd/tinycores/20131019-tinycore-netmap.hdd -enable-kvm -vnc :0 > > test between two vms. > > br1 without device. > > Use pktgen, I got the 237.95 kpps. > > Use the netserver/netperf I got the speed 1037M bits/sec with TCP_STREAM. The max speed is up to 1621M. > > Use the netserver/netperf I got the speed 3296/s with TCP_RR > > Use the netserver/netperf I got the speed 234M/86M bits/sec with UDP_STREAM > > > > When I add a device from host to the br1, the speed is 159.86 kpps. > > Use the netserver/netperf I got the speed 720M bits/sec with TCP_STREAM. The max speed is up to 1000M. > > Use the netserver/netperf I got the speed 3556/s with TCP_RR > > Use the netserver/netperf I got the speed 181M/181M bits/sec with UDP_STREAM > > > > What do you think of these data? > > > > > > You are using the old/deprecated QEMU command line syntax (-net), and therefore honestly It's not clear to me what kind of network configuration you are running. > > > > Please use our scripts "launch-qemu.sh", "prep-taps.sh", according to what described in the README.images file (attached). > > Alternatively, use the syntax like in the following examples > > > > (#1) qemu-system-x86_64 archdisk.qcow -enable-kvm -device virtio-net-pci,netdev=mynet -netdev tap,ifname=tap01,id=mynet,script=no,downscript=no -smp 2 > > (#2) qemu-system-x86_64 archdisk.qcow -enable-kvm -device e1000,mitigation=off,mac=00:AA:BB:CC:DD:01,netdev=mynet -netdev netmap,ifname=vale0:01,id=mynet -smp 2 > > > Here I use the 20131019-tinycore-netmap.hdd (download from the http://info.iet.unipi.it/~luigi/netmap/) instead archdisk.qcow with #2. > And I can't do the "cpufreq-set -g performance # on linux" as the README.image. > > > This is our fault, thanks for reporting (probably the tinycore kernel doesn't include the cpufreq governors). However, if you use a different linux O.S. as host machine you should be able to cpufreq-set -gperformance on the host machine, while keep using tinycore into the vms. > > Although, I use the pkt-gen test the vms, I got the tx speed is ~3Mpps while the other vm's rx speed is only 1.44Mpps. > > Is it right? I can't get ~4Mpps is the reason that I can't set the "CPU power saving". > > > It can be OK, depending on your machine and maybe on the CPU power saving.k On my machine > - Processor: Intel i7-3770K CPU @ 3.50GHz (8 cores) > - Memory @ 1333 MHz > - Host O.S.: Archlinux with Linux 3.12 > I get: > > - 3.9 Mpps on TX, 3.5 Mpps RX when guests are given 1 vCPU each > - 4.5 Mpps on TX, 3.2 Mpps RX when guests are given 2 vCPU each. > > > > > > so that it's clear to us what network frontend (e.g. emulated NIC) and network backend (e.g. netmap, tap, vde, ecc..) you are using. > > In example #1 we are using virtio-net as frontend and tap as backend, while in example #2 we are using e1000 as frontend and netmap as backend. > > Also consider giving more than one core (e.g. -smp 2) to each guest, to mitigate receiver livelock problems. > > > > > > > > 2. I use the vale below: > > qemu-system-x86_64 -m 2048 -boot c -net nic -net netmap,vale0:0 -hda /home/wwd/tinycores/20131019-tinycore-netmap.hdd -enable-kvm -vnc :0 > > > > Same for here, it's not clear what you are using. I guess each guest has an e1000 device and is connected to a different port of the same vale switch (e.g. vale0:0 and vale0:1)? > > > > Test with 2 vms from the same host > > vale0 without device. > > I use the pkt-gen, the speed is 938 Kpps > > > > > > You should get ~4Mpps with e1000 frontend + netmap backend on a reasonably good machine. Make sure you have ./configure'd QEMU with --enable-e1000-paravirt. > > > > > > I use netperf -H 10.0.0.2 -t UDP_STREAM, I got the speed is 195M/195M, then add -- -m 8, I only got 1.07M/1.07M. > > When use the smaller msg size, the speed will smaller? > > > > > > If you use e1000 with netperf (without pkt-gen) your performance is doomed to be horrible. Use e1000-paravirt (as a frontend) instead if you are interested in netperf experiment. > > Also consider that the point in using the "-- -m8" options is experimenting high packet rates, so what you should measure here is not the througput in Mbps, but the packet rate: netperf reports the number of packets sent and received, so you can obtain the packet rate by dividing by the running time. > > The throughput in Mbps is uninteresting, if you want high bulk throughput you just don't use "-- -m 8", but leave the defaults. > > Using virtio-net in this case will help because of the TSO offloadings. > > > Here, I am a little interested in netperf. So I did that: > qemu-system-x86_64 20131019-tinycore-netmap.hdd -enable-kvm -device *e1000-paravirt*,mitigation=off,mac=00:AA:BB:CC:DD:01,netdev=mynet -netdev netmap,ifname=vale0:01,id=mynet -smp 2 -m 2048 -vnc :0 > qemu-system-x86_64 20131019-tinycore-netmap.hdd -enable-kvm -device *e1000-paravirt*,mitigation=off,mac=00:AA:BB:CC:DD:02,netdev=mynet -netdev netmap,ifname=vale0:02,id=mynet -smp 2 -m 2048 -vnc :1 > I think this cmd is error. because after I set ipv4 addrs to the device, But I find two vms can't communicate with each other. > > > This is the command that I use on my host machine (can be generated using the script launch-qemu.sh) > > qemu-system-x86_64 20131019-tinycore-netmap.hdd -enable-kvm -device e1000-paravirt,mitigation=off,ioeventfd=on,v1000=off,mac=00:AA:BB:CC:DD:01,netdev=mynet -netdev netmap,ifname=vale0:01,id=mynet -smp 2 -m 3G -vga std > > and it works to me. It may be that your problem is due to the fact that you are using a netmap version which mismatches with the QEMU version. > Please use the netmap version I attached. > > Also try "virtio-net-pci" as frontend, like in the following > > qemu-system-x86_64 20131019-tinycore-netmap.hdd -enable-kvm -device virtio-net-pci,mrg_rxbuf=on,ioeventfd=on,mac=00:AA:BB:CC:DD:01,netdev=mynet -netdev netmap,ifname=vale0:01,id=mynet -smp 2 -m 3G -vga std > > Consider that: > - e1000-paravirt is optimized for high packet rate (e.g. UDP_STREAM -- -m8) > - virtio-net-pci is optimized for TCP_STREAM and TCP_RR. > and you should be able to deduce this from the numbers you measure. > > > Cheers, > Vincenzo > > Nice, I will test again. Regards, Wang > > > Thanks, > Wang > > > cheers > > Vincenzo > > > > > > > > with vale-ctl -a vale0:eth2, > > use pkt-gen, the speed is 928 Kpps > > I use netperf -H 10.0.0.2 -t UDP_STREAM, I got the speed is 209M/208M, then add -- -m 8, I only got 1.06M/1.06M. > > > > with vale-ctl -h vale0:eth2, > > use pkt-gen, the speed is 928 Kpps > > I use netperf -H 10.0.0.2 -t UDP_STREAM, I got the speed is 192M/192M, then add -- -m 8, I only got 1.06M/1.06M. > > > > Test with 2 vms form two host, > > I only can test it by vale-ctl -h vale0:eth2 and set eth2 into promisc > > use pkt-gen with the default params, the speed is about 750 Kpps > > use netperf -H 10.0.0.2 -t UDP_STREAM, I got the speed is 160M/160M > > Is this right? > > > > > > 3. I can't use the l2 utils. > > When I do the "sudo l2open -t eth0 l2recv[l2send], I got that "l2open ioctl(TUNSETIFF...): Invalid argument" > > and "use l2open -r eth0 l2recv", wait a moment (only several seconds), I got the result: > > TEST-RESULT: 0.901 kpps 1pkts > > select/read=100.00 err=0 > > > > And I can't find the l2 utils from the net? Is it implemented by your team? > > > > All of them is tested on vms. > > > > Cheers. > > Wang > > > > > > > > > > Cheers, > > > Giuseppe > > > > > > Il 17/01/2014 04:39, Wang Weidong ha scritto: > > >> On 2014/1/16 18:24, facoltà wrote: > > [...] > > >> > > >> > > > > > > > > > > > > > > > > > > -- > > Vincenzo Maffione > > > > > > -- > Vincenzo Maffione
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?52E6319A.8070601>