Date: Tue, 29 Nov 2016 15:44:00 +0000 From: Steven Crangle <Steven@stream-technologies.com> To: Vincenzo Maffione <v.maffione@gmail.com> Cc: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: Looking for some help with netmap/bhyve Message-ID: <DB5PR07MB1685470E0EB0B93A48E7AA229B8D0@DB5PR07MB1685.eurprd07.prod.outlook.com> In-Reply-To: <CA%2B_eA9jW_O_a5uRBAA9XSPspnwATrGhXM3NYusyXxemftn3uZw@mail.gmail.com> References: <DB5PR07MB1685DAA40193595950A464FB9BB40@DB5PR07MB1685.eurprd07.prod.outlook.com> <CA%2B_eA9gggyo_ncSZDribOr%2BsWoFELbPKEeLdaZ8AZwgHAYjcRA@mail.gmail.com> <DB5PR07MB168582129D05D52878D3DCE59B8D0@DB5PR07MB1685.eurprd07.prod.outlook.com>, <CA%2B_eA9jW_O_a5uRBAA9XSPspnwATrGhXM3NYusyXxemftn3uZw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Vincenzo, No problem! We've decided that we will try testing with 11.0 too, as it wou= ld help us to have our test system locked to a specific version like 11.0 a= nd can hopefully help you with testing too. We'll compile netmap from sourc= e as advised, but I was wondering what steps we would take in order to port= the changes to bhyve/ptnetmap from HEAD into 11.0? I've currently only tested with pkt-gen as our bhyve instances are pretty m= inimal right now, below are a few samples of the speeds seen in each of the= different configurations: Using netmap on top of standard tap devices with no vale: vm1: 197.259100 main_thread [2325] 30.977 Kpps (32.435 Kpkts 15.569 Mbps in = 1047062 usec) 1.93 avg_batch 826 min_space 203.306126 main_thread [2325] 32.149 Kpps (32.411 Kpkts 15.557 Mbps in = 1008156 usec) 1.88 avg_batch 866 min_space 204.313055 main_thread [2325] 33.095 Kpps (33.324 Kpkts 15.996 Mbps in = 1006929 usec) 1.98 avg_batch 911 min_space vm2: 245.397418 main_thread [2325] 31.422 Kpps (33.262 Kpkts 15.966 Mbps in = 1058559 usec) 313.79 avg_batch 99999 min_space 246.429810 main_thread [2325] 31.254 Kpps (32.266 Kpkts 15.488 Mbps in = 1032392 usec) 319.47 avg_batch 99999 min_space 251.621436 main_thread [2325] 31.606 Kpps (33.329 Kpkts 15.998 Mbps in = 1054531 usec) 314.42 avg_batch 99999 min_space After then trying to switch to the netmap in userspace/netmap in kernel spa= ce with ptnetmap, I realised I made a silly mistake and built from head ins= tead of the branch ptnet-head... haha So I'm currently doing a quick rebuil= d on the correct branch and will report back with the results from pkt-gen = using just vale, and also with the new ptnetmap changes! Kind Regards ________________________________ [Stream Logo] Steven Crangle Systems Developer | Stream Technologies | Glasgow, UK [http://www.stream-technologies.com/img/phone.png] +44 (0)844 800 8520 | [http://www.stream-technologies.com/img/mouse.png] www.stream-technologies.com<http://www.stream-technologies.com/> ________________________________ From: Vincenzo Maffione <v.maffione@gmail.com> Sent: 29 November 2016 11:52:18 To: Steven Crangle Cc: FreeBSD Net Subject: Re: Looking for some help with netmap/bhyve Hi Steven, Thanks for your testing this in HEAD, good news to know that it works ok! I think there is not going to be much difference between HEAD and 11.0, bec= ause in the end bhyve code didn't really change (for the parts I touched), = and you need to use the latest netmap version (github) anyway, irrespective= of whether you are testing on 10.3, 11.0, head, etc. In other words, you w= ould be testing approximately the same code. Anyway, if you happen to test = 11.0 by chance please let me know. What kind of tests did you perform? netmap applications (e.g. pkt-gen), or = netperf/iperf standard TCP/IP tools? Could you please share the performance number you got for configurations (A= ) and (B) (or just the one you tried)? I think it is useful to compare your results with the ones I collected here= (see the performance evaluation section): https://wiki.freebsd.org/DevSummit/201609?action=3DAttachFile&do=3Dview&tar= get=3D20160923-freebsd-summit-ptnet.pdf Regarding the crashes, thanks for reporting, I would be nice if you could d= escribe in more detail how you trigger them (with some higher likelyhood); = also, compiling the kernel with debug symbols should help in understanding = which part of the code is involved in the crash. Cheers, Vincenzo 2016-11-29 12:26 GMT+01:00 Steven Crangle <Steven@stream-technologies.com<m= ailto:Steven@stream-technologies.com>>: Hi Vincenzo! Thank you so much for your help! With your instructions I've managed to get= a set of bhyves up and running with netmap working properly within them! I= chose to go with the recent version of head instead of 10.3. I will also = try and build a version on 11.0 if I get chance! I will make sure and ask any further questions on the github page for netma= p, but while I'm emailing, I've attached a text file containing a few kerne= l page fault errors I've ran into. They don't happen repeatedly. They seem = to just happen randomly if I try to shut the process down and possibly catc= h netmap in an incorrect state! Figured they might be useful to you. If I f= igure out what the error is myself I will try and help out! And thanks again for your help! Kind Regards ________________________________ [Stream Logo] Steven Crangle Systems Developer | Stream Technologies | Glasgow, UK [http://www.stream-technologies.com/img/phone.png] +44 (0)844 800 8520<tel:%2B44%20%280%29844%20800%208520> | [http://www.stream-technologies.com/img/mouse.png] www.stream-technologies.com<http://www.stream-technologies.com/> ________________________________ From: Vincenzo Maffione <v.maffione@gmail.com<mailto:v.maffione@gmail.com>> Sent: 23 November 2016 08:33:11 To: Steven Crangle Cc: FreeBSD Net Subject: Re: Looking for some help with netmap/bhyve 2016-11-22 11:31 GMT+01:00 Steven Crangle <Steven@stream-technologies.com<m= ailto:Steven@stream-technologies.com>>: Hi, I've recently been trying to boot up several bhyves so that I can test netm= ap communication between instances. The problem is, no matter what configur= ation I try, the guest vm running in bhyve completely hangs and becomes unu= sable as soon as a packet hits the netmap interface. When testing with pkt-= gen, the TX side successfully starts sending packets, but the RX side will = reliable freeze with the only option being killing the bhyve process. The bhyve command used for the above test was: bhyve -c 1 -s 0,hostbridge -s 1,lpc -s 2,virtio-blk,/dev/zvol/zroot/vip= er1vol -s 3,virtio-net,tap0,mac=3D00:01:23:45:67:83 -s 4,virtio-net,tap4= -l com1,/dev/nmdm0A -A -H -P -m 6g viper1 & bhyve -c 1 -s 0,hostbridge -s 1,lpc -s 2,virtio-blk,/dev/zvol/zroot/vip= er2vol -s 3,virtio-net,tap1,mac=3D00:01:23:45:67:84 -s 4,virtio-net,tap5= -l com1,/dev/nmdm1A -A -H -P -m 6g viper2 For this test the host OS was FreeBSD-11.0-p3 and the guest OS was FreeBSD-= 11.0-p3. After failing to get this solution working, I pulled down the source from t= he following url and installed it on the host box: https://svnweb.freebsd.org/socsvn/soc2016/vincenzo/head/ I then ran the following commands to try and bring up the machines using th= e ptnetmap interface (the guest still running 11.0p3) : bhyve -c 1 -s 0,hostbridge -s 1,lpc -s 1:1,virtio-blk,/dev/zvol/zroot/v= iper1vol -s 2:0,virtio-net,tap0,mac=3D00:01:23:45:67:83 -s 2:1,ptnetmap-= memdev -s 2:2,ptnet,vale0:0 -l com1,/dev/nmdm0A -A -H -P -m 6g viper1 & bhyve -c 1 -s 0,hostbridge -s 1,lpc -s 1:1,virtio-blk,/dev/zvol/zroot/v= iper2vol -s 2:0,virtio-net,tap1,mac=3D00:01:23:45:67:84 -s 2:1,ptnetmap-= memdev -s 2:2,ptnet,vale0:1 -l com1,/dev/nmdm1A -A -H -P -m 6g viper2 With the above commands the vm's fail to boot with the following message: ptnet_init: failed to get ptnetmap Output in /var/log/messages seems to just show the ptnetmap driver allocati= ng one RX/TX ring for each vm, while bringing the device up, the device the= n goes down and the above error is seen in the console. Is there something I'm doing wrong with regards to running netmap or ptnetm= ap within a bhyve? Any pointers in the right direction will be much appreci= ated! Kind Regards Steven Hi Steven, The code you are looking at is the final code released by my gsoc 2016 pr= oject at the end of August 2016. However, I've been working on that for a w= hile after the gsoc, so that code is not updated anymore. My modification b= asically involves two subsystems: netmap and bhyve. The updates to netmap are already available in HEAD. The updates to bhyve a= re not yet upstream, as we are in the process to review that with the bhyve= maintainers. Anyway, you need two sources to get the latest code: 1) https://github.com/luigirizzo/netmap for the latest netmap code, that yo= u could also compile as an external kernel module. 2) https://github.com/vmaffione/freebsd to get the updates to bhyve (and vm= m.ko). You can use this as the host system. There are two branches here: pt= net-10.3, and ptnet-head. My original code was developed under FreeBSD 10.3= , so a first possibility is to try this as the host system (using the ptnet= -10.3 branch). The other branch (ptnet-head) contains a porting of the work= to a recent version of HEAD. I would be very glad if you could test the co= de also on FreeBSD 11. We support two combinations of bhyve networking with netmap: (A) virtio-net + netmap: that is something like "-s 2:0,virtio-net,vale0:= 0" in the bhyve command line. (B) ptnet + ptnetmap: that is something like "-s 2:1,ptnetmap-memdev -s 2= :2,ptnet,vale0:0" in the bhyve command line. so you may want to try A first (netmap backend in user-space, slower) and t= hen B (netmap backend in kernel-space, faster). Sorry about the confusion on the code repositories, I'll try also to update= the wiki page (and/or the gsoc svn repo) to reflect these updates. It's pe= rfectly ok for me to discuss these issues here in the ML, however for more = detailed/low-level discussion and support about problems you are running in= to, feel free to open Github issues here https://github.com/luigirizzo/netm= ap. Cheers, Vincenzo ________________________________ [Stream Logo] Steven Crangle Systems Developer | Stream Technologies | Glasgow, UK [http://www.stream-technologies.com/img/phone.png] +44 (0)844 800 8520<tel:%2B44%20%280%29844%20800%208520> | [http://www.stream-technologies.com/img/mouse.png] www.stream-technologies.com<http://www.stream-technologies.com><http://www.= stream-technologies.com/> _______________________________________________ freebsd-net@freebsd.org<mailto:freebsd-net@freebsd.org> mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org<mailt= o:freebsd-net-unsubscribe@freebsd.org>" -- Vincenzo Maffione -- Vincenzo Maffione
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?DB5PR07MB1685470E0EB0B93A48E7AA229B8D0>