Date: Fri, 29 Jan 2016 16:12:55 -0600 From: Xiaoye Sun <Xiaoye.Sun@rice.edu> To: Luigi Rizzo <rizzo@iet.unipi.it> Cc: Pavel Odintsov <pavel.odintsov@gmail.com>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: swaping ring slots between NIC ring and Host ring does not always success Message-ID: <CAJnByziBS8o6LtmpUrUu5xtRUd008Z2hnCsp=WVFv35r2J0rHw@mail.gmail.com> In-Reply-To: <CA%2BhQ2%2BgMWY0eabjHGw0=PJCAkS-wO=RBrN5brSbaqWc3_AOYoQ@mail.gmail.com> References: <CAJnByzj6Dj3vouZ2NbxqvCV-2-7TVtTR4FaWKuCFaaRN2X%2ByAA@mail.gmail.com> <CALgsdbd3XuE3wMYp4ey%2B1aer%2BHSVNojLYoVqwqTBPAXXdf9i%2BQ@mail.gmail.com> <CAJnByzirLXdCe-kwHV2s_E6ytGJG0Dth=0Ms12RrEk7FK_%2B8Og@mail.gmail.com> <CA%2BhQ2%2BgMWY0eabjHGw0=PJCAkS-wO=RBrN5brSbaqWc3_AOYoQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Luigi, Thanks for your advice. I forgot to mention that I use the command "ethtool -L eth1 combined 1" to set the number of rings of the nic to 1. The host also only has one ring. I understand the situation where the first tx ring is full so the bridge will swap the packets to the second tx ring and then the host/nic might drain either rings. But this is not the case in the experiment. Thanks a lot! Xiaoye On Fri, Jan 29, 2016 at 3:45 PM, Luigi Rizzo <rizzo@iet.unipi.it> wrote: > On Fri, Jan 29, 2016 at 1:39 PM, Xiaoye Sun <Xiaoye.Sun@rice.edu> wrote: > > Hi Pavel, > > > > Our code is somewhat complicate. > > So I did a very simple experiment having the same problem so that you > could > > regenerate the problem. > > Xiaoye, > > before doing more experiments can you please clarify > the operating conditions, specifically make sure that > you are only using a single receive and transmit queue > on the NICs in the bridge machine ? > I suspect the problem lies here (it was not clear > from your previous description). > netmap cannot make any guarantee on the transmit > order if you are using multiple transmit queues > (same problem on receive, though it is not relevant > here because almost surely all input traffic goes to > the same queue). > > cheers > luigi > > > In the new experiment, I used a very simple udp sender and udp receiver > > that sends and receives udp packet with seq num. > > > > I put the code here. > > http://www.owlnet.rice.edu/~xs6/receiver.c > > http://www.owlnet.rice.edu/~xs6/sender.c > > > > To repeat the experiment, on machine A, I run the sender program > > (command: sender > > 10.10.10.1 (your receiver IP)). > > On machien B, I run a netmap bridge that swap the slot printer between > the > > host ring and the nic ring (the bridge example in netmap package > > /examples/bridge.c using the command ./bridge netmap:eth1 netmap:eth1) > and > > also run the receiver program. > > > > I am able to see that the receiver print message saying that the new seq > > number is less than the seq number received in the previous round. > > > > Please let me know if you successfully generates the same problem. > > > > Thank! > > Xiaoye > > > > > > > > On Tue, Jan 26, 2016 at 1:53 AM, Pavel Odintsov < > pavel.odintsov@gmail.com> > > wrote: > > > >> Hello! > >> > >> Really useful and detailed research. Thanks a lot! Could you share > >> your validation code? It could be useful for consistency tests for > >> netmap. > >> > >> So I could not help with your original question. Let's wait answer > >> from developers of awesome Netmap :) > >> > >> On Tue, Jan 26, 2016 at 2:55 AM, Xiaoye Sun <Xiaoye.Sun@rice.edu> > wrote: > >> > Hi all, > >> > > >> > I wrote a netmap application that is very similar to bridge.c in > >> master/example > >> > directory of the netmap code. The major difference between my program > and > >> > bridge.c is that my application also creates custom packets that are > >> > directly put into the tx ring of the NIC (like a packet generator with > >> > NIC-Host packet forwarding). Each generated packet has an unique > sequence > >> > number so that the receiver can tell if a packet is lost. > >> > > >> > Like bridge.c, the packet forwarding between the nic and the host uses > >> > 'zerocopy', so that the program only swaps the buf_idx of the slots to > >> > virtually forward the packet to the other end. NS_BUF_CHANGED is set > for > >> > the slots (both rx and tx) whose buf_idx has been changed due to > >> zerocopy. > >> > > >> > I set the number of rings on nic to 1 using the ethtool command, i.e., > >> one > >> > pair of netmap rings for the nic (one for host tx and one for host rx) > >> and > >> > another pair of rings for the host (one for host tx and one for host > rx). > >> > > >> > However, at the receiver side, I found that there is a chance (very > >> little > >> > chance, less than 1% of the swaps) where swapping the buf_idx does not > >> > success. The receiver might get the packet in the buffer swapped out. > For > >> > example, the netmap program wants to swap host slot *SH* with NIC slot > >> *SN* > >> > in order to send the packet in *SH* from host to the receiver; the > >> receiver > >> > might get the packet in *SN* instead. This usually happens to the very > >> end > >> > of the available slots. > >> > > >> > Our experiment is done on *Linux* machine (Linux 3.16.0-4-amd64 #1 SMP > >> > Debian 3.16.7-ckt11-1 (2015-05-24) x86_64 GNU/Linux) with* intel NIC* > >> > (Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection) > using > >> > the *ixgbe* driver. > >> > > >> > To understand this problem, I look into the patch code to the ixgbe > >> driver. > >> > I found that the flag NS_BUF_CHANGED is not used in the linux driver > for > >> > ixgbe. please look the header file /LINUX/ixgbe_netmap_linux.h. In > >> funtions > >> > ixgbe_netmap_txsync and ixgbe_netmap_rxsync, the function call > >> > netmap_reload_map is commented out. However, the ixgbe's driver patch > for > >> > FreeBSD calls the reload function. > >> > > >> > So, I am wondering if this is a known problem when using netmap on > LINUX. > >> > Has anybody found the same problem? > >> > Is netmap_reload_map required in Linux? why it is not called in the > >> driver > >> > patch? > >> > What is the recommended solution for this problem? > >> > > >> > Thanks! > >> > > >> > Best, > >> > Xiaoye > >> > _______________________________________________ > >> > freebsd-net@freebsd.org mailing list > >> > https://lists.freebsd.org/mailman/listinfo/freebsd-net > >> > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org > " > >> > >> > >> > >> -- > >> Sincerely yours, Pavel Odintsov > >> > >> > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > > -- > -----------------------------------------+------------------------------- > Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione > http://www.iet.unipi.it/~luigi/ . Universita` di Pisa > TEL +39-050-2217533 . via Diotisalvi 2 > Mobile +39-338-6809875 . 56122 PISA (Italy) > -----------------------------------------+------------------------------- > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJnByziBS8o6LtmpUrUu5xtRUd008Z2hnCsp=WVFv35r2J0rHw>