From owner-freebsd-net@freebsd.org Fri Jan 29 21:39:23 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9BF68A7360B for ; Fri, 29 Jan 2016 21:39:23 +0000 (UTC) (envelope-from sunxiaoye07@gmail.com) Received: from mail-oi0-x22d.google.com (mail-oi0-x22d.google.com [IPv6:2607:f8b0:4003:c06::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5FBCF1595 for ; Fri, 29 Jan 2016 21:39:23 +0000 (UTC) (envelope-from sunxiaoye07@gmail.com) Received: by mail-oi0-x22d.google.com with SMTP id r14so55721918oie.0 for ; Fri, 29 Jan 2016 13:39:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=B2q0vwnGEVV1yLjc+KFKi8TsUtPW0ShupsnDyoUQB70=; b=cDtUcBwLWRpwHiRTqxMkB4iZtLbsyshAuqnqRQK13sTFB2kvLSOd+QmG/0n20ZOlZy N5hbtFv6Ho7KuZzHGEP0PjCpn1tn60z6KdTCTPUj/1WAdw+/PjNkHZasWwnsXzuRCKeG CQKCI8xuIJodWuq26CcofNNLG11/TKnLdO0KPAqAutI5FOagrd6ioWMP1W3i+rg1Eygg d5R9ts4E/SlheLyz7GNeMlF+/bky3EU9eR/Ua/rZ9l6LW+Z5rWhD7aef+wKse9XdGBtR 1J/CMWg8K3yBy3WOIUxku7qLRHeMkekbPeMlg+DrrR0/NClLnttqVjmld4aMPVT2e6gy CZXA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rice-edu.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=B2q0vwnGEVV1yLjc+KFKi8TsUtPW0ShupsnDyoUQB70=; b=aNnIpTsl+cgmWFrnUAi9oGFaKWTJT/hK0kCs4p58dRTJQvT84JBQFJB1x6G0XQLq6O c6tEA24aUYMDFMDzqRUCCENkInAjiPkjB93/tMtOcWTnva0o+Z2583TApIXjWlzSViIw a3FTFNogUUOUl2sbNUBwOWknKF23TOTvlaePAymI3DcyEXCYvy2wGTQK7uE1OOx8l/VG Agsld5s4CF2dqSe8Yk7rGCqwacpzRQNCoknRgl1yyI4hEG2XS6AqeEDs882SpkQhx6me Pvk9fDr1n2eQ/Synn49At3DzG936iC5zo2bspcSYUCoq5c/gFo/XsQeF/r6XpwTJXDOO KkBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=B2q0vwnGEVV1yLjc+KFKi8TsUtPW0ShupsnDyoUQB70=; b=QL20UObhepxQ/Wx3dmIjbV3aVvdJwWFW6BXFy8JlvAbonu1gHMryn6r/wPkj9c0weE wYfNNKcSlHkim8M5tZISXJNoptNbXfa2VywFpYQa5DqOgMhBlrwY7HxPTedqbfcMIz8w mYww0/fm39CUdS6I//uA/ur+PFTWz7YS6kKQ3zR2sp8NeZmX8ennrL5udyRtCdh29Q8T 23whvqDPfX0YCxlfEC17BxP74i0Jqc3pOcXDfc/yHL729UFdVe8oj2f5dHouRGcBNgiN qT7snn6/gXIMuSSoDzPm0ZVjKAp+MWTI8QBKpncj9IgNw1PhMIeuiH4pM99QTJHO2K9g S8yw== X-Gm-Message-State: AG10YOSb2bLujEv/eyqxhqxl+VjYvZJQwGTeN95wh8lF4mc3ZyDFj6HSZUbxvT5qeeJVJSzviEqQX8UHFIaizg== MIME-Version: 1.0 X-Received: by 10.202.108.69 with SMTP id h66mr7271178oic.67.1454103562609; Fri, 29 Jan 2016 13:39:22 -0800 (PST) Sender: sunxiaoye07@gmail.com Received: by 10.202.223.5 with HTTP; Fri, 29 Jan 2016 13:39:22 -0800 (PST) In-Reply-To: References: Date: Fri, 29 Jan 2016 15:39:22 -0600 X-Google-Sender-Auth: OcC115aT0Nfxlebwvbv7RWMaaYA Message-ID: Subject: Re: swaping ring slots between NIC ring and Host ring does not always success From: Xiaoye Sun To: Pavel Odintsov Cc: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jan 2016 21:39:23 -0000 Hi Pavel, Our code is somewhat complicate. So I did a very simple experiment having the same problem so that you could regenerate the problem. In the new experiment, I used a very simple udp sender and udp receiver that sends and receives udp packet with seq num. I put the code here. http://www.owlnet.rice.edu/~xs6/receiver.c http://www.owlnet.rice.edu/~xs6/sender.c To repeat the experiment, on machine A, I run the sender program (command: sender 10.10.10.1 (your receiver IP)). On machien B, I run a netmap bridge that swap the slot printer between the host ring and the nic ring (the bridge example in netmap package /examples/bridge.c using the command ./bridge netmap:eth1 netmap:eth1) and also run the receiver program. I am able to see that the receiver print message saying that the new seq number is less than the seq number received in the previous round. Please let me know if you successfully generates the same problem. Thank! Xiaoye On Tue, Jan 26, 2016 at 1:53 AM, Pavel Odintsov wrote: > Hello! > > Really useful and detailed research. Thanks a lot! Could you share > your validation code? It could be useful for consistency tests for > netmap. > > So I could not help with your original question. Let's wait answer > from developers of awesome Netmap :) > > On Tue, Jan 26, 2016 at 2:55 AM, Xiaoye Sun wrote: > > Hi all, > > > > I wrote a netmap application that is very similar to bridge.c in > master/example > > directory of the netmap code. The major difference between my program and > > bridge.c is that my application also creates custom packets that are > > directly put into the tx ring of the NIC (like a packet generator with > > NIC-Host packet forwarding). Each generated packet has an unique sequence > > number so that the receiver can tell if a packet is lost. > > > > Like bridge.c, the packet forwarding between the nic and the host uses > > 'zerocopy', so that the program only swaps the buf_idx of the slots to > > virtually forward the packet to the other end. NS_BUF_CHANGED is set for > > the slots (both rx and tx) whose buf_idx has been changed due to > zerocopy. > > > > I set the number of rings on nic to 1 using the ethtool command, i.e., > one > > pair of netmap rings for the nic (one for host tx and one for host rx) > and > > another pair of rings for the host (one for host tx and one for host rx). > > > > However, at the receiver side, I found that there is a chance (very > little > > chance, less than 1% of the swaps) where swapping the buf_idx does not > > success. The receiver might get the packet in the buffer swapped out. For > > example, the netmap program wants to swap host slot *SH* with NIC slot > *SN* > > in order to send the packet in *SH* from host to the receiver; the > receiver > > might get the packet in *SN* instead. This usually happens to the very > end > > of the available slots. > > > > Our experiment is done on *Linux* machine (Linux 3.16.0-4-amd64 #1 SMP > > Debian 3.16.7-ckt11-1 (2015-05-24) x86_64 GNU/Linux) with* intel NIC* > > (Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection) using > > the *ixgbe* driver. > > > > To understand this problem, I look into the patch code to the ixgbe > driver. > > I found that the flag NS_BUF_CHANGED is not used in the linux driver for > > ixgbe. please look the header file /LINUX/ixgbe_netmap_linux.h. In > funtions > > ixgbe_netmap_txsync and ixgbe_netmap_rxsync, the function call > > netmap_reload_map is commented out. However, the ixgbe's driver patch for > > FreeBSD calls the reload function. > > > > So, I am wondering if this is a known problem when using netmap on LINUX. > > Has anybody found the same problem? > > Is netmap_reload_map required in Linux? why it is not called in the > driver > > patch? > > What is the recommended solution for this problem? > > > > Thanks! > > > > Best, > > Xiaoye > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > > -- > Sincerely yours, Pavel Odintsov > >