From owner-freebsd-net@freebsd.org Fri Jan 29 21:46:03 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 29187A73861 for ; Fri, 29 Jan 2016 21:46:03 +0000 (UTC) (envelope-from sunxiaoye07@gmail.com) Received: from mail-oi0-x22b.google.com (mail-oi0-x22b.google.com [IPv6:2607:f8b0:4003:c06::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DD4721C39 for ; Fri, 29 Jan 2016 21:46:02 +0000 (UTC) (envelope-from sunxiaoye07@gmail.com) Received: by mail-oi0-x22b.google.com with SMTP id p187so55928786oia.2 for ; Fri, 29 Jan 2016 13:46:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=7Zu7ZVIkaB/Bcq+06WPBh3zhipkHUdDyGLIh1ADvGSU=; b=EWYosyvn75SsA30IoVVaSMFW8gVzSmevFpozjZNNlsfJr7C3p3uAaFjc0ltdPk/Pk3 g0MO30cJzNx4KvMDrTeLvRFZ+4t4OEvqgj/H3y8LCYHyu5bhG+n19S1nYFSpLp0JIqFM XioXV0AY8Q1GouCNgZ01Yydiz5igPT6K7doRXAC3En1hnHQEwZ9fTHLwGKaS/1EsebrQ anabmr8WbSlxxnbux8iK/aHZDbrJ0TKuPHN2fGsocQRkXUTJ+3Zmzop9EyrEoSHdQOJO 4RoYMWK6dlp6lygvRomCdMT5PZ+14YU1yqiK+ysBwfOVa/Nr2sN0Ih5WgSIDsYmOf95p ncPQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rice-edu.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=7Zu7ZVIkaB/Bcq+06WPBh3zhipkHUdDyGLIh1ADvGSU=; b=paHqMA5WPw3rODLfaJWP1vGNcxTO1niaEYFkEGa9t3ZGzr4xXqyt7u9757RkWJILnX fVDZlymRHBKTQ6n07mH/6YRPOusUjy5QWxzC6y+95j1vluebI5IYaZd5lHMtJC1PkS2i D2rEvCaX7dKYD4J0wDuddccQgc8FQRQt+3qZvyUppDvT7yGB2JTBLyyNBY5Vo0rXIleH Cbj3NHJSjocP96FLtZ39CejQKIjWH4zZafkoWpY/1UUEH+Pgj+4/7iaQSjoMIquJev4s X2rRgIpuijKCT/ITwiZmKvoyeCTLCQ6b4hEZhJLhsUFKEez1L2R+Nf9jzuUDdOLPmKyU 8wHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=7Zu7ZVIkaB/Bcq+06WPBh3zhipkHUdDyGLIh1ADvGSU=; b=FkUyfpdVr1UNSVIVNxcMC+fOCPBCdEtv8z7s3AipxJjcgL3y1Li8aoVxKj4CC94fpz 4m3aEmnXVorkXMTdCDTEM3ZFDmLXqWBt/Xl0y/8sBRK+0VlZWerdjZGVt02KT9tre/HX LKuy+3WskLVIyWqERyr77eN0IcukZEtx6JmkxSgcJIycp7XjXRZblyHbIMMDFecXdofR MNzZtfj6ZHTmnMxK4aS55fyK5HeYGNRt6fO2BTacNrliSkUbAU2Qqxi6YerjM/kMkuND EZylIDBeIE4uu27ZProMChBGogAcmfQV3h//+Z49TxFg3X1pefR/uwIREiL6Xej48L88 7ldg== X-Gm-Message-State: AG10YOT0NGJe+JMl8KawHU3VEUNsnIPZNZiYRdCZpB8qUYZTSNtjvyFmxZZnhEqubktzfLuqB6efEd7OnN9sOQ== MIME-Version: 1.0 X-Received: by 10.202.108.69 with SMTP id h66mr7289378oic.67.1454103962222; Fri, 29 Jan 2016 13:46:02 -0800 (PST) Sender: sunxiaoye07@gmail.com Received: by 10.202.223.5 with HTTP; Fri, 29 Jan 2016 13:46:02 -0800 (PST) In-Reply-To: References: Date: Fri, 29 Jan 2016 15:46:02 -0600 X-Google-Sender-Auth: BN-tHkhRZR2x4HEb5yexkbwEQkA Message-ID: Subject: Re: swaping ring slots between NIC ring and Host ring does not always success From: Xiaoye Sun To: Luigi Rizzo Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jan 2016 21:46:03 -0000 Hi Luigi, Thanks for your reply. I try to disable iommu (add a line of "GRUB_CMDLINE_LINUX="iommu=off amd_iommu=off intel_iommu=off"" in /etc/default/grub and also add memory barrier (__asm__ volatile("mfence" : : : "memory");) before it updates the slot pointer cur and head. However, the problem still there. In my reply to Pavel, I post how to generate the problem using very simple code and exmperiment. Could you please also try it on Linux and see if you could generate the same problem? Thanks! Best, Xiaoye On Tue, Jan 26, 2016 at 2:01 AM, Luigi Rizzo wrote: > Hi, > in FreeBSD, netmap_reload_map() is basically a NOP unless the NIC uses > bounce buffers (in which case it is pointless to use netmap, and I am > not even sure the code works), or there is an iommu which may need to > be reprogrammed (but in that case you don't want to do it dynamically > on individual buffers, but once for all when the device is opened.) > > This said, can you check that the iommu is disabled in the bios and > re-run the test ? > > This said I am not sure that the problem is the iommu -- a wrong entry > should result in bogus content, certainly not the one before the swap. > I suspect some kind of race, possibly missing memory barriers. > > cheers > luigi > > > On Mon, Jan 25, 2016 at 3:55 PM, Xiaoye Sun wrote: > > Hi all, > > > > I wrote a netmap application that is very similar to bridge.c in > master/example > > directory of the netmap code. The major difference between my program and > > bridge.c is that my application also creates custom packets that are > > directly put into the tx ring of the NIC (like a packet generator with > > NIC-Host packet forwarding). Each generated packet has an unique sequence > > number so that the receiver can tell if a packet is lost. > > > > Like bridge.c, the packet forwarding between the nic and the host uses > > 'zerocopy', so that the program only swaps the buf_idx of the slots to > > virtually forward the packet to the other end. NS_BUF_CHANGED is set for > > the slots (both rx and tx) whose buf_idx has been changed due to > zerocopy. > > > > I set the number of rings on nic to 1 using the ethtool command, i.e., > one > > pair of netmap rings for the nic (one for host tx and one for host rx) > and > > another pair of rings for the host (one for host tx and one for host rx). > > > > However, at the receiver side, I found that there is a chance (very > little > > chance, less than 1% of the swaps) where swapping the buf_idx does not > > success. The receiver might get the packet in the buffer swapped out. For > > example, the netmap program wants to swap host slot *SH* with NIC slot > *SN* > > in order to send the packet in *SH* from host to the receiver; the > receiver > > might get the packet in *SN* instead. This usually happens to the very > end > > of the available slots. > > > > Our experiment is done on *Linux* machine (Linux 3.16.0-4-amd64 #1 SMP > > Debian 3.16.7-ckt11-1 (2015-05-24) x86_64 GNU/Linux) with* intel NIC* > > (Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection) using > > the *ixgbe* driver. > > > > To understand this problem, I look into the patch code to the ixgbe > driver. > > I found that the flag NS_BUF_CHANGED is not used in the linux driver for > > ixgbe. please look the header file /LINUX/ixgbe_netmap_linux.h. In > funtions > > ixgbe_netmap_txsync and ixgbe_netmap_rxsync, the function call > > netmap_reload_map is commented out. However, the ixgbe's driver patch for > > FreeBSD calls the reload function. > > > > So, I am wondering if this is a known problem when using netmap on LINUX. > > Has anybody found the same problem? > > Is netmap_reload_map required in Linux? why it is not called in the > driver > > patch? > > What is the recommended solution for this problem? > > > > Thanks! > > > > Best, > > Xiaoye > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > > -- > -----------------------------------------+------------------------------- > Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione > http://www.iet.unipi.it/~luigi/ . Universita` di Pisa > TEL +39-050-2217533 . via Diotisalvi 2 > Mobile +39-338-6809875 . 56122 PISA (Italy) > -----------------------------------------+------------------------------- > >