From owner-freebsd-net@FreeBSD.ORG Tue Jan 6 00:16:45 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CA443ACC for ; Tue, 6 Jan 2015 00:16:45 +0000 (UTC) Received: from mail-la0-x22e.google.com (mail-la0-x22e.google.com [IPv6:2a00:1450:4010:c03::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 241A52BBF for ; Tue, 6 Jan 2015 00:16:45 +0000 (UTC) Received: by mail-la0-f46.google.com with SMTP id q1so19581916lam.33 for ; Mon, 05 Jan 2015 16:16:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=8tJZOy4UvP9qUYM8H3jkk7BCfRvBTL70RWMbMZ64S6g=; b=tYOyI3+udjTlVHOhOPiVWX0OAt/QF92UxhhegfE+cKMAnvrvKzhZ35xqjp1GRdhPsA 3QYeZnAM2Qe8JUGHDdpSPWf9SfHTjlli/nqCWsCyyqSlU24Ey1db0kAZgrAag3UhaNKw 7uWlID7bxltG5eIKWy6o6gE0kM+JYf/YS2bdXpvTm8auExkWkNo3pnc9A1sVSN0yKVYa vJX5nBzOsas3+5RkFvu2OUw4bJkT0Qx8vDSGWY2/8PwSRaxfhFa9Ad65jyl5zNcVSa5M PibSfEinvTxT8uz505tCuD6575cAns1cfGAvNJivH+V3W8ZLFjUFCDyrMGjW7jZrdpOD EODA== X-Received: by 10.112.180.135 with SMTP id do7mr54028072lbc.23.1420503401749; Mon, 05 Jan 2015 16:16:41 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.36.215 with HTTP; Mon, 5 Jan 2015 16:16:01 -0800 (PST) In-Reply-To: References: From: Carlos Ferreira Date: Tue, 6 Jan 2015 00:16:01 +0000 Message-ID: Subject: Re: Regarding Netmap internal memory allocation. To: Luigi Rizzo Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jan 2015 00:16:46 -0000 First of all, Happy New Year :) Now, to more serious developments.. I was unable to reduce the netmap memory usage by manipulating the buf_num value. The lowest I could get was around the 20MB which is still high. I'm going to change the strategy and try to directly mmap the RX and TX buffers from the interface sockets. The results are going to be worse and I know that I'm not going to have Zero-Copy capability between interfaces but... oh well, I guess at least I'm going to be able to do something better than basic layer 2 sockets. Thank you for the info regarding netmap. On 30 December 2014 at 18:55, Carlos Ferreira wrote: > Well... due to budget constraints I'm using USB 100Mb ports :) > This is for experimental purposes only for now. > > Btw, can netmap work with wireless interfaces? I believe you once answered > this question, but I could not find it in the mail list. > > On 30 December 2014 at 18:40, Luigi Rizzo wrote: > >> On Tue, Dec 30, 2014 at 6:38 PM, Carlos Ferreira >> wrote: >> > Ok, I'm having some trouble in tuning the amount of memory for netmap. >> > >> > I have been following the man page from FreeBSD in other to understand >> the >> > values at /sys/modules/netmap/parameters for linux but I'm having some >> > trouble in understanding what each value actually means. >> > For the following values: >> > >> > dev.netmap.ring_num: 200 -> Is this the number of rings in the Ring >> Buffer >> > Pool? >> >> yes. For interfaces with a single queue you need 4 rings (rx+tx for the >> nic and >> another rx+tx for the host port) >> >> > dev.netmap.ring_size: 36864 -> Is this value, the number of slots per >> ring? >> >> this is the size in bytes of each ring. The number of slots is set by the >> hardware (low end devices as in the openwrt devices will probably use >> 256 or 512 slots, so 10-12k should suffice. But this is not worth >> changing. >> >> Instead, you should reduce the number of buffers, though 8MB is only 4000 >> buffers and it is a bit on the low side for 5 ports. >> >> However, as far as I know most openwrt devices only have one physical NIC, >> and a switch implementing various vlans. >> >> cheers >> luigi >> >> > >> > I'm trying to keep the amount of memory used by netmap as low as 4MB - >> 8MB >> > since I'm going to use only up to 4 NICs and one TAP. >> > >> > Thanks for the help! >> > >> > >> > On 30 December 2014 at 16:16, Luigi Rizzo wrote: >> >> >> >> you can #undefine WITH_VALE. >> >> But it is only 20K of code (and 150K of data structures, which you >> >> can further reduce by lowering NM_BRIDGS). >> >> The saving is probably not worth the effort. >> >> >> >> cheers >> >> luigi >> >> >> >> On Tue, Dec 30, 2014 at 5:08 PM, Carlos Ferreira < >> carlosmf.pt@gmail.com> >> >> wrote: >> >> > By the way, another question. >> >> > Is there a way to not compile the code regarding the VALE switch? I'm >> >> > only >> >> > interested in using netmap with Tap Devices and NICs, so I was >> hoping to >> >> > save some memory. >> >> > >> >> > On 30 December 2014 at 15:47, Carlos Ferreira > > >> >> > wrote: >> >> >> >> >> >> You mean netmap_mem2.c ? It was there where I found the >> >> >> NETMAP_BUF_MAX_NUM >> >> >> define. >> >> >> >> >> >> >> >> >> >> >> >> On 30 December 2014 at 15:43, Carlos Ferreira < >> carlosmf.pt@gmail.com> >> >> >> wrote: >> >> >>> >> >> >>> Ok thanks. I was hoping not having to recompile the module, but >> it's >> >> >>> ok. >> >> >>> Thank you for the info! >> >> >>> >> >> >>> >> >> >>> On 30 December 2014 at 15:38, Luigi Rizzo >> wrote: >> >> >>>> >> >> >>>> you can reduce the amount of ram (buffers, mostly) by >> >> >>>> tweaking the values in netmap_mem2.c :: >> >> >>>> struct netmap_obj_params netmap_params[NETMAP_POOLS_NR] = { >> >> >>>> ... >> >> >>>> } >> >> >>>> >> >> >>>> or you can simply modify the constant >> >> >>>> >> >> >>>> netmap_mem2.h:#define NETMAP_BUF_MAX_NUM 20*4096*2 >> >> >>>> >> >> >>>> to something smaller that suits an openwrt box >> >> >>>> (in which i am very interested, as I'd like to deploy one of these >> >> >>>> soon) >> >> >>>> >> >> >>>> cheers >> >> >>>> luigi >> >> >>>> >> >> >>>> >> >> >>>> On Tue, Dec 30, 2014 at 4:12 PM, Carlos Ferreira >> >> >>>> >> >> >>>> wrote: >> >> >>>> > Update: >> >> >>>> > >> >> >>>> > I noticed that the netmap module was still crashing, after >> >> >>>> > changing >> >> >>>> > the >> >> >>>> > OpenWRT VM ram to 256MB. I now raised to 1GB and it no longer >> >> >>>> > crashed. >> >> >>>> > The >> >> >>>> > netmap module is now consuming about 350MB of Ram, which for my >> >> >>>> > objectives >> >> >>>> > is just too much... >> >> >>>> > >> >> >>>> > On 30 December 2014 at 14:06, Carlos Ferreira >> >> >>>> > >> >> >>>> > wrote: >> >> >>>> > >> >> >>>> >> To Luigi and to whom may be able to help >> >> >>>> >> >> >> >>>> >> Hello all. >> >> >>>> >> >> >> >>>> >> Is it possible to reduce the size of the memory buffer >> allocated >> >> >>>> >> by >> >> >>>> >> the >> >> >>>> >> netmap module? >> >> >>>> >> I'm asking this because I was implementing some testing code, >> >> >>>> >> using >> >> >>>> >> NICs >> >> >>>> >> and a Tap device in an OpenWRT VM with 64MB of RAM. >> >> >>>> >> Because of the small RAM amount, the nm_open crashed when the >> >> >>>> >> program >> >> >>>> >> tried to netmap the tap device, after I previously netmapped >> one >> >> >>>> >> NIC >> >> >>>> >> successfully. >> >> >>>> >> After the crash, I bumped the VM RAM to 256MB and the test >> program >> >> >>>> >> ran >> >> >>>> >> well, but not without me noticing that the VM RAM consumption >> was >> >> >>>> >> increased about 90 MB by netmap. >> >> >>>> >> >> >> >>>> >> Resuming, I want to know if there is a way to reduce the memory >> >> >>>> >> buffer >> >> >>>> >> allocation, without recompiling the netmap kernel module. >> >> >>>> >> >> >> >>>> >> Thank you for the attention. >> >> >>>> >> >> >> >>>> >> -- >> >> >>>> >> >> >> >>>> >> Carlos Miguel Ferreira >> >> >>>> >> Researcher at Telecommunications Institute >> >> >>>> >> Aveiro - Portugal >> >> >>>> >> Work E-mail - cmf@av.it.pt >> >> >>>> >> Skype & GTalk -> carlosmf.pt@gmail.com >> >> >>>> >> LinkedIn -> http://www.linkedin.com/in/carlosmferreira >> >> >>>> >> >> >> >>>> > >> >> >>>> > >> >> >>>> > >> >> >>>> > -- >> >> >>>> > >> >> >>>> > Carlos Miguel Ferreira >> >> >>>> > Researcher at Telecommunications Institute >> >> >>>> > Aveiro - Portugal >> >> >>>> > Work E-mail - cmf@av.it.pt >> >> >>>> > Skype & GTalk -> carlosmf.pt@gmail.com >> >> >>>> > LinkedIn -> http://www.linkedin.com/in/carlosmferreira >> >> >>>> > _______________________________________________ >> >> >>>> > freebsd-net@freebsd.org mailing list >> >> >>>> > http://lists.freebsd.org/mailman/listinfo/freebsd-net >> >> >>>> > To unsubscribe, send any mail to >> >> >>>> > "freebsd-net-unsubscribe@freebsd.org" >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> -- >> >> >>>> >> >> >>>> >> >> >>>> >> -----------------------------------------+------------------------------- >> >> >>>> Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. >> >> >>>> dell'Informazione >> >> >>>> http://www.iet.unipi.it/~luigi/ . Universita` di Pisa >> >> >>>> TEL +39-050-2211611 . via Diotisalvi 2 >> >> >>>> Mobile +39-338-6809875 . 56122 PISA (Italy) >> >> >>>> >> >> >>>> >> >> >>>> >> -----------------------------------------+------------------------------- >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> -- >> >> >>> >> >> >>> Carlos Miguel Ferreira >> >> >>> Researcher at Telecommunications Institute >> >> >>> Aveiro - Portugal >> >> >>> Work E-mail - cmf@av.it.pt >> >> >>> Skype & GTalk -> carlosmf.pt@gmail.com >> >> >>> LinkedIn -> http://www.linkedin.com/in/carlosmferreira >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> >> >> Carlos Miguel Ferreira >> >> >> Researcher at Telecommunications Institute >> >> >> Aveiro - Portugal >> >> >> Work E-mail - cmf@av.it.pt >> >> >> Skype & GTalk -> carlosmf.pt@gmail.com >> >> >> LinkedIn -> http://www.linkedin.com/in/carlosmferreira >> >> > >> >> > >> >> > >> >> > >> >> > -- >> >> > >> >> > Carlos Miguel Ferreira >> >> > Researcher at Telecommunications Institute >> >> > Aveiro - Portugal >> >> > Work E-mail - cmf@av.it.pt >> >> > Skype & GTalk -> carlosmf.pt@gmail.com >> >> > LinkedIn -> http://www.linkedin.com/in/carlosmferreira >> >> >> >> >> >> >> >> -- >> >> >> -----------------------------------------+------------------------------- >> >> Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. >> dell'Informazione >> >> http://www.iet.unipi.it/~luigi/ . Universita` di Pisa >> >> TEL +39-050-2211611 . via Diotisalvi 2 >> >> Mobile +39-338-6809875 . 56122 PISA (Italy) >> >> >> -----------------------------------------+------------------------------- >> > >> > >> > >> > >> > -- >> > >> > Carlos Miguel Ferreira >> > Researcher at Telecommunications Institute >> > Aveiro - Portugal >> > Work E-mail - cmf@av.it.pt >> > Skype & GTalk -> carlosmf.pt@gmail.com >> > LinkedIn -> http://www.linkedin.com/in/carlosmferreira >> >> >> >> -- >> -----------------------------------------+------------------------------- >> Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione >> http://www.iet.unipi.it/~luigi/ . Universita` di Pisa >> TEL +39-050-2211611 . via Diotisalvi 2 >> Mobile +39-338-6809875 . 56122 PISA (Italy) >> -----------------------------------------+------------------------------- >> > > > > -- > > Carlos Miguel Ferreira > Researcher at Telecommunications Institute > Aveiro - Portugal > Work E-mail - cmf@av.it.pt > Skype & GTalk -> carlosmf.pt@gmail.com > LinkedIn -> http://www.linkedin.com/in/carlosmferreira > -- Carlos Miguel Ferreira Researcher at Telecommunications Institute Aveiro - Portugal Work E-mail - cmf@av.it.pt Skype & GTalk -> carlosmf.pt@gmail.com LinkedIn -> http://www.linkedin.com/in/carlosmferreira