From owner-freebsd-net@FreeBSD.ORG Thu Jan 22 19:03:27 2015 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 31D712A3 for ; Thu, 22 Jan 2015 19:03:27 +0000 (UTC) Received: from mail-lb0-x230.google.com (mail-lb0-x230.google.com [IPv6:2a00:1450:4010:c04::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A6F5D676 for ; Thu, 22 Jan 2015 19:03:26 +0000 (UTC) Received: by mail-lb0-f176.google.com with SMTP id z12so3263916lbi.7 for ; Thu, 22 Jan 2015 11:03:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=Dlo4EAThH99KhMxjnwUPhJflP0GHfqAVF54u9+JuOqw=; b=ph+2xkM3LLcwOaTBB5j3nvbP80VWXMNvMHRO6dxwit+YaBbNcLOuL0cv7oS25dnMnn Hw8wjR0BVYKS+B+FEfkQxctTQVOiLIGJ4a7yCf0TuaytdpZFKTYiYE6p53jzsAbQqXpp uqbYvujRpUqE9O1PA9tysNvw1PAiBvJ84+82Bc9zgdRLBWvDv2MIP8beugKHa4DNyRx4 APCqve5ZxPBoVcIZkRNkUNtJ58Otohoby6BnbUxn6eYmCNY5IqmRt7l7GqnalMBxNxlM jzv3Hg3yLzG3xyySQd4R4A9EyDyM02hY0G/KvU554ic1IR3fsnzKuUP0Q3oeBLDp31rd hKAQ== MIME-Version: 1.0 X-Received: by 10.112.98.99 with SMTP id eh3mr3174617lbb.32.1421953404749; Thu, 22 Jan 2015 11:03:24 -0800 (PST) Sender: rizzo.unipi@gmail.com Received: by 10.114.10.168 with HTTP; Thu, 22 Jan 2015 11:03:24 -0800 (PST) In-Reply-To: References: Date: Thu, 22 Jan 2015 11:03:24 -0800 X-Google-Sender-Auth: vbiTgRodEktQhy61ttlmxfl7t0w Message-ID: Subject: Re: Netmap mempool From: Luigi Rizzo To: Ciprian Barbu Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Jan 2015 19:03:27 -0000 On Thu, Jan 22, 2015 at 10:12 AM, Ciprian Barbu wrote: > Hello, > > I have some questions related to OpenDataPlane. I have implemented the > I/O access using netmap for ODP but currently it's suffering from low > performance due to packets being copied between the netmap slots and > ODP buffers. > > I want to create ODP buffer pools on top of netmap memory but I'm > concerned with three things: > 1. how to keep a buffer for as long as needed without affecting the > receive process; I guess I could use the NS_BUF_CHANGED flag in some > way > 2. the netmap memory may not be large enough to accommodate a complex > ODP application > 3. the ODP application may occasionally hold on to buffers for longer time > there is a way to hold buffers without returning them to the ring using the 'cur' pointer in the ring, see the netmap manpage how this is implemented and how it works on the tx and rx path. However, before embarking in zero copy i'd like you to elaborate on what you mean for "low performance" (how many pps you see) and what are your expectations. In my experience low performance almost always comes from insufficient batching or other application bottlenecks. Zero copy tricks come with great complications to track who owns what, sometimes with additional hidden overhead (e.g. if you change buffers the driver has to reprogram the IOMMU) and often have a very limited effect in real world applications. Sure the numbers do look good on benchmarks or academic papers, but there people don't worry about robustness and code maintainability. cheers luigi