From owner-freebsd-ipfw@FreeBSD.ORG Wed Apr 24 20:21:16 2013 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 00110490; Wed, 24 Apr 2013 20:21:15 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id B7B781CD9; Wed, 24 Apr 2013 20:21:15 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 5CA8B7300A; Wed, 24 Apr 2013 22:23:08 +0200 (CEST) Date: Wed, 24 Apr 2013 22:23:08 +0200 From: Luigi Rizzo To: "Alexander V. Chernikov" Subject: Re: [patch] ipfw interface tracking and opcode rewriting Message-ID: <20130424202308.GA11146@onelab2.iet.unipi.it> References: <517801D3.5040502@FreeBSD.org> <20130424162349.GA8439@onelab2.iet.unipi.it> <51780C49.7000204@FreeBSD.org> <20130424190930.GA10395@onelab2.iet.unipi.it> <51783798.4020004@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <51783798.4020004@FreeBSD.org> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-ipfw@freebsd.org, luigi@freebsd.org X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Apr 2013 20:21:16 -0000 On Wed, Apr 24, 2013 at 11:50:48PM +0400, Alexander V. Chernikov wrote: > On 24.04.2013 23:09, Luigi Rizzo wrote: > > On Wed, Apr 24, 2013 at 08:46:01PM +0400, Alexander V. Chernikov wrote: > >> On 24.04.2013 20:23, Luigi Rizzo wrote: ... > >> Well, actually I'm thinking of the next 2 steps: > >> 1) making kernel rule header more compact (20 bytes instead of 48) and > >> making it invisible for userland. > >> This involves rule counters to be stored separately (and possibly as > >> pcpu-based ones). > >> 2) since ruleset is now nearly readonly and more or less compact we can > >> try to store it in > >> contiguous address space to optimize cache line usage. > > certainly a worthwhile goal (also using gleb's new counters) > > but i suspect that compacting rules are a second order effect. > > I a bit skeptical they make a big difference on the in-kernel > > version of ipfw. You might see some difference in the > My current numbers are ~5mpps of IPv4 forwarding with ipfw turned on (1 > rule) for vlans over ixgbe, with 60% cpu usage (2xE5646). yes, but that means about 1mpps per core. the userspace version, i have been told, reached 14.2mpps with one core when running on netmap interfaces (single rule). So the impact of 20-30ns per rule is much higher in the second case. Speaking of which -- it would be better to report performance in terms of ns per packet (or per rule), rather than Mpps, because it is much easier to compare results. Saying "20% better" or "300Kpps more" requires a lot more context to understand how much things improved. > For lagg with 2x ixgbe it is ~7mpps with the same 60% usage. > (And, say, 70% of CPU usage on our production is ipfw, despite low > number of rules). > > userspace version, which runs on top of netmap. > We are preparing to move forward in this direction (and thinking of > 20-30mpps as our goal). > (And I hope some changes of kernel-based version can migrate to userland > one :)) yes, my goal is to have a single source tree that builds both in kernel and userspace. The diffs are very small, I just have a little bit of glue code to hook the packet path and the control socket. cheers luigi