From owner-freebsd-ipfw@FreeBSD.ORG Fri May 16 04:20:40 2008 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 26E591065675; Fri, 16 May 2008 04:20:40 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from gaia.nimnet.asn.au (nimbin.lnk.telstra.net [139.130.45.143]) by mx1.freebsd.org (Postfix) with ESMTP id 9B4168FC19; Fri, 16 May 2008 04:20:37 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from localhost (smithi@localhost) by gaia.nimnet.asn.au (8.8.8/8.8.8R1.5) with SMTP id NAA15526; Fri, 16 May 2008 13:56:50 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Fri, 16 May 2008 13:56:49 +1000 (EST) From: Ian Smith To: Jeremy Chadwick In-Reply-To: <20080515162056.GA17187@eos.sc1.parodius.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: Vivek Khera , "Andrey V. Elsukov" , "Bruce M. Simpson" , freebsd-stable@freebsd.org, freebsd-ipfw@freebsd.org Subject: Re: how much memory does increasing max rules for IPFW take up? X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 May 2008 04:20:40 -0000 On Thu, 15 May 2008, Jeremy Chadwick wrote: > On Thu, May 15, 2008 at 11:03:53AM +0100, Bruce M. Simpson wrote: > > Andrey V. Elsukov wrote: > >> Vivek Khera wrote: > >>> I had a box run out of dynamic state space yesterday. I found I can > >>> increase the number of dynamic rules by increasing the sysctl parameter > >>> net.inet.ip.fw.dyn_max. I can't find, however, how this affects memory > >>> usage on the system. Is it dyanamically allocated and de-allocated, or > >>> is it a static memory buffer? > >> > >> Each dynamic rule allocated dynamically. Be careful, too many dynamic > >> rules will work very slow. > > > > Got any figures for this? I took a quick glance and it looks like it just > > uses a hash over dst/src/dport/sport. If there are a lot of raw IP or ICMP > > flows then that's going to result in hash collisions. > > > > It might be a good project for someone to optimize if it isn't scaling for > > folk. "Bloomier" filters are probably worth a look -- bloom filters are a > > class of probabilistic hash which may return a false positive, "bloomier" > > filters are a refinement which tries to limit the false positives. > > > > Having said that the default tunable of 256 state entries is probably quite > > low for use cases other than "home/small office NAT gateway". > > It's far too low for home/small office. Standard Linux NAT routers, > such as the Linksys WRT54G/GL, come with a default state table count of > 2048, and often is increased by third-party firmwares to 8192 based on > justified necessity. Search for "conntrack" below: > > http://www.polarcloud.com/firmware > > 256 can easily be exhausted by more than one user loading multiple HTTP > 1.0 web pages at one time (such is the case with many users now have > browsers that load 7-8 web pages into separate tabs during startup). > > And if that's not enough reason, consider torrents, which is quite often > what results in a home or office router exhausting its state table. > > Bottom line: the 256 default is too low. It needs to be increased to at > least 2048. I think there may be some confusion in terms. Looking at defaults on my older 5.5 system - sure, call it a "home/small office NAT gateway": net.inet.ip.fw.dyn_buckets: 256 net.inet.ip.fw.curr_dyn_buckets: 256 net.inet.ip.fw.dyn_count: 212 net.inet.ip.fw.dyn_max: 4096 net.inet.ip.fw.static_count: 153 What defaults to 256 is the number of hash table buckets, not the max number of dynamic rules, here 4096 (though the 5.5 manual says 8192). On hash collisions, a linked list is used for duplicate hashes of: i = (id->dst_ip) ^ (id->src_ip) ^ (id->dst_port) ^ (id->src_port); i &= (curr_dyn_buckets - 1); So while 256 may well be too few buckets for many systems, and like Bruce I wonder about the effectiveness of the xor hash for raw IP & ICMP and wouldn't mind seeing some stats on bucket use vs linked list lengths for various workloads, it doesn't determine the max no. of dynamic rules available, which is adjustable up without any apparent static memory allocation, and is moderated by the various expiry timeout sysctls. For reference, I admin a 4.8 filtering bridge with up to 20 boxes behind it, that has only very rarely reported exceeding the max no. of dynamic rules with the (4.8) default net.inet.ip.fw.dyn_max of 1000 .. however it only keeps state for UDP connections (and yes, it only ever hits that limit on torrents or skype, which are generally admin. prohib. :) cheers, Ian (not subscribed to -ipfw)