From owner-freebsd-ipfw@FreeBSD.ORG  Tue May 27 15:50:41 2003
Return-Path: <owner-freebsd-ipfw@FreeBSD.ORG>
Delivered-To: freebsd-ipfw@freebsd.org
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 8AF2C37B401
	for <freebsd-ipfw@freebsd.org>; Tue, 27 May 2003 15:50:41 -0700 (PDT)
Received: from horsey.gshapiro.net (horsey.gshapiro.net [64.105.95.154])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 07A0543F3F
	for <freebsd-ipfw@freebsd.org>; Tue, 27 May 2003 15:50:41 -0700 (PDT)
	(envelope-from gshapiro@gshapiro.net)
Received: from horsey.gshapiro.net (localhost [127.0.0.1])
	h4RMoeiR057401
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <freebsd-ipfw@freebsd.org>; Tue, 27 May 2003 15:50:40 -0700 (PDT)
Received: (from gshapiro@localhost)h4RMoeFD057400
	for freebsd-ipfw@freebsd.org; Tue, 27 May 2003 15:50:40 -0700 (PDT)
Date: Tue, 27 May 2003 15:50:40 -0700
From: Gregory Neil Shapiro <gshapiro@freebsd.org>
To: freebsd-ipfw@freebsd.org
Message-ID: <20030527225040.GV13285@horsey.gshapiro.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.4i
Subject: IPFW V2 dynamic keepalives broken
X-BeenThere: freebsd-ipfw@freebsd.org
X-Mailman-Version: 2.1.1
Precedence: list
List-Id: IPFW Technical Discussions <freebsd-ipfw.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw>,
	<mailto:freebsd-ipfw-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-ipfw>
List-Post: <mailto:freebsd-ipfw@freebsd.org>
List-Help: <mailto:freebsd-ipfw-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw>,
	<mailto:freebsd-ipfw-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 27 May 2003 22:50:41 -0000

Since enabling IPFW V2 on RELENG_4, I've had a fairly busy web/ftp
server run out dynamic buckets for new rules.  Stopping the web/ftp
server processes and starting it helped allevaite the problem somewhat.
When it gets in this state, there are thousands of connections in
FIN_WAIT or FIN_WAIT_2.  It takes about 2 weeks to collect enough of
these FIN_WAIT* sockets to cause a problem.

After about 5 days, the count is already up to 461:

# netstat -anf inet | grep FIN_WAIT | wc -l
     461

I discovered however that it is somehow dyn_keepalives that is causing
the problem.  If I turn them off, things return to normal:

# sysctl net.inet.ip.fw.dyn_keepalive=0
net.inet.ip.fw.dyn_keepalive: 1 -> 0
(wait a few seconds)
# netstat -anf inet | grep FIN_WAIT | wc -l
      16

Here is a snapshot of how things looked before disabling dyn_keepalive:

# sysctl -a | grep net.inet.ip.fw
net.inet.ip.fw.enable: 1
net.inet.ip.fw.autoinc_step: 100
net.inet.ip.fw.one_pass: 1
net.inet.ip.fw.debug: 1
net.inet.ip.fw.verbose: 1
net.inet.ip.fw.verbose_limit: 0
net.inet.ip.fw.dyn_buckets: 256
net.inet.ip.fw.curr_dyn_buckets: 256
net.inet.ip.fw.dyn_count: 497
net.inet.ip.fw.dyn_max: 2000
net.inet.ip.fw.static_count: 65
net.inet.ip.fw.dyn_ack_lifetime: 300
net.inet.ip.fw.dyn_syn_lifetime: 20
net.inet.ip.fw.dyn_fin_lifetime: 1
net.inet.ip.fw.dyn_rst_lifetime: 1
net.inet.ip.fw.dyn_udp_lifetime: 10
net.inet.ip.fw.dyn_short_lifetime: 60
net.inet.ip.fw.dyn_keepalive: 1

Any ideas?  Could enabling dyn_keepalives prevent the FIN_WAIT* process
from completing?