From owner-freebsd-ipfw@FreeBSD.ORG Thu Sep 13 16:11:09 2012 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 003381065670 for ; Thu, 13 Sep 2012 16:11:08 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) by mx1.freebsd.org (Postfix) with ESMTP id 53AFB8FC15 for ; Thu, 13 Sep 2012 16:11:07 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id q8DGAuYQ098336; Fri, 14 Sep 2012 02:10:56 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Fri, 14 Sep 2012 02:10:56 +1000 (EST) From: Ian Smith To: Soren Dreijer In-Reply-To: Message-ID: <20120914020023.K51539@sola.nimnet.asn.au> References: <20120913221758.E51539@sola.nimnet.asn.au> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-ipfw@freebsd.org Subject: Re: Significant network latency when using ipfw and in-kernel NAT X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Sep 2012 16:11:09 -0000 On Thu, 13 Sep 2012 0:48:01 -0500, Soren Dreijer wrote: > Definitely. Since this is a server in production, I've obfuscated some > of the IPs, etc. > > First off, here's the ifconfig. Our setup consists of a private (ix0) > and a public nic (ix1) and an ip tunnel (gif0), which is what we use > in ipfw to forward incoming packets to our internal boxes: > > ix0: flags=8843 metric 0 mtu 1500 > options=401bb > ether XX:XX:XX:XX:XX:XX > inet netmask 0xffffffc0 broadcast xx > inet6 xxxx::xxx:xxxx:xxxx:xxxx%ix0 prefixlen 64 scopeid 0x7 > nd6 options=29 > media: Ethernet autoselect (10Gbase-Twinax ) > status: active > ix1: flags=8843 metric 0 mtu 1500 > options=400bb > ether XX:XX:XX:XX:XX:XX > inet netmask 0xfffffff8 broadcast xx > inet6 xxxx::xxx:xxxx:xxxx:xxxx%ix1 prefixlen 64 scopeid 0x8 > inet netmask 0xffffffff broadcast xx > inet netmask 0xffffffff broadcast xx > nd6 options=29 > media: Ethernet autoselect (10Gbase-Twinax ) > status: active [ Soren and I had some off-list discussion which doesn't seem to have helped matters, but I'll repost this as the only clue I had. Anybody? ] Before anything else .. % man ipfw | tail | head -4 Due to the architecture of libalias(3), ipfw nat is not compatible with the TCP segmentation offloading (TSO). Thus, to reliably nat your net- work traffic, please disable TSO on your NICs using ifconfig(8). I don't know if this applies to VLAN_HWTSO, but likely to TSO4 on ix0? Do things change if you try disabling all TSO? I'd also change all 'out via ix1' to 'out xmit ix1', given the former also applies to traffic going out anywhere that came _in_ on ix1. cheers, Ian