From owner-freebsd-ipfw@FreeBSD.ORG Mon Sep 10 11:09:47 2012 Return-Path: Delivered-To: freebsd-ipfw@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2EA141065686 for ; Mon, 10 Sep 2012 11:09:47 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 1817E8FC19 for ; Mon, 10 Sep 2012 11:09:47 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q8AB9kgI067370 for ; Mon, 10 Sep 2012 11:09:46 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q8AB9ieX066966 for freebsd-ipfw@FreeBSD.org; Mon, 10 Sep 2012 11:09:44 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 10 Sep 2012 11:09:44 GMT Message-Id: <201209101109.q8AB9ieX066966@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-ipfw@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-ipfw@FreeBSD.org X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Sep 2012 11:09:47 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/169206 ipfw [ipfw] ipfw does not flush entries in table o conf/167822 ipfw [ipfw] [patch] start script doesn't load firewall_type o kern/166406 ipfw [ipfw] ipfw does not set ALTQ identifier for ipv6 traf o kern/165939 ipfw [ipw] bug: incomplete firewall rules loaded if tables o kern/165190 ipfw [ipfw] [lo] [patch] loopback interface is not marking f kern/163873 ipfw [ipfw] ipfw fwd does not work with 'via interface' in o kern/158066 ipfw [ipfw] ipfw + netgraph + multicast = multicast packets o kern/157796 ipfw [ipfw] IPFW in-kernel NAT nat loopback / Default Route o kern/157689 ipfw [ipfw] ipfw nat config does not accept nonexistent int f kern/155927 ipfw [ipfw] ipfw stops to check packets for compliance with o bin/153252 ipfw [ipfw][patch] ipfw lockdown system in subsequent call o kern/153161 ipfw [ipfw] does not support specifying rules with ICMP cod o kern/152113 ipfw [ipfw] page fault on 8.1-RELEASE caused by certain amo o kern/148827 ipfw [ipfw] divert broken with in-kernel ipfw o kern/148689 ipfw [ipfw] antispoof wrongly triggers on link local IPv6 a o kern/148430 ipfw [ipfw] IPFW schedule delete broken. o kern/148091 ipfw [ipfw] ipfw ipv6 handling broken. o kern/143973 ipfw [ipfw] [panic] ipfw forward option causes kernel reboo o kern/143621 ipfw [ipfw] [dummynet] [patch] dummynet and vnet use result o kern/137346 ipfw [ipfw] ipfw nat redirect_proto is broken o kern/137232 ipfw [ipfw] parser troubles o kern/135476 ipfw [ipfw] IPFW table breaks after adding a large number o o kern/129036 ipfw [ipfw] 'ipfw fwd' does not change outgoing interface n p kern/128260 ipfw [ipfw] [patch] ipfw_divert damages IPv6 packets o kern/127230 ipfw [ipfw] [patch] Feature request to add UID and/or GID l f kern/122963 ipfw [ipfw] tcpdump does not show packets redirected by 'ip s kern/121807 ipfw [request] TCP and UDP port_table in ipfw o kern/121122 ipfw [ipfw] [patch] add support to ToS IP PRECEDENCE fields o kern/116009 ipfw [ipfw] [patch] Ignore errors when loading ruleset from o bin/104921 ipfw [patch] ipfw(8) sometimes treats ipv6 input as ipv4 (a o kern/104682 ipfw [ipfw] [patch] Some minor language consistency fixes a o kern/103454 ipfw [ipfw] [patch] [request] add a facility to modify DF b o kern/103328 ipfw [ipfw] [request] sugestions about ipfw table o kern/102471 ipfw [ipfw] [patch] add tos and dscp support o kern/97951 ipfw [ipfw] [patch] ipfw does not tie interface details to o kern/95084 ipfw [ipfw] [regression] [patch] IPFW2 ignores "recv/xmit/v o kern/86957 ipfw [ipfw] [patch] ipfw mac logging o bin/83046 ipfw [ipfw] ipfw2 error: "setup" is allowed for icmp, but s o kern/82724 ipfw [ipfw] [patch] [request] Add setnexthop and defaultrou o bin/78785 ipfw [patch] ipfw(8) verbosity locks machine if /etc/rc.fir o bin/65961 ipfw [ipfw] ipfw2 memory corruption inside add() o kern/60719 ipfw [ipfw] Headerless fragments generate cryptic error mes s kern/55984 ipfw [ipfw] [patch] time based firewalling support for ipfw o kern/48172 ipfw [ipfw] [patch] ipfw does not log size and flags o kern/46159 ipfw [ipfw] [patch] [request] ipfw dynamic rules lifetime f a kern/26534 ipfw [ipfw] Add an option to ipfw to log gid/uid of who cau 46 problems total. From owner-freebsd-ipfw@FreeBSD.ORG Thu Sep 13 04:09:33 2012 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D3C1C106566C for ; Thu, 13 Sep 2012 04:09:33 +0000 (UTC) (envelope-from dreijer@echobit.net) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 927318FC14 for ; Thu, 13 Sep 2012 04:09:33 +0000 (UTC) Received: by obbun3 with SMTP id un3so4782002obb.13 for ; Wed, 12 Sep 2012 21:09:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:content-type:x-gm-message-state; bh=jlrz8ml7bczViqthEWfGYe+jxPFupsLRgguLVYmvwwg=; b=Ll4ZXYS0n8b0P0EwlHkhwLVK91L22iiMnt7pSmCpSgEx4RG5MiX9W0vIEulosP/HPu krC1AE/FuR29YkspSiGal40WF0NWnRZmxzDssc/gBMUj0wLHYDAU+toARoLKTETdYYCF hwRPjM+HP9yc5wIaijU9tjQdw+fsn1R2yg/1H2VBjPT3UF39nN9HhPePWF4saU33WdFe Jjg8Pq8OGrA7586/TQ1OpzlTRsiGus1EJnrkY/61MkwuuRENAQt6yduxLrfqPi/RhApf q0/egHPqGIARTk4I8bsx7DQ4Dvi+cTlWkUrtR6g/4cGf0y3QIo+/7ilbayPy3Py0phEh mYRA== MIME-Version: 1.0 Received: by 10.182.37.41 with SMTP id v9mr545888obj.23.1347509367155; Wed, 12 Sep 2012 21:09:27 -0700 (PDT) Sender: dreijer@echobit.net Received: by 10.76.99.75 with HTTP; Wed, 12 Sep 2012 21:09:27 -0700 (PDT) Date: Wed, 12 Sep 2012 23:09:27 -0500 X-Google-Sender-Auth: oFbgGqyHBkbebm0OEYhIN2ExhS0 Message-ID: From: Soren Dreijer To: freebsd-ipfw@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQlFpRIDOmpYy/ss7XLjnqy4ZWUhNIBJyjQlMuHRz/tVzT2cNAzz4/ky4r3TxQSOjLeMr/w8 Subject: Significant network latency when using ipfw and in-kernel NAT X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Sep 2012 04:09:33 -0000 Hi there, We're running freebsd 9.0-RELEASE on a box whose primary purpose is to act as a firewall and a gateway. Up until today, we've been using ipfw in conjunction with natd and the divert action in ipfw to forward packets between the freebsd box (i.e. the public Internet) and our private servers. Unfortunately, natd appears to be quite the CPU hog and we therefore decided to switch to the in-kernel NAT support in ipfw. The issue we're running in to is that the network latency appears to be skyrocketing when ipfw contains nat rules. Basically all TCP traffic originating from the box times out and pinging google.com on the box gives an average of ~10 SECONDS -- and that's even if I explicitly allow all ICMP traffic before the packets even get to the nat rules in ipfw. The really odd part, however, is that I can ping the freebsd box just fine externally. For instance, pinging the server from my home connection gives an average of 45 ms. I'm also able to communicate just fine with the internal servers through the freebsd box. Does anybody have any idea what's going on? I assume I must've misconfigured something big here... Thanks, Soren Dreijer From owner-freebsd-ipfw@FreeBSD.ORG Thu Sep 13 12:42:05 2012 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 90C9B106566B for ; Thu, 13 Sep 2012 12:42:05 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) by mx1.freebsd.org (Postfix) with ESMTP id 11FED8FC08 for ; Thu, 13 Sep 2012 12:42:04 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id q8DCfu1w091269; Thu, 13 Sep 2012 22:41:57 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Thu, 13 Sep 2012 22:41:56 +1000 (EST) From: Ian Smith To: Soren Dreijer In-Reply-To: Message-ID: <20120913221758.E51539@sola.nimnet.asn.au> References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-ipfw@freebsd.org Subject: Re: Significant network latency when using ipfw and in-kernel NAT X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Sep 2012 12:42:05 -0000 On Wed, 12 Sep 2012 23:09:27 -0500, Soren Dreijer wrote: > Hi there, > > We're running freebsd 9.0-RELEASE on a box whose primary purpose is to > act as a firewall and a gateway. Up until today, we've been using ipfw > in conjunction with natd and the divert action in ipfw to forward > packets between the freebsd box (i.e. the public Internet) and our > private servers. > > Unfortunately, natd appears to be quite the CPU hog and we therefore > decided to switch to the in-kernel NAT support in ipfw. The issue > we're running in to is that the network latency appears to be > skyrocketing when ipfw contains nat rules. Basically all TCP traffic > originating from the box times out and pinging google.com on the box > gives an average of ~10 SECONDS -- and that's even if I explicitly > allow all ICMP traffic before the packets even get to the nat rules in > ipfw. > > The really odd part, however, is that I can ping the freebsd box just > fine externally. For instance, pinging the server from my home > connection gives an average of 45 ms. I'm also able to communicate > just fine with the internal servers through the freebsd box. > > Does anybody have any idea what's going on? I assume I must've > misconfigured something big here... Or maybe only something small .. but without seeing your basic ruleset and network config - obscured as need be - we can only guess. Maybe an 'ifconfig', 'ipfw show' and 'ipfw nat show config' would illustrate? cheers, Ian From owner-freebsd-ipfw@FreeBSD.ORG Thu Sep 13 15:48:03 2012 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 10C8A106564A for ; Thu, 13 Sep 2012 15:48:03 +0000 (UTC) (envelope-from dreijer@echobit.net) Received: from mail-oa0-f54.google.com (mail-oa0-f54.google.com [209.85.219.54]) by mx1.freebsd.org (Postfix) with ESMTP id BFAE48FC1D for ; Thu, 13 Sep 2012 15:48:02 +0000 (UTC) Received: by oagm1 with SMTP id m1so2492916oag.13 for ; Thu, 13 Sep 2012 08:48:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :x-gm-message-state; bh=X1PH97+/A5HeTXZyWVVOWZmvjgLZOCX0H02EQTH5uXw=; b=mrdszTg90Voh3is7dgnZp8A1lGaT2x2oYDMldTNOIa/749caZsPiUzvPWZDRwoqDIc XUM20qXxK235lrKPGXiG8BNQZ0lEa/FmAZpkZTnBR4rdYndt2bG+OJKHR4tdeRVTcnhq e3s3sIjOx3j94MGFdjGJ/AG7DB6ApkZcmP4wvGU7u6kIuD1MYc5qwmgoGHx9iI7v6d1o nUITVPiV2AlThw0IRH0y7Sd4eebjHHWMT0Dn2RUh4ZrD0dLzG0bevUmObdTskGXs3Wjw E4bmKdjaCqWKhitrfgnjJwd2+EhAfePKQLyg9doHCutwinqK0xYhMQ+bZgsK3PdUSBwS dBaQ== MIME-Version: 1.0 Received: by 10.182.154.70 with SMTP id vm6mr3038359obb.50.1347551281745; Thu, 13 Sep 2012 08:48:01 -0700 (PDT) Sender: dreijer@echobit.net Received: by 10.76.99.75 with HTTP; Thu, 13 Sep 2012 08:48:01 -0700 (PDT) In-Reply-To: <20120913221758.E51539@sola.nimnet.asn.au> References: <20120913221758.E51539@sola.nimnet.asn.au> Date: Thu, 13 Sep 2012 10:48:01 -0500 X-Google-Sender-Auth: wPxFi2koeyJrEk-wEXVUZgDoZtw Message-ID: From: Soren Dreijer To: Ian Smith Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQmsQIWtNpKcWTYhIib+ko2NiRXdFLWyx0xLpvPKGJ/McvsiQJLkOvxpUQEv/t4Fx2V6DmDB Cc: freebsd-ipfw@freebsd.org Subject: Re: Significant network latency when using ipfw and in-kernel NAT X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Sep 2012 15:48:03 -0000 Definitely. Since this is a server in production, I've obfuscated some of the IPs, etc. First off, here's the ifconfig. Our setup consists of a private (ix0) and a public nic (ix1) and an ip tunnel (gif0), which is what we use in ipfw to forward incoming packets to our internal boxes: ix0: flags=8843 metric 0 mtu 1500 options=401bb ether XX:XX:XX:XX:XX:XX inet netmask 0xffffffc0 broadcast xx inet6 xxxx::xxx:xxxx:xxxx:xxxx%ix0 prefixlen 64 scopeid 0x7 nd6 options=29 media: Ethernet autoselect (10Gbase-Twinax ) status: active ix1: flags=8843 metric 0 mtu 1500 options=400bb ether XX:XX:XX:XX:XX:XX inet netmask 0xfffffff8 broadcast xx inet6 xxxx::xxx:xxxx:xxxx:xxxx%ix1 prefixlen 64 scopeid 0x8 inet netmask 0xffffffff broadcast xx inet netmask 0xffffffff broadcast xx nd6 options=29 media: Ethernet autoselect (10Gbase-Twinax ) status: active ipfw0: flags=8801 metric 0 mtu 65536 nd6 options=29 lo0: flags=8049 metric 0 mtu 16384 options=3 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa inet 127.0.0.1 netmask 0xff000000 nd6 options=21 gif0: flags=8051 metric 0 mtu 1500 tunnel inet --> inet 172.16.1.1 --> 172.16.1.2 netmask 0xffff0000 nd6 options=29 options=1 The basic ruleset looks like this. One-pass is off so that packets are reinjected after going through NAT'ing and pipes: 00001 16653 4417407 allow ip from any to any via ix0 00003 14588 2860344 allow ip from any to any via gif1 00006 0 0 allow ip from any to any via lo0 00010 0 0 deny ip from 192.168.0.0/16 to any in via ix1 00011 0 0 deny ip from 172.16.0.0/12 to any in via ix1 00012 0 0 deny ip from 10.0.0.0/8 to any in via ix1 00013 0 0 deny ip from 127.0.0.0/8 to any in via ix1 00014 0 0 deny ip from 0.0.0.0/8 to any in via ix1 00015 0 0 deny ip from 169.254.0.0/16 to any in via ix1 00016 0 0 deny ip from 192.0.2.0/24 to any in via ix1 00017 0 0 deny ip from 204.152.64.0/23 to any in via ix1 00018 0 0 deny ip from 224.0.0.0/3 to any in via ix1 00019 15 1020 allow icmp from any to any via ix1 # For testing purposes, allow all ICMP in and out of the public adapter 00020 7537 647951 nat 1 ip from any to any in via ix1 # NAT all incoming traffic 00030 0 0 check-state # For some reason, this never gets matched even though rule #100 is matched 00100 161 124340 skipto 805 tcp from any to any out via ix1 setup keep-state # For testing purposes, allow all TCP originating from the box out of the public adapter 00110 0 0 skipto 805 icmp from any to any out via ix1 keep-state 00200 36557 1996626 skipto 500 tcp from any to 172.16.1.2 dst-port 443 in via ix1 # Forward NAT'ed traffic for port 443 over the ip tunnel 00201 46593 63973143 skipto 805 tcp from 172.16.1.2 443 to any out via ix1 00400 8 6192 deny ip from any to any via ix1 00500 0 0 pipe 1 ip from any to any in via ix1 # Packet shaping 00501 0 0 allow ip from any to any in via ix1 00805 8963 3412995 nat 1 ip from any to any out via ix1 00806 8963 3412995 allow ip from any to any 10000 0 0 deny ip from any to any via ix1 # Last ditch catch 65535 864357 867120912 allow ip from any to any 'ipfw nat show config' yields: ipfw nat 1 config if ix1 log reset redirect_port tcp 172.16.1.2:443 :443 And finally, here are the horrifying ping times (furthermore, all outgoing TCP traffic originating from this box, such as wget or pkg_add, time out. I've managed to get an outgoing telnet working, but it's horrible slow and takes a while to establish): PING google.com (74.125.227.14): 56 data bytes 64 bytes from 74.125.227.14: icmp_seq=0 ttl=56 time=2746.953 ms 64 bytes from 74.125.227.14: icmp_seq=1 ttl=56 time=2097.460 ms 64 bytes from 74.125.227.14: icmp_seq=2 ttl=56 time=2186.068 ms 64 bytes from 74.125.227.14: icmp_seq=3 ttl=56 time=4292.776 ms 64 bytes from 74.125.227.14: icmp_seq=4 ttl=56 time=5056.965 ms 64 bytes from 74.125.227.14: icmp_seq=5 ttl=56 time=5323.720 ms 64 bytes from 74.125.227.14: icmp_seq=6 ttl=56 time=5007.974 ms 64 bytes from 74.125.227.14: icmp_seq=7 ttl=56 time=4756.587 ms It's worth mentioning that when I switch back to using natd and divert in the ruleset (which really only changes the nat portions and everything else stays the same), the ping time drops to ~300ms, which is a big difference for simply "using" natd even when the ICMP packets aren't supposed to be going through NAT'ing whatsoever. The ~300ms ping time is still way too high, though, since our other boxes have a ping time to Google of ~0.300ms... Any ideas? On Thu, Sep 13, 2012 at 7:41 AM, Ian Smith wrote: > On Wed, 12 Sep 2012 23:09:27 -0500, Soren Dreijer wrote: > > Hi there, > > > > We're running freebsd 9.0-RELEASE on a box whose primary purpose is to > > act as a firewall and a gateway. Up until today, we've been using ipfw > > in conjunction with natd and the divert action in ipfw to forward > > packets between the freebsd box (i.e. the public Internet) and our > > private servers. > > > > Unfortunately, natd appears to be quite the CPU hog and we therefore > > decided to switch to the in-kernel NAT support in ipfw. The issue > > we're running in to is that the network latency appears to be > > skyrocketing when ipfw contains nat rules. Basically all TCP traffic > > originating from the box times out and pinging google.com on the box > > gives an average of ~10 SECONDS -- and that's even if I explicitly > > allow all ICMP traffic before the packets even get to the nat rules in > > ipfw. > > > > The really odd part, however, is that I can ping the freebsd box just > > fine externally. For instance, pinging the server from my home > > connection gives an average of 45 ms. I'm also able to communicate > > just fine with the internal servers through the freebsd box. > > > > Does anybody have any idea what's going on? I assume I must've > > misconfigured something big here... > > Or maybe only something small .. but without seeing your basic ruleset > and network config - obscured as need be - we can only guess. Maybe an > 'ifconfig', 'ipfw show' and 'ipfw nat show config' would illustrate? > > cheers, Ian From owner-freebsd-ipfw@FreeBSD.ORG Thu Sep 13 16:10:35 2012 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 406F31065674 for ; Thu, 13 Sep 2012 16:10:35 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 9E98A8FC0C for ; Thu, 13 Sep 2012 16:10:34 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 187347300A; Thu, 13 Sep 2012 18:30:13 +0200 (CEST) Date: Thu, 13 Sep 2012 18:30:13 +0200 From: Luigi Rizzo To: Soren Dreijer Message-ID: <20120913163013.GA22049@onelab2.iet.unipi.it> References: <20120913221758.E51539@sola.nimnet.asn.au> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: freebsd-ipfw@freebsd.org, Ian Smith Subject: Re: Significant network latency when using ipfw and in-kernel NAT X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Sep 2012 16:10:35 -0000 [top posting for readability] i have seen this kind of issues related to bad interaction between the nat code and the various accelerations (mostly TSO/RSC, but i would also try to disable the checksums). Try to remove tso,csum, possibly rsc if you have it, and see if the problem continues. Please post the result so people reading this thread in the future can tell whether my suggestion was useful or not. cheers luigi On Thu, Sep 13, 2012 at 10:48:01AM -0500, Soren Dreijer wrote: > Definitely. Since this is a server in production, I've obfuscated some > of the IPs, etc. > > First off, here's the ifconfig. Our setup consists of a private (ix0) > and a public nic (ix1) and an ip tunnel (gif0), which is what we use > in ipfw to forward incoming packets to our internal boxes: > > ix0: flags=8843 metric 0 mtu 1500 > options=401bb > ether XX:XX:XX:XX:XX:XX > inet netmask 0xffffffc0 broadcast xx > inet6 xxxx::xxx:xxxx:xxxx:xxxx%ix0 prefixlen 64 scopeid 0x7 > nd6 options=29 > media: Ethernet autoselect (10Gbase-Twinax ) > status: active > ix1: flags=8843 metric 0 mtu 1500 > options=400bb > ether XX:XX:XX:XX:XX:XX > inet netmask 0xfffffff8 broadcast xx > inet6 xxxx::xxx:xxxx:xxxx:xxxx%ix1 prefixlen 64 scopeid 0x8 > inet netmask 0xffffffff broadcast xx > inet netmask 0xffffffff broadcast xx > nd6 options=29 > media: Ethernet autoselect (10Gbase-Twinax ) > status: active > ipfw0: flags=8801 metric 0 mtu 65536 > nd6 options=29 > lo0: flags=8049 metric 0 mtu 16384 > options=3 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa > inet 127.0.0.1 netmask 0xff000000 > nd6 options=21 > gif0: flags=8051 metric 0 mtu 1500 > tunnel inet --> > inet 172.16.1.1 --> 172.16.1.2 netmask 0xffff0000 > nd6 options=29 > options=1 > > The basic ruleset looks like this. One-pass is off so that packets are > reinjected after going through NAT'ing and pipes: > > 00001 16653 4417407 allow ip from any to any via ix0 > 00003 14588 2860344 allow ip from any to any via gif1 > 00006 0 0 allow ip from any to any via lo0 > 00010 0 0 deny ip from 192.168.0.0/16 to any in via ix1 > 00011 0 0 deny ip from 172.16.0.0/12 to any in via ix1 > 00012 0 0 deny ip from 10.0.0.0/8 to any in via ix1 > 00013 0 0 deny ip from 127.0.0.0/8 to any in via ix1 > 00014 0 0 deny ip from 0.0.0.0/8 to any in via ix1 > 00015 0 0 deny ip from 169.254.0.0/16 to any in via ix1 > 00016 0 0 deny ip from 192.0.2.0/24 to any in via ix1 > 00017 0 0 deny ip from 204.152.64.0/23 to any in via ix1 > 00018 0 0 deny ip from 224.0.0.0/3 to any in via ix1 > 00019 15 1020 allow icmp from any to any via ix1 # For > testing purposes, allow all ICMP in and out of the public adapter > 00020 7537 647951 nat 1 ip from any to any in via ix1 # NAT all > incoming traffic > 00030 0 0 check-state # For some reason, this never gets > matched even though rule #100 is matched > 00100 161 124340 skipto 805 tcp from any to any out via ix1 > setup keep-state # For testing purposes, allow all TCP originating > from the box out of the public adapter > 00110 0 0 skipto 805 icmp from any to any out via ix1 keep-state > 00200 36557 1996626 skipto 500 tcp from any to 172.16.1.2 dst-port > 443 in via ix1 # Forward NAT'ed traffic for port 443 over the ip > tunnel > 00201 46593 63973143 skipto 805 tcp from 172.16.1.2 443 to any out via ix1 > 00400 8 6192 deny ip from any to any via ix1 > 00500 0 0 pipe 1 ip from any to any in via ix1 # Packet shaping > 00501 0 0 allow ip from any to any in via ix1 > 00805 8963 3412995 nat 1 ip from any to any out via ix1 > 00806 8963 3412995 allow ip from any to any > 10000 0 0 deny ip from any to any via ix1 # Last ditch catch > 65535 864357 867120912 allow ip from any to any > > 'ipfw nat show config' yields: > > ipfw nat 1 config if ix1 log reset redirect_port tcp 172.16.1.2:443 > :443 > > And finally, here are the horrifying ping times (furthermore, all > outgoing TCP traffic originating from this box, such as wget or > pkg_add, time out. I've managed to get an outgoing telnet working, but > it's horrible slow and takes a while to establish): > > PING google.com (74.125.227.14): 56 data bytes > 64 bytes from 74.125.227.14: icmp_seq=0 ttl=56 time=2746.953 ms > 64 bytes from 74.125.227.14: icmp_seq=1 ttl=56 time=2097.460 ms > 64 bytes from 74.125.227.14: icmp_seq=2 ttl=56 time=2186.068 ms > 64 bytes from 74.125.227.14: icmp_seq=3 ttl=56 time=4292.776 ms > 64 bytes from 74.125.227.14: icmp_seq=4 ttl=56 time=5056.965 ms > 64 bytes from 74.125.227.14: icmp_seq=5 ttl=56 time=5323.720 ms > 64 bytes from 74.125.227.14: icmp_seq=6 ttl=56 time=5007.974 ms > 64 bytes from 74.125.227.14: icmp_seq=7 ttl=56 time=4756.587 ms > > It's worth mentioning that when I switch back to using natd and divert > in the ruleset (which really only changes the nat portions and > everything else stays the same), the ping time drops to ~300ms, which > is a big difference for simply "using" natd even when the ICMP packets > aren't supposed to be going through NAT'ing whatsoever. The ~300ms > ping time is still way too high, though, since our other boxes have a > ping time to Google of ~0.300ms... > > Any ideas? > > On Thu, Sep 13, 2012 at 7:41 AM, Ian Smith wrote: > > On Wed, 12 Sep 2012 23:09:27 -0500, Soren Dreijer wrote: > > > Hi there, > > > > > > We're running freebsd 9.0-RELEASE on a box whose primary purpose is to > > > act as a firewall and a gateway. Up until today, we've been using ipfw > > > in conjunction with natd and the divert action in ipfw to forward > > > packets between the freebsd box (i.e. the public Internet) and our > > > private servers. > > > > > > Unfortunately, natd appears to be quite the CPU hog and we therefore > > > decided to switch to the in-kernel NAT support in ipfw. The issue > > > we're running in to is that the network latency appears to be > > > skyrocketing when ipfw contains nat rules. Basically all TCP traffic > > > originating from the box times out and pinging google.com on the box > > > gives an average of ~10 SECONDS -- and that's even if I explicitly > > > allow all ICMP traffic before the packets even get to the nat rules in > > > ipfw. > > > > > > The really odd part, however, is that I can ping the freebsd box just > > > fine externally. For instance, pinging the server from my home > > > connection gives an average of 45 ms. I'm also able to communicate > > > just fine with the internal servers through the freebsd box. > > > > > > Does anybody have any idea what's going on? I assume I must've > > > misconfigured something big here... > > > > Or maybe only something small .. but without seeing your basic ruleset > > and network config - obscured as need be - we can only guess. Maybe an > > 'ifconfig', 'ipfw show' and 'ipfw nat show config' would illustrate? > > > > cheers, Ian > _______________________________________________ > freebsd-ipfw@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org" From owner-freebsd-ipfw@FreeBSD.ORG Thu Sep 13 16:11:09 2012 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 003381065670 for ; Thu, 13 Sep 2012 16:11:08 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) by mx1.freebsd.org (Postfix) with ESMTP id 53AFB8FC15 for ; Thu, 13 Sep 2012 16:11:07 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id q8DGAuYQ098336; Fri, 14 Sep 2012 02:10:56 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Fri, 14 Sep 2012 02:10:56 +1000 (EST) From: Ian Smith To: Soren Dreijer In-Reply-To: Message-ID: <20120914020023.K51539@sola.nimnet.asn.au> References: <20120913221758.E51539@sola.nimnet.asn.au> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-ipfw@freebsd.org Subject: Re: Significant network latency when using ipfw and in-kernel NAT X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Sep 2012 16:11:09 -0000 On Thu, 13 Sep 2012 0:48:01 -0500, Soren Dreijer wrote: > Definitely. Since this is a server in production, I've obfuscated some > of the IPs, etc. > > First off, here's the ifconfig. Our setup consists of a private (ix0) > and a public nic (ix1) and an ip tunnel (gif0), which is what we use > in ipfw to forward incoming packets to our internal boxes: > > ix0: flags=8843 metric 0 mtu 1500 > options=401bb > ether XX:XX:XX:XX:XX:XX > inet netmask 0xffffffc0 broadcast xx > inet6 xxxx::xxx:xxxx:xxxx:xxxx%ix0 prefixlen 64 scopeid 0x7 > nd6 options=29 > media: Ethernet autoselect (10Gbase-Twinax ) > status: active > ix1: flags=8843 metric 0 mtu 1500 > options=400bb > ether XX:XX:XX:XX:XX:XX > inet netmask 0xfffffff8 broadcast xx > inet6 xxxx::xxx:xxxx:xxxx:xxxx%ix1 prefixlen 64 scopeid 0x8 > inet netmask 0xffffffff broadcast xx > inet netmask 0xffffffff broadcast xx > nd6 options=29 > media: Ethernet autoselect (10Gbase-Twinax ) > status: active [ Soren and I had some off-list discussion which doesn't seem to have helped matters, but I'll repost this as the only clue I had. Anybody? ] Before anything else .. % man ipfw | tail | head -4 Due to the architecture of libalias(3), ipfw nat is not compatible with the TCP segmentation offloading (TSO). Thus, to reliably nat your net- work traffic, please disable TSO on your NICs using ifconfig(8). I don't know if this applies to VLAN_HWTSO, but likely to TSO4 on ix0? Do things change if you try disabling all TSO? I'd also change all 'out via ix1' to 'out xmit ix1', given the former also applies to traffic going out anywhere that came _in_ on ix1. cheers, Ian From owner-freebsd-ipfw@FreeBSD.ORG Thu Sep 13 17:01:57 2012 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 957AD10656B0 for ; Thu, 13 Sep 2012 17:01:57 +0000 (UTC) (envelope-from dreijer@echobit.net) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 40E848FC14 for ; Thu, 13 Sep 2012 17:01:56 +0000 (UTC) Received: by obbun3 with SMTP id un3so6057711obb.13 for ; Thu, 13 Sep 2012 10:01:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :x-gm-message-state; bh=BzPnpNT0fWyLJxMEPakTJc4+ztTb0fZfiAR4GRuBN8E=; b=LXLdnrnIJU9X8fS5QeN7V6/PW/lG6Grwb5A9LPTHUr2bggxoThDEAAKhsa2dcZ7XdZ YYFkLuxWQqs8ecLIGscZspQT0GIb1O+upQbk61iNX6Ixb6jLKO7XKvKY6npleX4IoWoS lUCvtj8GlJD2VjMFUbZ/gTJbEYrJIAzWihr0LopOm5xA5C+xpGnL8bd6nJGumvUc1X6Y doF/Uiuzf/sDKZ1n1moBGfO+15ldQLge9yANbJpiZBBgZ3rP79u32M2PjBi0P8d1/6R2 NeKUoIFjh1SCRJIyR1nkmchm85QiH68VMw1BL0SH9xeTrhSE+vCcZ92W9m7lqBPZQ3YW svHQ== MIME-Version: 1.0 Received: by 10.182.37.41 with SMTP id v9mr3473977obj.23.1347555716154; Thu, 13 Sep 2012 10:01:56 -0700 (PDT) Sender: dreijer@echobit.net Received: by 10.76.99.75 with HTTP; Thu, 13 Sep 2012 10:01:56 -0700 (PDT) In-Reply-To: <20120913163013.GA22049@onelab2.iet.unipi.it> References: <20120913221758.E51539@sola.nimnet.asn.au> <20120913163013.GA22049@onelab2.iet.unipi.it> Date: Thu, 13 Sep 2012 12:01:56 -0500 X-Google-Sender-Auth: 19Wo_zz_7dNvZNJmLy5HcuRm1Zo Message-ID: From: Soren Dreijer To: Luigi Rizzo Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQnCmltqQn1NssvdzUV4c17PAyahEPQLii4/Va6ohFOYwAO/ORNPx9DiMVFU/X4N7irfnTlr Cc: freebsd-ipfw@freebsd.org, Ian Smith Subject: Re: Significant network latency when using ipfw and in-kernel NAT X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Sep 2012 17:01:57 -0000 Luigi and Ian, As Ian mentioned, we had some off-list discussion by accident and he suggested the TSO approach too (although I don't know how that would affect e.g. ICMP traffic). It seems to have been a known issue for a while (http://lists.freebsd.org/pipermail/freebsd-net/2010-July/025743.html). Does anybody know if this is still the case in 9-0-RELEASE? I've already done "ifconfig ix1 -tso" to disable TSO on the public nic, but there was no difference. I'm not sure what VLAN_HWTSO means, though. Is the nic doing TSO on its own? Do I need to turn that off as well?. also, do I need to turn off TSO on ix0, which is what the ip tunnel runs over? Thanks, Soren On Thu, Sep 13, 2012 at 11:30 AM, Luigi Rizzo wrote: > > [top posting for readability] > i have seen this kind of issues related to bad interaction > between the nat code and the various accelerations > (mostly TSO/RSC, but i would also try to disable the > checksums). > Try to remove tso,csum, possibly rsc if you have it, and see > if the problem continues. Please post the result so people > reading this thread in the future can tell whether my suggestion > was useful or not. > > cheers > luigi > > > On Thu, Sep 13, 2012 at 10:48:01AM -0500, Soren Dreijer wrote: >> Definitely. Since this is a server in production, I've obfuscated some >> of the IPs, etc. >> >> First off, here's the ifconfig. Our setup consists of a private (ix0) >> and a public nic (ix1) and an ip tunnel (gif0), which is what we use >> in ipfw to forward incoming packets to our internal boxes: >> >> ix0: flags=8843 metric 0 mtu 1500 >> options=401bb >> ether XX:XX:XX:XX:XX:XX >> inet netmask 0xffffffc0 broadcast xx >> inet6 xxxx::xxx:xxxx:xxxx:xxxx%ix0 prefixlen 64 scopeid 0x7 >> nd6 options=29 >> media: Ethernet autoselect (10Gbase-Twinax ) >> status: active >> ix1: flags=8843 metric 0 mtu 1500 >> options=400bb >> ether XX:XX:XX:XX:XX:XX >> inet netmask 0xfffffff8 broadcast xx >> inet6 xxxx::xxx:xxxx:xxxx:xxxx%ix1 prefixlen 64 scopeid 0x8 >> inet netmask 0xffffffff broadcast xx >> inet netmask 0xffffffff broadcast xx >> nd6 options=29 >> media: Ethernet autoselect (10Gbase-Twinax ) >> status: active >> ipfw0: flags=8801 metric 0 mtu 65536 >> nd6 options=29 >> lo0: flags=8049 metric 0 mtu 16384 >> options=3 >> inet6 ::1 prefixlen 128 >> inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa >> inet 127.0.0.1 netmask 0xff000000 >> nd6 options=21 >> gif0: flags=8051 metric 0 mtu 1500 >> tunnel inet --> >> inet 172.16.1.1 --> 172.16.1.2 netmask 0xffff0000 >> nd6 options=29 >> options=1 >> >> The basic ruleset looks like this. One-pass is off so that packets are >> reinjected after going through NAT'ing and pipes: >> >> 00001 16653 4417407 allow ip from any to any via ix0 >> 00003 14588 2860344 allow ip from any to any via gif1 >> 00006 0 0 allow ip from any to any via lo0 >> 00010 0 0 deny ip from 192.168.0.0/16 to any in via ix1 >> 00011 0 0 deny ip from 172.16.0.0/12 to any in via ix1 >> 00012 0 0 deny ip from 10.0.0.0/8 to any in via ix1 >> 00013 0 0 deny ip from 127.0.0.0/8 to any in via ix1 >> 00014 0 0 deny ip from 0.0.0.0/8 to any in via ix1 >> 00015 0 0 deny ip from 169.254.0.0/16 to any in via ix1 >> 00016 0 0 deny ip from 192.0.2.0/24 to any in via ix1 >> 00017 0 0 deny ip from 204.152.64.0/23 to any in via ix1 >> 00018 0 0 deny ip from 224.0.0.0/3 to any in via ix1 >> 00019 15 1020 allow icmp from any to any via ix1 # For >> testing purposes, allow all ICMP in and out of the public adapter >> 00020 7537 647951 nat 1 ip from any to any in via ix1 # NAT all >> incoming traffic >> 00030 0 0 check-state # For some reason, this never gets >> matched even though rule #100 is matched >> 00100 161 124340 skipto 805 tcp from any to any out via ix1 >> setup keep-state # For testing purposes, allow all TCP originating >> from the box out of the public adapter >> 00110 0 0 skipto 805 icmp from any to any out via ix1 keep-state >> 00200 36557 1996626 skipto 500 tcp from any to 172.16.1.2 dst-port >> 443 in via ix1 # Forward NAT'ed traffic for port 443 over the ip >> tunnel >> 00201 46593 63973143 skipto 805 tcp from 172.16.1.2 443 to any out via ix1 >> 00400 8 6192 deny ip from any to any via ix1 >> 00500 0 0 pipe 1 ip from any to any in via ix1 # Packet shaping >> 00501 0 0 allow ip from any to any in via ix1 >> 00805 8963 3412995 nat 1 ip from any to any out via ix1 >> 00806 8963 3412995 allow ip from any to any >> 10000 0 0 deny ip from any to any via ix1 # Last ditch catch >> 65535 864357 867120912 allow ip from any to any >> >> 'ipfw nat show config' yields: >> >> ipfw nat 1 config if ix1 log reset redirect_port tcp 172.16.1.2:443 >> :443 >> >> And finally, here are the horrifying ping times (furthermore, all >> outgoing TCP traffic originating from this box, such as wget or >> pkg_add, time out. I've managed to get an outgoing telnet working, but >> it's horrible slow and takes a while to establish): >> >> PING google.com (74.125.227.14): 56 data bytes >> 64 bytes from 74.125.227.14: icmp_seq=0 ttl=56 time=2746.953 ms >> 64 bytes from 74.125.227.14: icmp_seq=1 ttl=56 time=2097.460 ms >> 64 bytes from 74.125.227.14: icmp_seq=2 ttl=56 time=2186.068 ms >> 64 bytes from 74.125.227.14: icmp_seq=3 ttl=56 time=4292.776 ms >> 64 bytes from 74.125.227.14: icmp_seq=4 ttl=56 time=5056.965 ms >> 64 bytes from 74.125.227.14: icmp_seq=5 ttl=56 time=5323.720 ms >> 64 bytes from 74.125.227.14: icmp_seq=6 ttl=56 time=5007.974 ms >> 64 bytes from 74.125.227.14: icmp_seq=7 ttl=56 time=4756.587 ms >> >> It's worth mentioning that when I switch back to using natd and divert >> in the ruleset (which really only changes the nat portions and >> everything else stays the same), the ping time drops to ~300ms, which >> is a big difference for simply "using" natd even when the ICMP packets >> aren't supposed to be going through NAT'ing whatsoever. The ~300ms >> ping time is still way too high, though, since our other boxes have a >> ping time to Google of ~0.300ms... >> >> Any ideas? >> >> On Thu, Sep 13, 2012 at 7:41 AM, Ian Smith wrote: >> > On Wed, 12 Sep 2012 23:09:27 -0500, Soren Dreijer wrote: >> > > Hi there, >> > > >> > > We're running freebsd 9.0-RELEASE on a box whose primary purpose is to >> > > act as a firewall and a gateway. Up until today, we've been using ipfw >> > > in conjunction with natd and the divert action in ipfw to forward >> > > packets between the freebsd box (i.e. the public Internet) and our >> > > private servers. >> > > >> > > Unfortunately, natd appears to be quite the CPU hog and we therefore >> > > decided to switch to the in-kernel NAT support in ipfw. The issue >> > > we're running in to is that the network latency appears to be >> > > skyrocketing when ipfw contains nat rules. Basically all TCP traffic >> > > originating from the box times out and pinging google.com on the box >> > > gives an average of ~10 SECONDS -- and that's even if I explicitly >> > > allow all ICMP traffic before the packets even get to the nat rules in >> > > ipfw. >> > > >> > > The really odd part, however, is that I can ping the freebsd box just >> > > fine externally. For instance, pinging the server from my home >> > > connection gives an average of 45 ms. I'm also able to communicate >> > > just fine with the internal servers through the freebsd box. >> > > >> > > Does anybody have any idea what's going on? I assume I must've >> > > misconfigured something big here... >> > >> > Or maybe only something small .. but without seeing your basic ruleset >> > and network config - obscured as need be - we can only guess. Maybe an >> > 'ifconfig', 'ipfw show' and 'ipfw nat show config' would illustrate? >> > >> > cheers, Ian >> _______________________________________________ >> freebsd-ipfw@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw >> To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org" From owner-freebsd-ipfw@FreeBSD.ORG Thu Sep 13 17:26:34 2012 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E953B106564A for ; Thu, 13 Sep 2012 17:26:33 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 52A918FC12 for ; Thu, 13 Sep 2012 17:26:32 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 9D16F7300B; Thu, 13 Sep 2012 19:46:12 +0200 (CEST) Date: Thu, 13 Sep 2012 19:46:12 +0200 From: Luigi Rizzo To: Soren Dreijer Message-ID: <20120913174612.GB22571@onelab2.iet.unipi.it> References: <20120913221758.E51539@sola.nimnet.asn.au> <20120913163013.GA22049@onelab2.iet.unipi.it> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: freebsd-ipfw@freebsd.org, Ian Smith Subject: Re: Significant network latency when using ipfw and in-kernel NAT X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Sep 2012 17:26:34 -0000 On Thu, Sep 13, 2012 at 12:01:56PM -0500, Soren Dreijer wrote: > Luigi and Ian, > > As Ian mentioned, we had some off-list discussion by accident and he > suggested the TSO approach too (although I don't know how that would > affect e.g. ICMP traffic). It seems to have been a known issue for a > while (http://lists.freebsd.org/pipermail/freebsd-net/2010-July/025743.html). > Does anybody know if this is still the case in 9-0-RELEASE? > > I've already done "ifconfig ix1 -tso" to disable TSO on the public > nic, but there was no difference. I'm not sure what VLAN_HWTSO means, > though. Is the nic doing TSO on its own? Do I need to turn that off as > well?. also, do I need to turn off TSO on ix0, which is what the ip > tunnel runs over? i'd start by disabling all accelerations (and jumobgrams) and then move on from the results to figure out where is the problem. When the nat code was written it assumed well-formed 1500-byte packets, and it uses the checksums when rebuilding the headers. TSO/RSC can generate large segments causing buffer overflows, whereas the *XCSUM can generate invalid packets that are sometimes recovered by retransmissions. cheers luigi > Thanks, > Soren > > On Thu, Sep 13, 2012 at 11:30 AM, Luigi Rizzo wrote: > > > > [top posting for readability] > > i have seen this kind of issues related to bad interaction > > between the nat code and the various accelerations > > (mostly TSO/RSC, but i would also try to disable the > > checksums). > > Try to remove tso,csum, possibly rsc if you have it, and see > > if the problem continues. Please post the result so people > > reading this thread in the future can tell whether my suggestion > > was useful or not. > > > > cheers > > luigi > > > > > > On Thu, Sep 13, 2012 at 10:48:01AM -0500, Soren Dreijer wrote: > >> Definitely. Since this is a server in production, I've obfuscated some > >> of the IPs, etc. > >> > >> First off, here's the ifconfig. Our setup consists of a private (ix0) > >> and a public nic (ix1) and an ip tunnel (gif0), which is what we use > >> in ipfw to forward incoming packets to our internal boxes: > >> > >> ix0: flags=8843 metric 0 mtu 1500 > >> options=401bb > >> ether XX:XX:XX:XX:XX:XX > >> inet netmask 0xffffffc0 broadcast xx > >> inet6 xxxx::xxx:xxxx:xxxx:xxxx%ix0 prefixlen 64 scopeid 0x7 > >> nd6 options=29 > >> media: Ethernet autoselect (10Gbase-Twinax ) > >> status: active > >> ix1: flags=8843 metric 0 mtu 1500 > >> options=400bb > >> ether XX:XX:XX:XX:XX:XX > >> inet netmask 0xfffffff8 broadcast xx > >> inet6 xxxx::xxx:xxxx:xxxx:xxxx%ix1 prefixlen 64 scopeid 0x8 > >> inet netmask 0xffffffff broadcast xx > >> inet netmask 0xffffffff broadcast xx > >> nd6 options=29 > >> media: Ethernet autoselect (10Gbase-Twinax ) > >> status: active > >> ipfw0: flags=8801 metric 0 mtu 65536 > >> nd6 options=29 > >> lo0: flags=8049 metric 0 mtu 16384 > >> options=3 > >> inet6 ::1 prefixlen 128 > >> inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa > >> inet 127.0.0.1 netmask 0xff000000 > >> nd6 options=21 > >> gif0: flags=8051 metric 0 mtu 1500 > >> tunnel inet --> > >> inet 172.16.1.1 --> 172.16.1.2 netmask 0xffff0000 > >> nd6 options=29 > >> options=1 > >> > >> The basic ruleset looks like this. One-pass is off so that packets are > >> reinjected after going through NAT'ing and pipes: > >> > >> 00001 16653 4417407 allow ip from any to any via ix0 > >> 00003 14588 2860344 allow ip from any to any via gif1 > >> 00006 0 0 allow ip from any to any via lo0 > >> 00010 0 0 deny ip from 192.168.0.0/16 to any in via ix1 > >> 00011 0 0 deny ip from 172.16.0.0/12 to any in via ix1 > >> 00012 0 0 deny ip from 10.0.0.0/8 to any in via ix1 > >> 00013 0 0 deny ip from 127.0.0.0/8 to any in via ix1 > >> 00014 0 0 deny ip from 0.0.0.0/8 to any in via ix1 > >> 00015 0 0 deny ip from 169.254.0.0/16 to any in via ix1 > >> 00016 0 0 deny ip from 192.0.2.0/24 to any in via ix1 > >> 00017 0 0 deny ip from 204.152.64.0/23 to any in via ix1 > >> 00018 0 0 deny ip from 224.0.0.0/3 to any in via ix1 > >> 00019 15 1020 allow icmp from any to any via ix1 # For > >> testing purposes, allow all ICMP in and out of the public adapter > >> 00020 7537 647951 nat 1 ip from any to any in via ix1 # NAT all > >> incoming traffic > >> 00030 0 0 check-state # For some reason, this never gets > >> matched even though rule #100 is matched > >> 00100 161 124340 skipto 805 tcp from any to any out via ix1 > >> setup keep-state # For testing purposes, allow all TCP originating > >> from the box out of the public adapter > >> 00110 0 0 skipto 805 icmp from any to any out via ix1 keep-state > >> 00200 36557 1996626 skipto 500 tcp from any to 172.16.1.2 dst-port > >> 443 in via ix1 # Forward NAT'ed traffic for port 443 over the ip > >> tunnel > >> 00201 46593 63973143 skipto 805 tcp from 172.16.1.2 443 to any out via ix1 > >> 00400 8 6192 deny ip from any to any via ix1 > >> 00500 0 0 pipe 1 ip from any to any in via ix1 # Packet shaping > >> 00501 0 0 allow ip from any to any in via ix1 > >> 00805 8963 3412995 nat 1 ip from any to any out via ix1 > >> 00806 8963 3412995 allow ip from any to any > >> 10000 0 0 deny ip from any to any via ix1 # Last ditch catch > >> 65535 864357 867120912 allow ip from any to any > >> > >> 'ipfw nat show config' yields: > >> > >> ipfw nat 1 config if ix1 log reset redirect_port tcp 172.16.1.2:443 > >> :443 > >> > >> And finally, here are the horrifying ping times (furthermore, all > >> outgoing TCP traffic originating from this box, such as wget or > >> pkg_add, time out. I've managed to get an outgoing telnet working, but > >> it's horrible slow and takes a while to establish): > >> > >> PING google.com (74.125.227.14): 56 data bytes > >> 64 bytes from 74.125.227.14: icmp_seq=0 ttl=56 time=2746.953 ms > >> 64 bytes from 74.125.227.14: icmp_seq=1 ttl=56 time=2097.460 ms > >> 64 bytes from 74.125.227.14: icmp_seq=2 ttl=56 time=2186.068 ms > >> 64 bytes from 74.125.227.14: icmp_seq=3 ttl=56 time=4292.776 ms > >> 64 bytes from 74.125.227.14: icmp_seq=4 ttl=56 time=5056.965 ms > >> 64 bytes from 74.125.227.14: icmp_seq=5 ttl=56 time=5323.720 ms > >> 64 bytes from 74.125.227.14: icmp_seq=6 ttl=56 time=5007.974 ms > >> 64 bytes from 74.125.227.14: icmp_seq=7 ttl=56 time=4756.587 ms > >> > >> It's worth mentioning that when I switch back to using natd and divert > >> in the ruleset (which really only changes the nat portions and > >> everything else stays the same), the ping time drops to ~300ms, which > >> is a big difference for simply "using" natd even when the ICMP packets > >> aren't supposed to be going through NAT'ing whatsoever. The ~300ms > >> ping time is still way too high, though, since our other boxes have a > >> ping time to Google of ~0.300ms... > >> > >> Any ideas? > >> > >> On Thu, Sep 13, 2012 at 7:41 AM, Ian Smith wrote: > >> > On Wed, 12 Sep 2012 23:09:27 -0500, Soren Dreijer wrote: > >> > > Hi there, > >> > > > >> > > We're running freebsd 9.0-RELEASE on a box whose primary purpose is to > >> > > act as a firewall and a gateway. Up until today, we've been using ipfw > >> > > in conjunction with natd and the divert action in ipfw to forward > >> > > packets between the freebsd box (i.e. the public Internet) and our > >> > > private servers. > >> > > > >> > > Unfortunately, natd appears to be quite the CPU hog and we therefore > >> > > decided to switch to the in-kernel NAT support in ipfw. The issue > >> > > we're running in to is that the network latency appears to be > >> > > skyrocketing when ipfw contains nat rules. Basically all TCP traffic > >> > > originating from the box times out and pinging google.com on the box > >> > > gives an average of ~10 SECONDS -- and that's even if I explicitly > >> > > allow all ICMP traffic before the packets even get to the nat rules in > >> > > ipfw. > >> > > > >> > > The really odd part, however, is that I can ping the freebsd box just > >> > > fine externally. For instance, pinging the server from my home > >> > > connection gives an average of 45 ms. I'm also able to communicate > >> > > just fine with the internal servers through the freebsd box. > >> > > > >> > > Does anybody have any idea what's going on? I assume I must've > >> > > misconfigured something big here... > >> > > >> > Or maybe only something small .. but without seeing your basic ruleset > >> > and network config - obscured as need be - we can only guess. Maybe an > >> > 'ifconfig', 'ipfw show' and 'ipfw nat show config' would illustrate? > >> > > >> > cheers, Ian > >> _______________________________________________ > >> freebsd-ipfw@freebsd.org mailing list > >> http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > >> To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org" From owner-freebsd-ipfw@FreeBSD.ORG Thu Sep 13 17:37:24 2012 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9DF64106566C for ; Thu, 13 Sep 2012 17:37:24 +0000 (UTC) (envelope-from dreijer@echobit.net) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 52A808FC17 for ; Thu, 13 Sep 2012 17:37:24 +0000 (UTC) Received: by obbun3 with SMTP id un3so6136290obb.13 for ; Thu, 13 Sep 2012 10:37:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :x-gm-message-state; bh=xw5zZvgMqh4EZbLF+kDRitCbLOKFhutzdR1BAxdVhpM=; b=VCL8qTmuLD3AKcfRf98XFQGYnRmb68PopVV5C6mCOorWY86S0Gu3uk65pKXjZkedcy uFKcVih29ohEf/H7zsl8R20qlhyXqiYNW6ziVzShRq86MbrHspqN90f1zu9Fz3idUBEj UoYBeNe8XM3KWfoJBbPNWi/DtujyEZRNl2EorUhAkm1CMEEe8vMFAToMtmsf4Lo3id+b cl6nOyxC30DyMddXnOuVTM34Vv24scQLm+HYnXgtkYAX3KrJYj6KHaLKlhMb2ZVJlhcu ENk14Y+F+W/5hd1z4ZYeNsR9KXiPb6h7a3E2ug4za0UKEeVGLi0DanLK6nbVcSWZArYb QcXQ== MIME-Version: 1.0 Received: by 10.60.20.69 with SMTP id l5mr3499442oee.114.1347557843823; Thu, 13 Sep 2012 10:37:23 -0700 (PDT) Sender: dreijer@echobit.net Received: by 10.76.99.75 with HTTP; Thu, 13 Sep 2012 10:37:23 -0700 (PDT) In-Reply-To: <20120913174612.GB22571@onelab2.iet.unipi.it> References: <20120913221758.E51539@sola.nimnet.asn.au> <20120913163013.GA22049@onelab2.iet.unipi.it> <20120913174612.GB22571@onelab2.iet.unipi.it> Date: Thu, 13 Sep 2012 12:37:23 -0500 X-Google-Sender-Auth: RlhW-k_sLHa4RSp63B-js-bzl70 Message-ID: From: Soren Dreijer To: Luigi Rizzo Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQm5evYHtaCARjUSgkLTuIwwkg6riqtjjFvgO9UV87g6seWJGs7ZbZODh+7HQCXKC645xYct Cc: freebsd-ipfw@freebsd.org, Ian Smith Subject: Re: Significant network latency when using ipfw and in-kernel NAT X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Sep 2012 17:37:24 -0000 > i'd start by disabling all accelerations (and jumobgrams) > and then move on from the results to figure out where is the problem. So, I went ahead and disabled TSO on ix0. That seemed to fix the intermittent connection issues I had been experiencing with keeping an XMPP connection alive to one of our internal boxes. It hasn't done anything for the ICMPs or TCP traffic originating from the FreeBSD box, of course. I'm very puzzled how I can ping the box just fine from my home connection, but I can't ping OUT of the box itself without seeing huge latency. Similarly, proxying XMPP traffic through the box to an internal server and proxying back the result is as fast as it should be, but trying a simple wget on the box times out due to heavy latency. / Soren On Thu, Sep 13, 2012 at 12:46 PM, Luigi Rizzo wrote: > On Thu, Sep 13, 2012 at 12:01:56PM -0500, Soren Dreijer wrote: >> Luigi and Ian, >> >> As Ian mentioned, we had some off-list discussion by accident and he >> suggested the TSO approach too (although I don't know how that would >> affect e.g. ICMP traffic). It seems to have been a known issue for a >> while (http://lists.freebsd.org/pipermail/freebsd-net/2010-July/025743.html). >> Does anybody know if this is still the case in 9-0-RELEASE? >> >> I've already done "ifconfig ix1 -tso" to disable TSO on the public >> nic, but there was no difference. I'm not sure what VLAN_HWTSO means, >> though. Is the nic doing TSO on its own? Do I need to turn that off as >> well?. also, do I need to turn off TSO on ix0, which is what the ip >> tunnel runs over? > > i'd start by disabling all accelerations (and jumobgrams) > and then move on from the results to figure out where is the problem. > > When the nat code was written it assumed well-formed > 1500-byte packets, and it uses the checksums when rebuilding the > headers. TSO/RSC can generate large segments causing buffer overflows, > whereas the *XCSUM can generate invalid packets that are sometimes > recovered by retransmissions. > > cheers > luigi > >> Thanks, >> Soren >> >> On Thu, Sep 13, 2012 at 11:30 AM, Luigi Rizzo wrote: >> > >> > [top posting for readability] >> > i have seen this kind of issues related to bad interaction >> > between the nat code and the various accelerations >> > (mostly TSO/RSC, but i would also try to disable the >> > checksums). >> > Try to remove tso,csum, possibly rsc if you have it, and see >> > if the problem continues. Please post the result so people >> > reading this thread in the future can tell whether my suggestion >> > was useful or not. >> > >> > cheers >> > luigi >> > >> > >> > On Thu, Sep 13, 2012 at 10:48:01AM -0500, Soren Dreijer wrote: >> >> Definitely. Since this is a server in production, I've obfuscated some >> >> of the IPs, etc. >> >> >> >> First off, here's the ifconfig. Our setup consists of a private (ix0) >> >> and a public nic (ix1) and an ip tunnel (gif0), which is what we use >> >> in ipfw to forward incoming packets to our internal boxes: >> >> >> >> ix0: flags=8843 metric 0 mtu 1500 >> >> options=401bb >> >> ether XX:XX:XX:XX:XX:XX >> >> inet netmask 0xffffffc0 broadcast xx >> >> inet6 xxxx::xxx:xxxx:xxxx:xxxx%ix0 prefixlen 64 scopeid 0x7 >> >> nd6 options=29 >> >> media: Ethernet autoselect (10Gbase-Twinax ) >> >> status: active >> >> ix1: flags=8843 metric 0 mtu 1500 >> >> options=400bb >> >> ether XX:XX:XX:XX:XX:XX >> >> inet netmask 0xfffffff8 broadcast xx >> >> inet6 xxxx::xxx:xxxx:xxxx:xxxx%ix1 prefixlen 64 scopeid 0x8 >> >> inet netmask 0xffffffff broadcast xx >> >> inet netmask 0xffffffff broadcast xx >> >> nd6 options=29 >> >> media: Ethernet autoselect (10Gbase-Twinax ) >> >> status: active >> >> ipfw0: flags=8801 metric 0 mtu 65536 >> >> nd6 options=29 >> >> lo0: flags=8049 metric 0 mtu 16384 >> >> options=3 >> >> inet6 ::1 prefixlen 128 >> >> inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa >> >> inet 127.0.0.1 netmask 0xff000000 >> >> nd6 options=21 >> >> gif0: flags=8051 metric 0 mtu 1500 >> >> tunnel inet --> >> >> inet 172.16.1.1 --> 172.16.1.2 netmask 0xffff0000 >> >> nd6 options=29 >> >> options=1 >> >> >> >> The basic ruleset looks like this. One-pass is off so that packets are >> >> reinjected after going through NAT'ing and pipes: >> >> >> >> 00001 16653 4417407 allow ip from any to any via ix0 >> >> 00003 14588 2860344 allow ip from any to any via gif1 >> >> 00006 0 0 allow ip from any to any via lo0 >> >> 00010 0 0 deny ip from 192.168.0.0/16 to any in via ix1 >> >> 00011 0 0 deny ip from 172.16.0.0/12 to any in via ix1 >> >> 00012 0 0 deny ip from 10.0.0.0/8 to any in via ix1 >> >> 00013 0 0 deny ip from 127.0.0.0/8 to any in via ix1 >> >> 00014 0 0 deny ip from 0.0.0.0/8 to any in via ix1 >> >> 00015 0 0 deny ip from 169.254.0.0/16 to any in via ix1 >> >> 00016 0 0 deny ip from 192.0.2.0/24 to any in via ix1 >> >> 00017 0 0 deny ip from 204.152.64.0/23 to any in via ix1 >> >> 00018 0 0 deny ip from 224.0.0.0/3 to any in via ix1 >> >> 00019 15 1020 allow icmp from any to any via ix1 # For >> >> testing purposes, allow all ICMP in and out of the public adapter >> >> 00020 7537 647951 nat 1 ip from any to any in via ix1 # NAT all >> >> incoming traffic >> >> 00030 0 0 check-state # For some reason, this never gets >> >> matched even though rule #100 is matched >> >> 00100 161 124340 skipto 805 tcp from any to any out via ix1 >> >> setup keep-state # For testing purposes, allow all TCP originating >> >> from the box out of the public adapter >> >> 00110 0 0 skipto 805 icmp from any to any out via ix1 keep-state >> >> 00200 36557 1996626 skipto 500 tcp from any to 172.16.1.2 dst-port >> >> 443 in via ix1 # Forward NAT'ed traffic for port 443 over the ip >> >> tunnel >> >> 00201 46593 63973143 skipto 805 tcp from 172.16.1.2 443 to any out via ix1 >> >> 00400 8 6192 deny ip from any to any via ix1 >> >> 00500 0 0 pipe 1 ip from any to any in via ix1 # Packet shaping >> >> 00501 0 0 allow ip from any to any in via ix1 >> >> 00805 8963 3412995 nat 1 ip from any to any out via ix1 >> >> 00806 8963 3412995 allow ip from any to any >> >> 10000 0 0 deny ip from any to any via ix1 # Last ditch catch >> >> 65535 864357 867120912 allow ip from any to any >> >> >> >> 'ipfw nat show config' yields: >> >> >> >> ipfw nat 1 config if ix1 log reset redirect_port tcp 172.16.1.2:443 >> >> :443 >> >> >> >> And finally, here are the horrifying ping times (furthermore, all >> >> outgoing TCP traffic originating from this box, such as wget or >> >> pkg_add, time out. I've managed to get an outgoing telnet working, but >> >> it's horrible slow and takes a while to establish): >> >> >> >> PING google.com (74.125.227.14): 56 data bytes >> >> 64 bytes from 74.125.227.14: icmp_seq=0 ttl=56 time=2746.953 ms >> >> 64 bytes from 74.125.227.14: icmp_seq=1 ttl=56 time=2097.460 ms >> >> 64 bytes from 74.125.227.14: icmp_seq=2 ttl=56 time=2186.068 ms >> >> 64 bytes from 74.125.227.14: icmp_seq=3 ttl=56 time=4292.776 ms >> >> 64 bytes from 74.125.227.14: icmp_seq=4 ttl=56 time=5056.965 ms >> >> 64 bytes from 74.125.227.14: icmp_seq=5 ttl=56 time=5323.720 ms >> >> 64 bytes from 74.125.227.14: icmp_seq=6 ttl=56 time=5007.974 ms >> >> 64 bytes from 74.125.227.14: icmp_seq=7 ttl=56 time=4756.587 ms >> >> >> >> It's worth mentioning that when I switch back to using natd and divert >> >> in the ruleset (which really only changes the nat portions and >> >> everything else stays the same), the ping time drops to ~300ms, which >> >> is a big difference for simply "using" natd even when the ICMP packets >> >> aren't supposed to be going through NAT'ing whatsoever. The ~300ms >> >> ping time is still way too high, though, since our other boxes have a >> >> ping time to Google of ~0.300ms... >> >> >> >> Any ideas? >> >> >> >> On Thu, Sep 13, 2012 at 7:41 AM, Ian Smith wrote: >> >> > On Wed, 12 Sep 2012 23:09:27 -0500, Soren Dreijer wrote: >> >> > > Hi there, >> >> > > >> >> > > We're running freebsd 9.0-RELEASE on a box whose primary purpose is to >> >> > > act as a firewall and a gateway. Up until today, we've been using ipfw >> >> > > in conjunction with natd and the divert action in ipfw to forward >> >> > > packets between the freebsd box (i.e. the public Internet) and our >> >> > > private servers. >> >> > > >> >> > > Unfortunately, natd appears to be quite the CPU hog and we therefore >> >> > > decided to switch to the in-kernel NAT support in ipfw. The issue >> >> > > we're running in to is that the network latency appears to be >> >> > > skyrocketing when ipfw contains nat rules. Basically all TCP traffic >> >> > > originating from the box times out and pinging google.com on the box >> >> > > gives an average of ~10 SECONDS -- and that's even if I explicitly >> >> > > allow all ICMP traffic before the packets even get to the nat rules in >> >> > > ipfw. >> >> > > >> >> > > The really odd part, however, is that I can ping the freebsd box just >> >> > > fine externally. For instance, pinging the server from my home >> >> > > connection gives an average of 45 ms. I'm also able to communicate >> >> > > just fine with the internal servers through the freebsd box. >> >> > > >> >> > > Does anybody have any idea what's going on? I assume I must've >> >> > > misconfigured something big here... >> >> > >> >> > Or maybe only something small .. but without seeing your basic ruleset >> >> > and network config - obscured as need be - we can only guess. Maybe an >> >> > 'ifconfig', 'ipfw show' and 'ipfw nat show config' would illustrate? >> >> > >> >> > cheers, Ian >> >> _______________________________________________ >> >> freebsd-ipfw@freebsd.org mailing list >> >> http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw >> >> To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org" From owner-freebsd-ipfw@FreeBSD.ORG Fri Sep 14 05:00:54 2012 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AFF50106564A for ; Fri, 14 Sep 2012 05:00:54 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) by mx1.freebsd.org (Postfix) with ESMTP id 2F9A38FC08 for ; Fri, 14 Sep 2012 05:00:53 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id q8E50ls1026136; Fri, 14 Sep 2012 15:00:47 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Fri, 14 Sep 2012 15:00:47 +1000 (EST) From: Ian Smith To: Soren Dreijer In-Reply-To: Message-ID: <20120914144529.R51539@sola.nimnet.asn.au> References: <20120913221758.E51539@sola.nimnet.asn.au> <20120913163013.GA22049@onelab2.iet.unipi.it> <20120913174612.GB22571@onelab2.iet.unipi.it> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-ipfw@freebsd.org, Luigi Rizzo Subject: Re: Significant network latency when using ipfw and in-kernel NAT X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Sep 2012 05:00:54 -0000 On Thu, 13 Sep 2012 12:37:23 -0500, Soren Dreijer wrote: [Luigi Rizzo wrote:] > > i'd start by disabling all accelerations (and jumobgrams) > > and then move on from the results to figure out where is the problem. > > So, I went ahead and disabled TSO on ix0. That seemed to fix the > intermittent connection issues I had been experiencing with keeping an > XMPP connection alive to one of our internal boxes. It hasn't done > anything for the ICMPs or TCP traffic originating from the FreeBSD > box, of course. Please show ifconfig for ix0 and ix1 again after disabling tso, rxcsum, txcsum, vlanmtu, vlanhwtag, vlanhwfilter, vlanhwtso and any other configured accelerations, as Luigi recommended? Then we'd know if your problem was related to any of that, or not. cheers, Ian From owner-freebsd-ipfw@FreeBSD.ORG Fri Sep 14 14:12:28 2012 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5EE691065675 for ; Fri, 14 Sep 2012 14:12:28 +0000 (UTC) (envelope-from dreijer@echobit.net) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 165038FC0A for ; Fri, 14 Sep 2012 14:12:27 +0000 (UTC) Received: by obbun3 with SMTP id un3so7828505obb.13 for ; Fri, 14 Sep 2012 07:12:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :x-gm-message-state; bh=mn6c/dETvz/3/jaZnbq2uim+qOki5OV8YhbmIIUS7Zs=; b=WuUBYvqU74GbvlI00risGTpmQruwtowSOAHQyhL+RI6XxK2il0QSoPCYXFAoFbClAV zmemAVPptZF3SKR/HTO4jLMVIp5x7scPa1hlSF/eqqiWeQJ076tvoEP2fNYjYTNFfED/ 67ghagORTE0q/3XeP8ptm2yPhekJO8ZeCwX3tSfoxNOqsybwKOQo4JovjRsNM/2UtGyD cWPLux28S6R0IQUNIIfBly1Be5rMyQbyAyR79Ca56FgTY+WfI0CfDIE3qW+UIEGqHLpY 3iP6S5dS1A/4X+weP7ryDehZzG1IiRTCMrCowpGgeuOoZnFXQrL9tv9XuU8w+11065h3 DQJw== MIME-Version: 1.0 Received: by 10.60.4.163 with SMTP id l3mr3328948oel.74.1347631947388; Fri, 14 Sep 2012 07:12:27 -0700 (PDT) Sender: dreijer@echobit.net Received: by 10.76.99.75 with HTTP; Fri, 14 Sep 2012 07:12:27 -0700 (PDT) In-Reply-To: <20120914144529.R51539@sola.nimnet.asn.au> References: <20120913221758.E51539@sola.nimnet.asn.au> <20120913163013.GA22049@onelab2.iet.unipi.it> <20120913174612.GB22571@onelab2.iet.unipi.it> <20120914144529.R51539@sola.nimnet.asn.au> Date: Fri, 14 Sep 2012 09:12:27 -0500 X-Google-Sender-Auth: 08w8U61NJi5fgrGY3nmczrSXl94 Message-ID: From: Soren Dreijer To: Ian Smith Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQkzkxZ8SvxIV7PmmreOdzr2q+8slVcEgvx0yXV5ibXsUQIEmWzITiWHkRoLs9tEKiIWD8gO Cc: freebsd-ipfw@freebsd.org, Luigi Rizzo Subject: Re: Significant network latency when using ipfw and in-kernel NAT X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Sep 2012 14:12:28 -0000 Can anybody confirm that disabling these other options (rxcsum, txcsum, vlanmtu, vlanhwtag, vlanhwfilter, vlanhwtso) won't cause my adapter to lose its connectivity? This is a server in production and I'd rather not cause an outage if I can prevent it. :) On Fri, Sep 14, 2012 at 12:00 AM, Ian Smith wrote: > On Thu, 13 Sep 2012 12:37:23 -0500, Soren Dreijer wrote: > [Luigi Rizzo wrote:] > > > i'd start by disabling all accelerations (and jumobgrams) > > > and then move on from the results to figure out where is the problem. > > > > So, I went ahead and disabled TSO on ix0. That seemed to fix the > > intermittent connection issues I had been experiencing with keeping an > > XMPP connection alive to one of our internal boxes. It hasn't done > > anything for the ICMPs or TCP traffic originating from the FreeBSD > > box, of course. > > Please show ifconfig for ix0 and ix1 again after disabling tso, > rxcsum, txcsum, vlanmtu, vlanhwtag, vlanhwfilter, vlanhwtso > and any other configured accelerations, as Luigi recommended? > > Then we'd know if your problem was related to any of that, or not. > > cheers, Ian From owner-freebsd-ipfw@FreeBSD.ORG Fri Sep 14 17:59:16 2012 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B5FCE106564A for ; Fri, 14 Sep 2012 17:59:16 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) by mx1.freebsd.org (Postfix) with ESMTP id 2D4A38FC0A for ; Fri, 14 Sep 2012 17:59:15 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id q8EHx3Jb053722; Sat, 15 Sep 2012 03:59:03 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Sat, 15 Sep 2012 03:59:03 +1000 (EST) From: Ian Smith To: Soren Dreijer In-Reply-To: Message-ID: <20120915034627.V51539@sola.nimnet.asn.au> References: <20120913221758.E51539@sola.nimnet.asn.au> <20120913163013.GA22049@onelab2.iet.unipi.it> <20120913174612.GB22571@onelab2.iet.unipi.it> <20120914144529.R51539@sola.nimnet.asn.au> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-ipfw@freebsd.org, Luigi Rizzo Subject: Re: Significant network latency when using ipfw and in-kernel NAT X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Sep 2012 17:59:16 -0000 On Fri, 14 Sep 2012 09:12:27 -0500, Soren Dreijer wrote: > Can anybody confirm that disabling these other options (rxcsum, > txcsum, vlanmtu, vlanhwtag, vlanhwfilter, vlanhwtso) won't cause my > adapter to lose its connectivity? This is a server in production and > I'd rather not cause an outage if I can prevent it. :) Fair question Soren. I've configured no VLANs; out of my depth, again! cheers, Ian > On Fri, Sep 14, 2012 at 12:00 AM, Ian Smith wrote: > > On Thu, 13 Sep 2012 12:37:23 -0500, Soren Dreijer wrote: > > [Luigi Rizzo wrote:] > > > > i'd start by disabling all accelerations (and jumobgrams) > > > > and then move on from the results to figure out where is the problem. > > > > > > So, I went ahead and disabled TSO on ix0. That seemed to fix the > > > intermittent connection issues I had been experiencing with keeping an > > > XMPP connection alive to one of our internal boxes. It hasn't done > > > anything for the ICMPs or TCP traffic originating from the FreeBSD > > > box, of course. > > > > Please show ifconfig for ix0 and ix1 again after disabling tso, > > rxcsum, txcsum, vlanmtu, vlanhwtag, vlanhwfilter, vlanhwtso > > and any other configured accelerations, as Luigi recommended? > > > > Then we'd know if your problem was related to any of that, or not. > > > > cheers, Ian >