From owner-freebsd-ipfw@FreeBSD.ORG Sun Sep 16 16:46:08 2012 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2D53B106566C for ; Sun, 16 Sep 2012 16:46:08 +0000 (UTC) (envelope-from dreijer@echobit.net) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id D5E748FC12 for ; Sun, 16 Sep 2012 16:46:07 +0000 (UTC) Received: by obbun3 with SMTP id un3so10163980obb.13 for ; Sun, 16 Sep 2012 09:46:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :x-gm-message-state; bh=H54sZj9Tshhc6kqmfZKFfP0Vj+TPYeiPnLI0zT0jhAM=; b=c7O+d+qpMde3vodRR7Cq3oihBnPnQYZ9yHJRaE2uQjP+JcobcLSmpPvkzoWxV1cItR tcbjOecCQvv5CoCTwanMJo4O/XbH0q8bivpRn5DYgSSw8pqGkBs9rERRkFUYZKmpTYXd 7/oSc/9mU7pNLzN/AHdO96sK8zr4I1hrwRdy7ydtSfcPfOkniJzYHyhiN+OSiJlmqWC2 y+IDXC+C0W85c+5Y4jU4X06FyBxDjZu0fLHKQjZHKaA/ytf9Ny2m6vjeVpwJWKlmUinf Xr8huFiYQNHbHU+PLLjAVrBmHIzl5AnZuygpuZzz9RcoV2sXiuHtAE72tmQUb2WHmzlV SosQ== MIME-Version: 1.0 Received: by 10.60.171.69 with SMTP id as5mr9594308oec.100.1347813966099; Sun, 16 Sep 2012 09:46:06 -0700 (PDT) Sender: dreijer@echobit.net Received: by 10.76.99.75 with HTTP; Sun, 16 Sep 2012 09:46:05 -0700 (PDT) In-Reply-To: <20120915034627.V51539@sola.nimnet.asn.au> References: <20120913221758.E51539@sola.nimnet.asn.au> <20120913163013.GA22049@onelab2.iet.unipi.it> <20120913174612.GB22571@onelab2.iet.unipi.it> <20120914144529.R51539@sola.nimnet.asn.au> <20120915034627.V51539@sola.nimnet.asn.au> Date: Sun, 16 Sep 2012 11:46:05 -0500 X-Google-Sender-Auth: erZvGcfuRKXsh101xGqlxyLGOb8 Message-ID: From: Soren Dreijer To: Ian Smith Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQn6mXSgEvn6T/kteBvwKjLdErGJLuv3JgljgX7D81ADy2kqMaLafxpDhPKMUmitJ0+qMpyS Cc: freebsd-ipfw@freebsd.org, Luigi Rizzo Subject: Re: Significant network latency when using ipfw and in-kernel NAT X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Sep 2012 16:46:08 -0000 Just to follow up on this a bit: I haven't disabled any other options on the NICs yet due to high server load over the weekend, but I'll give it a go in the next few days. Also, it looks like pings to the box are now no longer as fast as I had previously stated. Pinging it from my home connection now yields >3 second roundtrip times, which neatly matches the ping time from the box itself to google.com. As I mentioned before, I'm not sure how e.g. rxcsum and txcsum have anything to do with high latency on ICMP traffic, so I'm wondering if we're perhaps barking up the wrong tree here (especially since forwarded traffic *through* the FreeBSD box seems to work just fine)? Thanks again for helping out here, guys. I'm in pretty deep water when it comes to issues like this one. / Soren On Fri, Sep 14, 2012 at 12:59 PM, Ian Smith wrote: > On Fri, 14 Sep 2012 09:12:27 -0500, Soren Dreijer wrote: > > > Can anybody confirm that disabling these other options (rxcsum, > > txcsum, vlanmtu, vlanhwtag, vlanhwfilter, vlanhwtso) won't cause my > > adapter to lose its connectivity? This is a server in production and > > I'd rather not cause an outage if I can prevent it. :) > > Fair question Soren. I've configured no VLANs; out of my depth, again! > > cheers, Ian > > > On Fri, Sep 14, 2012 at 12:00 AM, Ian Smith wrote: > > > On Thu, 13 Sep 2012 12:37:23 -0500, Soren Dreijer wrote: > > > [Luigi Rizzo wrote:] > > > > > i'd start by disabling all accelerations (and jumobgrams) > > > > > and then move on from the results to figure out where is the problem. > > > > > > > > So, I went ahead and disabled TSO on ix0. That seemed to fix the > > > > intermittent connection issues I had been experiencing with keeping an > > > > XMPP connection alive to one of our internal boxes. It hasn't done > > > > anything for the ICMPs or TCP traffic originating from the FreeBSD > > > > box, of course. > > > > > > Please show ifconfig for ix0 and ix1 again after disabling tso, > > > rxcsum, txcsum, vlanmtu, vlanhwtag, vlanhwfilter, vlanhwtso > > > and any other configured accelerations, as Luigi recommended? > > > > > > Then we'd know if your problem was related to any of that, or not. > > > > > > cheers, Ian > > From owner-freebsd-ipfw@FreeBSD.ORG Mon Sep 17 03:39:37 2012 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 92E01106566B for ; Mon, 17 Sep 2012 03:39:37 +0000 (UTC) (envelope-from dreijer@echobit.net) Received: from mail-oa0-f54.google.com (mail-oa0-f54.google.com [209.85.219.54]) by mx1.freebsd.org (Postfix) with ESMTP id 45F318FC0C for ; Mon, 17 Sep 2012 03:39:36 +0000 (UTC) Received: by oagm1 with SMTP id m1so5415292oag.13 for ; Sun, 16 Sep 2012 20:39:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :x-gm-message-state; bh=yp8+H9IYvMwMCbKfI8Db9wS/HTc/YA0OOGfz5nKpeMY=; b=AJBKKCqZN0dNj1K0wAvRNcpqMBPeP45Nd0UT1coXBy0GvCYsOW+EGR+6eJ6Zftdw8N 1ZZ7522ZmmwsCVb50azSo1wnHJhEUaWxY2zwiYiFIRQFpN/bUZviyUwrfhKU1lwwG2yy Q+6fEQbmVzw51HKJx4CVHpWGKRuxy/bEH5WHBt7TGqI2FHo0vT168wrL6od0OgUm6jBE X15pU4QM6ZbRQWuuwU5bx0JvXhPD/QKXVmge0AJqf6qqWc7a1otQ4rV6Bx02qqbzWqZ8 7A+QDjZWE0LqnJvcSbJ4VcqUftnmhxcyrPvqTtPLmNI4ExuPMtQUwtXw0w4/9UWEiwmR pkAg== MIME-Version: 1.0 Received: by 10.182.174.9 with SMTP id bo9mr10471930obc.19.1347853176295; Sun, 16 Sep 2012 20:39:36 -0700 (PDT) Sender: dreijer@echobit.net Received: by 10.76.99.75 with HTTP; Sun, 16 Sep 2012 20:39:36 -0700 (PDT) In-Reply-To: References: <20120913221758.E51539@sola.nimnet.asn.au> <20120913163013.GA22049@onelab2.iet.unipi.it> <20120913174612.GB22571@onelab2.iet.unipi.it> <20120914144529.R51539@sola.nimnet.asn.au> <20120915034627.V51539@sola.nimnet.asn.au> Date: Sun, 16 Sep 2012 22:39:36 -0500 X-Google-Sender-Auth: DsJBhMaDoxbcsYeCWlwETtTJ29Y Message-ID: From: Soren Dreijer To: Ian Smith Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQm93ZjIJuQ1ufj6+4zrCpivpEgPR0rl3YRaHdDixsc0bhX8Sy0WENCZt/jZos8nETkhlXwB Cc: freebsd-ipfw@freebsd.org, Luigi Rizzo Subject: Re: Significant network latency when using ipfw and in-kernel NAT X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Sep 2012 03:39:37 -0000 Some more updates: I went ahead and disabled a few options on the ixgbe network interface today (most notably rxcsum and txcsum), which improved ping times to the FreeBSD box. I'm now able to reliably ping it with ~40ms from my house. TCP traffic in general also seems to be slightly "better" as I can actually 'wget google.com' now, although it's still horribly slow and takes maybe 20 seconds or so to download. The ifconfig for the public adapter now looks like this: ix1: flags=8843 metric 0 mtu 1500 options=b8 I also changed all "out via ix1" rules to "out xmit ix1", and have updated a few other "in via ix1" to "in recv ix1". None of these changes seemed to have any effect on traffic originating from the FreeBSD box, though, and I still have ping times of >3 seconds to google.com. Like I mentioned earlier, I tried putting "allow icmp from any to any via ix1" at the top of the ipfw ruleset (to avoid any NAT'ing whatsoever) to see if that had any effect on the ping times from the box and it didn't. What I did notice, however, (and I don't know if this is related to the overall network latency), was that the outgoing ping packets were severely delayed in tcpdump from when the ping utility sent a packet. The output in tcpdump would be so delayed that after having killed the ping utility, I'd still see a packet or two go out on the interface! I'm running out of ideas of what to do here... / Soren On Sun, Sep 16, 2012 at 11:46 AM, Soren Dreijer wrote: > Just to follow up on this a bit: > > I haven't disabled any other options on the NICs yet due to high > server load over the weekend, but I'll give it a go in the next few > days. Also, it looks like pings to the box are now no longer as fast > as I had previously stated. Pinging it from my home connection now > yields >3 second roundtrip times, which neatly matches the ping time > from the box itself to google.com. > > As I mentioned before, I'm not sure how e.g. rxcsum and txcsum have > anything to do with high latency on ICMP traffic, so I'm wondering if > we're perhaps barking up the wrong tree here (especially since > forwarded traffic *through* the FreeBSD box seems to work just fine)? > > Thanks again for helping out here, guys. I'm in pretty deep water when > it comes to issues like this one. > > / Soren > > On Fri, Sep 14, 2012 at 12:59 PM, Ian Smith wrote: >> On Fri, 14 Sep 2012 09:12:27 -0500, Soren Dreijer wrote: >> >> > Can anybody confirm that disabling these other options (rxcsum, >> > txcsum, vlanmtu, vlanhwtag, vlanhwfilter, vlanhwtso) won't cause my >> > adapter to lose its connectivity? This is a server in production and >> > I'd rather not cause an outage if I can prevent it. :) >> >> Fair question Soren. I've configured no VLANs; out of my depth, again! >> >> cheers, Ian >> >> > On Fri, Sep 14, 2012 at 12:00 AM, Ian Smith wrote: >> > > On Thu, 13 Sep 2012 12:37:23 -0500, Soren Dreijer wrote: >> > > [Luigi Rizzo wrote:] >> > > > > i'd start by disabling all accelerations (and jumobgrams) >> > > > > and then move on from the results to figure out where is the problem. >> > > > >> > > > So, I went ahead and disabled TSO on ix0. That seemed to fix the >> > > > intermittent connection issues I had been experiencing with keeping an >> > > > XMPP connection alive to one of our internal boxes. It hasn't done >> > > > anything for the ICMPs or TCP traffic originating from the FreeBSD >> > > > box, of course. >> > > >> > > Please show ifconfig for ix0 and ix1 again after disabling tso, >> > > rxcsum, txcsum, vlanmtu, vlanhwtag, vlanhwfilter, vlanhwtso >> > > and any other configured accelerations, as Luigi recommended? >> > > >> > > Then we'd know if your problem was related to any of that, or not. >> > > >> > > cheers, Ian >> > From owner-freebsd-ipfw@FreeBSD.ORG Mon Sep 17 04:01:56 2012 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0943E1065672 for ; Mon, 17 Sep 2012 04:01:56 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id B5AE18FC0C for ; Mon, 17 Sep 2012 04:01:54 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 24B2F7300A; Mon, 17 Sep 2012 06:21:34 +0200 (CEST) Date: Mon, 17 Sep 2012 06:21:34 +0200 From: Luigi Rizzo To: Soren Dreijer Message-ID: <20120917042134.GA67124@onelab2.iet.unipi.it> References: <20120913163013.GA22049@onelab2.iet.unipi.it> <20120913174612.GB22571@onelab2.iet.unipi.it> <20120914144529.R51539@sola.nimnet.asn.au> <20120915034627.V51539@sola.nimnet.asn.au> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: freebsd-ipfw@freebsd.org, Ian Smith Subject: Re: Significant network latency when using ipfw and in-kernel NAT X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Sep 2012 04:01:56 -0000 On Sun, Sep 16, 2012 at 10:39:36PM -0500, Soren Dreijer wrote: > Some more updates: > > I went ahead and disabled a few options on the ixgbe network interface > today (most notably rxcsum and txcsum), which improved ping times to > the FreeBSD box. I'm now able to reliably ping it with ~40ms from my > house. TCP traffic in general also seems to be slightly "better" as I > can actually 'wget google.com' now, although it's still horribly slow > and takes maybe 20 seconds or so to download. > > The ifconfig for the public adapter now looks like this: > > ix1: flags=8843 metric 0 mtu 1500 > options=b8 what about the other one ? Also, please disable jumbo_mtu as well. On both inside and outside. Finally, can you send the output of "ipfw show" and "ipfw pipe show" (anonymized if you like, but please preserve the counters) to see if there is any traffic that is looping ? thanks luigi > > I'm running out of ideas of what to do here... > > / Soren > From owner-freebsd-ipfw@FreeBSD.ORG Mon Sep 17 11:07:08 2012 Return-Path: Delivered-To: freebsd-ipfw@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 65CD41065688 for ; Mon, 17 Sep 2012 11:07:08 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 4F2B28FC0C for ; Mon, 17 Sep 2012 11:07:08 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q8HB783s004475 for ; Mon, 17 Sep 2012 11:07:08 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q8HB779r004473 for freebsd-ipfw@FreeBSD.org; Mon, 17 Sep 2012 11:07:07 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 17 Sep 2012 11:07:07 GMT Message-Id: <201209171107.q8HB779r004473@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-ipfw@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-ipfw@FreeBSD.org X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Sep 2012 11:07:08 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/169206 ipfw [ipfw] ipfw does not flush entries in table o conf/167822 ipfw [ipfw] [patch] start script doesn't load firewall_type o kern/166406 ipfw [ipfw] ipfw does not set ALTQ identifier for ipv6 traf o kern/165939 ipfw [ipw] bug: incomplete firewall rules loaded if tables o kern/165190 ipfw [ipfw] [lo] [patch] loopback interface is not marking f kern/163873 ipfw [ipfw] ipfw fwd does not work with 'via interface' in o kern/158066 ipfw [ipfw] ipfw + netgraph + multicast = multicast packets o kern/157796 ipfw [ipfw] IPFW in-kernel NAT nat loopback / Default Route o kern/157689 ipfw [ipfw] ipfw nat config does not accept nonexistent int f kern/155927 ipfw [ipfw] ipfw stops to check packets for compliance with o bin/153252 ipfw [ipfw][patch] ipfw lockdown system in subsequent call o kern/153161 ipfw [ipfw] does not support specifying rules with ICMP cod o kern/152113 ipfw [ipfw] page fault on 8.1-RELEASE caused by certain amo o kern/148827 ipfw [ipfw] divert broken with in-kernel ipfw o kern/148689 ipfw [ipfw] antispoof wrongly triggers on link local IPv6 a o kern/148430 ipfw [ipfw] IPFW schedule delete broken. o kern/148091 ipfw [ipfw] ipfw ipv6 handling broken. o kern/143973 ipfw [ipfw] [panic] ipfw forward option causes kernel reboo o kern/143621 ipfw [ipfw] [dummynet] [patch] dummynet and vnet use result o kern/137346 ipfw [ipfw] ipfw nat redirect_proto is broken o kern/137232 ipfw [ipfw] parser troubles o kern/135476 ipfw [ipfw] IPFW table breaks after adding a large number o o kern/129036 ipfw [ipfw] 'ipfw fwd' does not change outgoing interface n p kern/128260 ipfw [ipfw] [patch] ipfw_divert damages IPv6 packets o kern/127230 ipfw [ipfw] [patch] Feature request to add UID and/or GID l f kern/122963 ipfw [ipfw] tcpdump does not show packets redirected by 'ip s kern/121807 ipfw [request] TCP and UDP port_table in ipfw o kern/121122 ipfw [ipfw] [patch] add support to ToS IP PRECEDENCE fields o kern/116009 ipfw [ipfw] [patch] Ignore errors when loading ruleset from o bin/104921 ipfw [patch] ipfw(8) sometimes treats ipv6 input as ipv4 (a o kern/104682 ipfw [ipfw] [patch] Some minor language consistency fixes a o kern/103454 ipfw [ipfw] [patch] [request] add a facility to modify DF b o kern/103328 ipfw [ipfw] [request] sugestions about ipfw table o kern/102471 ipfw [ipfw] [patch] add tos and dscp support o kern/97951 ipfw [ipfw] [patch] ipfw does not tie interface details to o kern/95084 ipfw [ipfw] [regression] [patch] IPFW2 ignores "recv/xmit/v o kern/86957 ipfw [ipfw] [patch] ipfw mac logging o bin/83046 ipfw [ipfw] ipfw2 error: "setup" is allowed for icmp, but s o kern/82724 ipfw [ipfw] [patch] [request] Add setnexthop and defaultrou o bin/78785 ipfw [patch] ipfw(8) verbosity locks machine if /etc/rc.fir o bin/65961 ipfw [ipfw] ipfw2 memory corruption inside add() o kern/60719 ipfw [ipfw] Headerless fragments generate cryptic error mes s kern/55984 ipfw [ipfw] [patch] time based firewalling support for ipfw o kern/48172 ipfw [ipfw] [patch] ipfw does not log size and flags o kern/46159 ipfw [ipfw] [patch] [request] ipfw dynamic rules lifetime f a kern/26534 ipfw [ipfw] Add an option to ipfw to log gid/uid of who cau 46 problems total. From owner-freebsd-ipfw@FreeBSD.ORG Mon Sep 17 12:04:20 2012 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2DF01106564A for ; Mon, 17 Sep 2012 12:04:20 +0000 (UTC) (envelope-from dreijer@echobit.net) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id D4EA68FC1F for ; Mon, 17 Sep 2012 12:04:19 +0000 (UTC) Received: by obbun3 with SMTP id un3so11058662obb.13 for ; Mon, 17 Sep 2012 05:04:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :x-gm-message-state; bh=Evr129PoKF8/ZLIeWeB5jP968eJWpnQPrrXWv9a95IM=; b=O9VP2B3SVrmPR81p1jJCx/UbRpnh031WEMfvUIIWk8F5rMfBSXGkc8+yhpNIfVzpsT Z2nSp2GrIh3piUKD6/fp8c8uq1UpWcCVKTW53szy4EmL/aKMmskvl+aSge+VRNN9VKMq baJsAkXo4uPz86EgPNWeBF9EWO2GnygcukMv7cKp/LFzob1cvcngkY+RFzJ7FDEq2LT+ t4bpC8PZRMlR95OQ6W8VXgUaOyZH8zmHi2MIDmgokZ/UPV1hCN/PR0t5GOasNaixv+n3 Kgrk6+cAmUfPN8+wkQrtgZ4E0QwkU4KekZVXtrV3cktXh0xlTxq/C/6FIVPhsCG4tGUz Z/8g== MIME-Version: 1.0 Received: by 10.182.49.7 with SMTP id q7mr11149105obn.68.1347883458981; Mon, 17 Sep 2012 05:04:18 -0700 (PDT) Sender: dreijer@echobit.net Received: by 10.76.99.75 with HTTP; Mon, 17 Sep 2012 05:04:18 -0700 (PDT) In-Reply-To: <20120917042134.GA67124@onelab2.iet.unipi.it> References: <20120913163013.GA22049@onelab2.iet.unipi.it> <20120913174612.GB22571@onelab2.iet.unipi.it> <20120914144529.R51539@sola.nimnet.asn.au> <20120915034627.V51539@sola.nimnet.asn.au> <20120917042134.GA67124@onelab2.iet.unipi.it> Date: Mon, 17 Sep 2012 07:04:18 -0500 X-Google-Sender-Auth: 1nCj4QrDURHBceJ4OP_tySu0pFk Message-ID: From: Soren Dreijer To: Luigi Rizzo Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQlMvbgsPesNvDrFqIDImb2Y2rNoGsAval/xLS4XVJemEtlU02j4IntO11MjoFOy2wF7xPg7 Cc: freebsd-ipfw@freebsd.org, Ian Smith Subject: Re: Significant network latency when using ipfw and in-kernel NAT X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Sep 2012 12:04:20 -0000 > what about the other one ? Also, please disable jumbo_mtu as well. > On both inside and outside. As far as I was able to tell, VLAN_HWCSUM cannot be disabled (or I don't know which command to use): http://lists.freebsd.org/pipermail/freebsd-net/2004-March/003464.html I also don't know how to disable JUMBO_MTU and VLAN_MTU. Disabling VLAN_HWCSUM didn't seem to do anything. Everything still has just as much latency as before: ix1: flags=8843 metric 0 mtu 1500 options=a8 Here is the current ruleset: 00001 32195 17958479 allow ip from any to any via ix0 00002 0 0 allow ip from any to any via gif0 00003 14593 1030091 allow ip from any to any via gif1 00004 17210 16260592 allow ip from any to any via gif2 00005 0 0 allow ip from any to any via gif3 00006 0 0 allow ip from any to any via lo0 00015 0 0 deny ip from 192.168.0.0/16 to any in via ix1 00016 0 0 deny ip from 172.16.0.0/12 to any in via ix1 00017 0 0 deny ip from 10.0.0.0/8 to any in via ix1 00018 0 0 deny ip from 127.0.0.0/8 to any in via ix1 00019 0 0 deny ip from 0.0.0.0/8 to any in via ix1 00020 0 0 deny ip from 169.254.0.0/16 to any in via ix1 00021 0 0 deny ip from 192.0.2.0/24 to any in via ix1 00022 0 0 deny ip from 204.152.64.0/23 to any in via ix1 00023 0 0 deny ip from 224.0.0.0/3 to any in via ix1 00025 11 1118 allow icmp from any to any icmptypes 3,11 in recv ix1 00026 6 264 deny icmp from any to any in recv ix1 00040 13121 745760 nat 1 ip from any to any in recv ix1 00050 0 0 check-state 00100 17 924 skipto 805 tcp from any to any out xmit ix1 setup keep-state 00202 5903 293907 skipto 600 tcp from any to 172.16.1.3 dst-port 443 in via ix1 00203 11289 15948611 skipto 805 tcp from 172.16.1.3 443 to any out xmit ix1 00204 7212 451553 skipto 700 tcp from any to 172.16.1.4 dst-port 5222 in via ix1 00205 7377 578378 skipto 805 tcp from 172.16.1.4 5222 to any out xmit ix1 00400 11 3564 deny ip from any to any via ix1 00500 0 0 pipe 1 ip from any to any in via ix1 00501 0 0 allow ip from any to any in via ix1 00600 5902 293361 pipe 2 ip from any to any in via ix1 00601 5902 293361 allow ip from any to any in via ix1 00700 7210 451399 pipe 3 ip from any to any in via ix1 00701 7210 451399 allow ip from any to any in via ix1 00800 0 0 pipe 4 ip from any to any in via ix1 00801 0 0 allow ip from any to any in via ix1 00805 18672 16520573 nat 1 ip from any to any out xmit ix1 00806 18672 16520573 allow ip from any to any 10000 0 0 deny ip from any to any via ix1 65535 865391 867355171 allow ip from any to any And the pipes: 00001: XX.000 Mbit/s 0 ms burst 0 q131073 50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail sched 65537 type FIFO flags 0x0 0 buckets 0 active 00002: XX.000 Mbit/s 0 ms burst 0 q131074 50 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 droptail sched 65538 type FIFO flags 0x0 0 buckets 0 active 00003: XX.000 Mbit/s 0 ms burst 0 q131075 50 sl. 0 flows (1 buckets) sched 65539 weight 0 lmax 0 pri 0 droptail sched 65539 type FIFO flags 0x0 0 buckets 0 active 00004: XX.000 Mbit/s 0 ms burst 0 q131076 50 sl. 0 flows (1 buckets) sched 65540 weight 0 lmax 0 pri 0 droptail sched 65540 type FIFO flags 0x0 0 buckets 0 active Like I mentioned earlier, one-pass is set to 0 to allow for traffic to be put back in to ipfw after going through NAT'ing and the pipes. That couldn't affect negatively, right? Cheers, Soren On Sun, Sep 16, 2012 at 11:21 PM, Luigi Rizzo wrote: > On Sun, Sep 16, 2012 at 10:39:36PM -0500, Soren Dreijer wrote: >> Some more updates: >> >> I went ahead and disabled a few options on the ixgbe network interface >> today (most notably rxcsum and txcsum), which improved ping times to >> the FreeBSD box. I'm now able to reliably ping it with ~40ms from my >> house. TCP traffic in general also seems to be slightly "better" as I >> can actually 'wget google.com' now, although it's still horribly slow >> and takes maybe 20 seconds or so to download. >> >> The ifconfig for the public adapter now looks like this: >> >> ix1: flags=8843 metric 0 mtu 1500 >> options=b8 > > what about the other one ? Also, please disable jumbo_mtu as well. > On both inside and outside. > > Finally, can you send the output of > "ipfw show" and "ipfw pipe show" (anonymized if you like, but > please preserve the counters) to see if there is any traffic > that is looping ? > > thanks > luigi > >> >> I'm running out of ideas of what to do here... >> >> / Soren >> From owner-freebsd-ipfw@FreeBSD.ORG Tue Sep 18 21:28:43 2012 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CA799106566C for ; Tue, 18 Sep 2012 21:28:43 +0000 (UTC) (envelope-from matt.vanmater@gmail.com) Received: from mail-lpp01m010-f54.google.com (mail-lpp01m010-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 4A01A8FC08 for ; Tue, 18 Sep 2012 21:28:43 +0000 (UTC) Received: by lahe6 with SMTP id e6so121923lah.13 for ; Tue, 18 Sep 2012 14:28:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=BtXMETYmxmAYj933J/tMmagM8WsUYt6lE+fgjpG3lTY=; b=O3f1hg/sek7rBnZtMPc6N4lnpW4EXNTOP4m5nXsxjPNd5wsBFlfjTvY7Acvoh/uPmS ucOIZUJDO/MGRijqJ2JPPKy6rjMhcGtPwVqzfmfeT5QeNN4Cj/n8cFPHfB2vgvdgoM/p OaVqMzQQ/hu502q4+7ZhFMT/JN6htAA3pY5DBIusBxZWu4GlWmgNStM5CAIT5XW4OCsc ZpeAIMIb8rQYlPrNmbBt07ACd8rzmayh0quij4XVqqtykYIdwWhWzv6L0kpItJARvu0k HhlZ9BRcnBvKfo+N+CiK0sdSpyy3yrxrm24SpgyaPN/B2H5fGtWdRC3kfQ4OdVLsnaA3 H8qQ== Received: by 10.152.162.10 with SMTP id xw10mr955284lab.12.1348003722100; Tue, 18 Sep 2012 14:28:42 -0700 (PDT) MIME-Version: 1.0 Received: by 10.112.40.99 with HTTP; Tue, 18 Sep 2012 14:28:21 -0700 (PDT) From: Matt Van Mater Date: Tue, 18 Sep 2012 17:28:21 -0400 Message-ID: To: freebsd-ipfw@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Problems with dummynet delay in VMWare guest X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Sep 2012 21:28:43 -0000 The subject says it all... I have built several freebsd9.0-release (+updates) virtual machines with open-vm-tools installed. Each VM has 2 vNICs (on separate vswitches) and has static IPs on those vNICs and static routes forming a virtual hub and spoke topology, with the edge node NATing using natd (to allow for internet access, testing, etc). The intent is to use dummynet to emulate a production network that has multi-hop terrestrial and satellite links with distinct bandwidth, delay and loss. However during testing I have noticed the delay is significantly off from what I have configured in ipfw. Has anyone else encountered this problem running dummynet inside a VMWare ESXi 4.1.0 (build260247) hypervisor? I have a hunch is is due to bad timing due to the use of VMware and I have tried changing the /boot/loader.conf kern.hz setting to 1000, and 100 but it seems to have no effect. For example: If I do not have any dummnet configuration, a ping from my edge vm to my external gateway is about .8 ms (perfectly acceptable) If I have a dummynet config with 7ms delay, the same ping shows a latency of .8ms (seemingly ignoring the dummynet config, not good) If I have a dummynet config with 10ms delay, i will sometimes get ~11ms latency (good, expected) and if i re-apply the config i get about 21ms latency (about twice what i expect) if i have a dummynet config with 30 ms delay, i consistently get 55ms latency. etc.. Simply put, I am seeing more latency than i expect. It is often twice what i configured, but not always in a nice predictable way. I could live with a very predictable offset (i.e.a 2x increase over my dummynet delay configuration) but it doesn't seem predictable. Any words of wisdom to help troubleshoot this? Thanks, Matt