Date: Sun, 16 Sep 2012 22:39:36 -0500 From: Soren Dreijer <dreijer+bsd@echobit.net> To: Ian Smith <smithi@nimnet.asn.au> Cc: freebsd-ipfw@freebsd.org, Luigi Rizzo <rizzo@iet.unipi.it> Subject: Re: Significant network latency when using ipfw and in-kernel NAT Message-ID: <CALoZf3gsXbOszP=G-NLtimJD6awaNv0TFfcm42%2B0ZMtF_JJfKQ@mail.gmail.com> In-Reply-To: <CALoZf3isvgnqXLSP3v6qDw1BpbxJ2%2Bus%2BdHYX4ZKcnrBwzfKKg@mail.gmail.com> References: <CALoZf3hfZDQQ4ZEXMrGUkYiGvb5QPoAcbpUikAq1adqVY4fLyg@mail.gmail.com> <20120913221758.E51539@sola.nimnet.asn.au> <CALoZf3iCf1_fHgAWUXa3fgudOe66sbk35P0CYhgsneBuhCORJg@mail.gmail.com> <20120913163013.GA22049@onelab2.iet.unipi.it> <CALoZf3iRzx5V=1th32LE8OCa0_GTBNGSZeGuH9qTp4Fk1j3ZRw@mail.gmail.com> <20120913174612.GB22571@onelab2.iet.unipi.it> <CALoZf3jRpcryGE0TXxdmZ0d6eD1KbJTY-KaNQEiUPuBuPzWtBA@mail.gmail.com> <20120914144529.R51539@sola.nimnet.asn.au> <CALoZf3hxdEcwXLVLi9Xdx2S%2Bv45-GNQo4b4XVcn-fGhB4y1Z%2Bw@mail.gmail.com> <20120915034627.V51539@sola.nimnet.asn.au> <CALoZf3isvgnqXLSP3v6qDw1BpbxJ2%2Bus%2BdHYX4ZKcnrBwzfKKg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Some more updates:
I went ahead and disabled a few options on the ixgbe network interface
today (most notably rxcsum and txcsum), which improved ping times to
the FreeBSD box. I'm now able to reliably ping it with ~40ms from my
house. TCP traffic in general also seems to be slightly "better" as I
can actually 'wget google.com' now, although it's still horribly slow
and takes maybe 20 seconds or so to download.
The ifconfig for the public adapter now looks like this:
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
I also changed all "out via ix1" rules to "out xmit ix1", and have
updated a few other "in via ix1" to "in recv ix1". None of these
changes seemed to have any effect on traffic originating from the
FreeBSD box, though, and I still have ping times of >3 seconds to
google.com.
Like I mentioned earlier, I tried putting "allow icmp from any to any
via ix1" at the top of the ipfw ruleset (to avoid any NAT'ing
whatsoever) to see if that had any effect on the ping times from the
box and it didn't. What I did notice, however, (and I don't know if
this is related to the overall network latency), was that the outgoing
ping packets were severely delayed in tcpdump from when the ping
utility sent a packet. The output in tcpdump would be so delayed that
after having killed the ping utility, I'd still see a packet or two go
out on the interface!
I'm running out of ideas of what to do here...
/ Soren
On Sun, Sep 16, 2012 at 11:46 AM, Soren Dreijer <dreijer+bsd@echobit.net> wrote:
> Just to follow up on this a bit:
>
> I haven't disabled any other options on the NICs yet due to high
> server load over the weekend, but I'll give it a go in the next few
> days. Also, it looks like pings to the box are now no longer as fast
> as I had previously stated. Pinging it from my home connection now
> yields >3 second roundtrip times, which neatly matches the ping time
> from the box itself to google.com.
>
> As I mentioned before, I'm not sure how e.g. rxcsum and txcsum have
> anything to do with high latency on ICMP traffic, so I'm wondering if
> we're perhaps barking up the wrong tree here (especially since
> forwarded traffic *through* the FreeBSD box seems to work just fine)?
>
> Thanks again for helping out here, guys. I'm in pretty deep water when
> it comes to issues like this one.
>
> / Soren
>
> On Fri, Sep 14, 2012 at 12:59 PM, Ian Smith <smithi@nimnet.asn.au> wrote:
>> On Fri, 14 Sep 2012 09:12:27 -0500, Soren Dreijer wrote:
>>
>> > Can anybody confirm that disabling these other options (rxcsum,
>> > txcsum, vlanmtu, vlanhwtag, vlanhwfilter, vlanhwtso) won't cause my
>> > adapter to lose its connectivity? This is a server in production and
>> > I'd rather not cause an outage if I can prevent it. :)
>>
>> Fair question Soren. I've configured no VLANs; out of my depth, again!
>>
>> cheers, Ian
>>
>> > On Fri, Sep 14, 2012 at 12:00 AM, Ian Smith <smithi@nimnet.asn.au> wrote:
>> > > On Thu, 13 Sep 2012 12:37:23 -0500, Soren Dreijer wrote:
>> > > [Luigi Rizzo wrote:]
>> > > > > i'd start by disabling all accelerations (and jumobgrams)
>> > > > > and then move on from the results to figure out where is the problem.
>> > > >
>> > > > So, I went ahead and disabled TSO on ix0. That seemed to fix the
>> > > > intermittent connection issues I had been experiencing with keeping an
>> > > > XMPP connection alive to one of our internal boxes. It hasn't done
>> > > > anything for the ICMPs or TCP traffic originating from the FreeBSD
>> > > > box, of course.
>> > >
>> > > Please show ifconfig for ix0 and ix1 again after disabling tso,
>> > > rxcsum, txcsum, vlanmtu, vlanhwtag, vlanhwfilter, vlanhwtso
>> > > and any other configured accelerations, as Luigi recommended?
>> > >
>> > > Then we'd know if your problem was related to any of that, or not.
>> > >
>> > > cheers, Ian
>> >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CALoZf3gsXbOszP=G-NLtimJD6awaNv0TFfcm42%2B0ZMtF_JJfKQ>