Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 10 Nov 2009 08:42:24 +0000
From:      Doug Rabson <dfr@rabson.org>
To:        Rick Macklem <rmacklem@uoguelph.ca>
Cc:        O.Seibert@cs.ru.nl, dfr@FreeBSD.org, freebsd-current@FreeBSD.org, Rui Paulo <rpaulo@FreeBSD.org>, kometen@gmail.com
Subject:   Re: Help needed: TCP Wizards (was 8.0-RC1 NFS client timeout issue)
Message-ID:  <3D80A486-FB8F-451D-BF44-F62C647D8556@rabson.org>
In-Reply-To: <Pine.GSO.4.63.0911091744570.18527@muncher.cs.uoguelph.ca>
References:  <Pine.GSO.4.63.0911011644410.19276@muncher.cs.uoguelph.ca> <4AF0B7DF.9030405@freebsd.org> <Pine.GSO.4.63.0911051121340.5409@muncher.cs.uoguelph.ca> <030A8229-9707-4F70-B4BE-584F1BF9ECEC@FreeBSD.org> <Pine.GSO.4.63.0911091744570.18527@muncher.cs.uoguelph.ca>

next in thread | previous in thread | raw e-mail | index | archive | help

On 9 Nov 2009, at 23:04, Rick Macklem wrote:

>
>
> On Thu, 5 Nov 2009, Rui Paulo wrote:
>
>>> Now, here's where someone may be able to help?
>>> Grep'ng around, I found 4 places where the TCP stack called  
>>> ip_output()
>>> (one in each of tcp_output.c, tcp_subr.c, tcp_timewait.c and  
>>> tcp_syncache.c) and I put a printf like this just before them:
>>> 	if (flags & TH_RST)
>>> 		printf("sent a reset\n");
>>>
>>> 	(The exact format varies for each, because of where the TCP
>>>       header flags are and have different printf messages.)
>>> Now, the weird part is, that when the extraneous RST is sent to the
>>> server, I don't get any printf. (I do get a few of these, but at  
>>> other
>>> times for what appear to be legitimate RSTs.)
>>> I can't see anywhere else that the TCP stack would send an RST  
>>> and, so,
>>> I'm stuck w.r.t. figuring out what is sending them?
> Ok, if you found the previous posts entertaining, you might enjoy  
> this:-)
>
> Along with the printfs before all the ip_output() calls, I added calls
> inside ip_output() and, eventually, even calls in front of every  
> if_output(). Never got anything that indicated an RST was being sent.
> (I only saw what I expected, which was an ACK reply being sent.)
>
> BUT, at almost exactly the same time, there were the FreeBSD8-CURRENT
> client->server RST packets on the server's snoop trace.
>
> Hmm, did a tcpdump in the client and, yes, the same packets were  
> there.
>
> To keep it simple, I had done the dinosaur thing and plugged both the
> client and server into an old, dumb 10baseT hub, so that I could  
> easily
> watch everything. (I also had an uplink cable to the net port in the
> wall, so I could move kernels around from the machine I usually build
> them on.)
>
> I was at the point where I couldn't conceivably figure out where the
> FreeBSD-CURRENT client was generating these RSTs. So guess what...
> --> it wasn't
>
> I unplugged the uplink cable and, no more RSTs. I've been testing for
> long enough now, that I am 99% certain they were being injected. Since
> the from address and even the MAC address is correct, I can only  
> assume
> that it was the Cisco switch that was doing the injecting. (How else
> could a packet come in from the Cisco switch with the MAC address of
> the FreeBSD-CURRENT client machine?)
>
> It was usually triggered by a server reboot. After the server reboot,
> the server does send an RST to the client. This seems legit, but might
> be what makes Cisco think that "bad things" are happening? (I have no
> access to info about the switches or their configuration, although the
> campus standard is for these ports to be used by a single desktop  
> machine
> only and not a switch or hub.)
>
> So, it seems that FreeBSD8-CURRENT reconnects fine now, so long as
> nothing is injecting RSTs into the newly created connection.
>
> Well, I'm not sure I found this fun, but hopefully others are
> entertained:-), rick
>

We had some issues at work where the Cisco intrusion detection thing  
was resetting connections bogusly. Do you have IDS enabled on your  
Cisco gear?




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3D80A486-FB8F-451D-BF44-F62C647D8556>