From owner-freebsd-current@FreeBSD.ORG Tue Nov 10 08:43:05 2009 Return-Path: Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D308C106566B; Tue, 10 Nov 2009 08:43:05 +0000 (UTC) (envelope-from dfr@rabson.org) Received: from itchy.rabson.org (router.rabson.org [80.177.232.241]) by mx1.freebsd.org (Postfix) with ESMTP id BA4678FC1B; Tue, 10 Nov 2009 08:43:04 +0000 (UTC) Received: from [IPv6:2001:470:909f:1:225:ff:feed:9426] (unknown [IPv6:2001:470:909f:1:225:ff:feed:9426]) by itchy.rabson.org (Postfix) with ESMTP id D38C05E7F; Tue, 10 Nov 2009 08:42:24 +0000 (GMT) Mime-Version: 1.0 (Apple Message framework v1076) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes From: Doug Rabson In-Reply-To: Date: Tue, 10 Nov 2009 08:42:24 +0000 Content-Transfer-Encoding: 7bit Message-Id: <3D80A486-FB8F-451D-BF44-F62C647D8556@rabson.org> References: <4AF0B7DF.9030405@freebsd.org> <030A8229-9707-4F70-B4BE-584F1BF9ECEC@FreeBSD.org> To: Rick Macklem X-Mailer: Apple Mail (2.1076) Cc: O.Seibert@cs.ru.nl, dfr@FreeBSD.org, freebsd-current@FreeBSD.org, Rui Paulo , kometen@gmail.com Subject: Re: Help needed: TCP Wizards (was 8.0-RC1 NFS client timeout issue) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Nov 2009 08:43:06 -0000 On 9 Nov 2009, at 23:04, Rick Macklem wrote: > > > On Thu, 5 Nov 2009, Rui Paulo wrote: > >>> Now, here's where someone may be able to help? >>> Grep'ng around, I found 4 places where the TCP stack called >>> ip_output() >>> (one in each of tcp_output.c, tcp_subr.c, tcp_timewait.c and >>> tcp_syncache.c) and I put a printf like this just before them: >>> if (flags & TH_RST) >>> printf("sent a reset\n"); >>> >>> (The exact format varies for each, because of where the TCP >>> header flags are and have different printf messages.) >>> Now, the weird part is, that when the extraneous RST is sent to the >>> server, I don't get any printf. (I do get a few of these, but at >>> other >>> times for what appear to be legitimate RSTs.) >>> I can't see anywhere else that the TCP stack would send an RST >>> and, so, >>> I'm stuck w.r.t. figuring out what is sending them? > Ok, if you found the previous posts entertaining, you might enjoy > this:-) > > Along with the printfs before all the ip_output() calls, I added calls > inside ip_output() and, eventually, even calls in front of every > if_output(). Never got anything that indicated an RST was being sent. > (I only saw what I expected, which was an ACK reply being sent.) > > BUT, at almost exactly the same time, there were the FreeBSD8-CURRENT > client->server RST packets on the server's snoop trace. > > Hmm, did a tcpdump in the client and, yes, the same packets were > there. > > To keep it simple, I had done the dinosaur thing and plugged both the > client and server into an old, dumb 10baseT hub, so that I could > easily > watch everything. (I also had an uplink cable to the net port in the > wall, so I could move kernels around from the machine I usually build > them on.) > > I was at the point where I couldn't conceivably figure out where the > FreeBSD-CURRENT client was generating these RSTs. So guess what... > --> it wasn't > > I unplugged the uplink cable and, no more RSTs. I've been testing for > long enough now, that I am 99% certain they were being injected. Since > the from address and even the MAC address is correct, I can only > assume > that it was the Cisco switch that was doing the injecting. (How else > could a packet come in from the Cisco switch with the MAC address of > the FreeBSD-CURRENT client machine?) > > It was usually triggered by a server reboot. After the server reboot, > the server does send an RST to the client. This seems legit, but might > be what makes Cisco think that "bad things" are happening? (I have no > access to info about the switches or their configuration, although the > campus standard is for these ports to be used by a single desktop > machine > only and not a switch or hub.) > > So, it seems that FreeBSD8-CURRENT reconnects fine now, so long as > nothing is injecting RSTs into the newly created connection. > > Well, I'm not sure I found this fun, but hopefully others are > entertained:-), rick > We had some issues at work where the Cisco intrusion detection thing was resetting connections bogusly. Do you have IDS enabled on your Cisco gear?