From owner-freebsd-current@FreeBSD.ORG Mon Nov 9 22:57:05 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 083DC1065676; Mon, 9 Nov 2009 22:57:05 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 7EE3E8FC31; Mon, 9 Nov 2009 22:57:04 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApoEAM8s+EqDaFvH/2dsb2JhbADeFYQ+BA X-IronPort-AV: E=Sophos;i="4.44,711,1249272000"; d="scan'208";a="54747631" Received: from danube.cs.uoguelph.ca ([131.104.91.199]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 09 Nov 2009 17:57:03 -0500 Received: from localhost (localhost.localdomain [127.0.0.1]) by danube.cs.uoguelph.ca (Postfix) with ESMTP id 82A69108498E; Mon, 9 Nov 2009 17:57:03 -0500 (EST) X-Virus-Scanned: amavisd-new at danube.cs.uoguelph.ca Received: from danube.cs.uoguelph.ca ([127.0.0.1]) by localhost (danube.cs.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Iwx+ZpzMErL0; Mon, 9 Nov 2009 17:57:02 -0500 (EST) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by danube.cs.uoguelph.ca (Postfix) with ESMTP id 385DF10845A3; Mon, 9 Nov 2009 17:57:02 -0500 (EST) Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id nA9N4dL23236; Mon, 9 Nov 2009 18:04:39 -0500 (EST) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Mon, 9 Nov 2009 18:04:39 -0500 (EST) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Rui Paulo In-Reply-To: <030A8229-9707-4F70-B4BE-584F1BF9ECEC@FreeBSD.org> Message-ID: References: <4AF0B7DF.9030405@freebsd.org> <030A8229-9707-4F70-B4BE-584F1BF9ECEC@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: dfr@FreeBSD.org, freebsd-current@FreeBSD.org, O.Seibert@cs.ru.nl, kometen@gmail.com Subject: Re: Help needed: TCP Wizards (was 8.0-RC1 NFS client timeout issue) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Nov 2009 22:57:05 -0000 On Thu, 5 Nov 2009, Rui Paulo wrote: >> >> Now, here's where someone may be able to help? >> >> Grep'ng around, I found 4 places where the TCP stack called ip_output() >> (one in each of tcp_output.c, tcp_subr.c, tcp_timewait.c and >> tcp_syncache.c) and I put a printf like this just before them: >> if (flags & TH_RST) >> printf("sent a reset\n"); >> >> (The exact format varies for each, because of where the TCP >> header flags are and have different printf messages.) >> >> Now, the weird part is, that when the extraneous RST is sent to the >> server, I don't get any printf. (I do get a few of these, but at other >> times for what appear to be legitimate RSTs.) >> >> I can't see anywhere else that the TCP stack would send an RST and, so, >> I'm stuck w.r.t. figuring out what is sending them? >> Ok, if you found the previous posts entertaining, you might enjoy this:-) Along with the printfs before all the ip_output() calls, I added calls inside ip_output() and, eventually, even calls in front of every if_output(). Never got anything that indicated an RST was being sent. (I only saw what I expected, which was an ACK reply being sent.) BUT, at almost exactly the same time, there were the FreeBSD8-CURRENT client->server RST packets on the server's snoop trace. Hmm, did a tcpdump in the client and, yes, the same packets were there. To keep it simple, I had done the dinosaur thing and plugged both the client and server into an old, dumb 10baseT hub, so that I could easily watch everything. (I also had an uplink cable to the net port in the wall, so I could move kernels around from the machine I usually build them on.) I was at the point where I couldn't conceivably figure out where the FreeBSD-CURRENT client was generating these RSTs. So guess what... --> it wasn't I unplugged the uplink cable and, no more RSTs. I've been testing for long enough now, that I am 99% certain they were being injected. Since the from address and even the MAC address is correct, I can only assume that it was the Cisco switch that was doing the injecting. (How else could a packet come in from the Cisco switch with the MAC address of the FreeBSD-CURRENT client machine?) It was usually triggered by a server reboot. After the server reboot, the server does send an RST to the client. This seems legit, but might be what makes Cisco think that "bad things" are happening? (I have no access to info about the switches or their configuration, although the campus standard is for these ports to be used by a single desktop machine only and not a switch or hub.) So, it seems that FreeBSD8-CURRENT reconnects fine now, so long as nothing is injecting RSTs into the newly created connection. Well, I'm not sure I found this fun, but hopefully others are entertained:-), rick