From owner-freebsd-current@FreeBSD.ORG Tue Nov 3 09:52:58 2009 Return-Path: Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 88DDF1065672 for ; Tue, 3 Nov 2009 09:52:58 +0000 (UTC) (envelope-from gavin@FreeBSD.org) Received: from mail-gw1.york.ac.uk (mail-gw1.york.ac.uk [144.32.128.246]) by mx1.freebsd.org (Postfix) with ESMTP id 078D68FC26 for ; Tue, 3 Nov 2009 09:52:57 +0000 (UTC) Received: from mail-gw7.york.ac.uk (mail-gw7.york.ac.uk [144.32.129.30]) by mail-gw1.york.ac.uk (8.13.6/8.13.6) with ESMTP id nA39qpSZ008885; Tue, 3 Nov 2009 09:52:51 GMT Received: from buffy-128.york.ac.uk ([144.32.128.160] helo=buffy.york.ac.uk) by mail-gw7.york.ac.uk with esmtps (TLSv1:AES256-SHA:256) (Exim 4.68) (envelope-from ) id 1N5G4J-00008L-Nr; Tue, 03 Nov 2009 09:52:51 +0000 Received: from buffy.york.ac.uk (localhost [127.0.0.1]) by buffy.york.ac.uk (8.14.3/8.14.3) with ESMTP id nA2IGvUb091537; Mon, 2 Nov 2009 18:16:57 GMT (envelope-from gavin@FreeBSD.org) Received: (from ga9@localhost) by buffy.york.ac.uk (8.14.3/8.14.3/Submit) id nA2IGvi7091536; Mon, 2 Nov 2009 18:16:57 GMT (envelope-from gavin@FreeBSD.org) X-Authentication-Warning: buffy.york.ac.uk: ga9 set sender to gavin@FreeBSD.org using -f From: Gavin Atkinson To: Weldon S Godfrey 3 In-Reply-To: References: Content-Type: text/plain Content-Transfer-Encoding: 7bit Date: Mon, 02 Nov 2009 18:16:56 +0000 Message-Id: <1257185816.44755.29.camel@buffy.york.ac.uk> Mime-Version: 1.0 X-Mailer: Evolution 2.26.3 FreeBSD GNOME Team Port X-York-MailScanner: Found to be clean X-York-MailScanner-From: gavin@freebsd.org Cc: freebsd-current@FreeBSD.org Subject: Re: FreeBSD 8.0 - network stack crashes? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2009 09:52:58 -0000 On Mon, 2009-11-02 at 10:52 -0500, Weldon S Godfrey 3 wrote: > Up until yesterday, we have been running FreeBSD-CURRENT of 12/08. We > started to see a couple months ago some very odd network behavior. > Something happens to the stack that causes processes accessing the network > to just hang. After the problem happens, usually (but not always), you > can't ssh in. Always, you can't ssh or telnet out, and nothing can access > the NFS shares on the server. You can ping everything from the server. > You can't even do a route add, you can't ssh if you use just the IP > address (although pinging with hostnames it doesn't have cached or in > hosts table resolves). When you try to ssh out, do a route add from the > box, the process just hangs. You can't control C it at all, it hangs > forever. There is nothing in dmesg or messages to indicate an issue. I > try to up/down the interfaces. In CURRENT-12/08, it may allow things to > work for like 30s. Some things that would be useful: - Does "arp -da" fix things? - What's the output of "netstat -m" while the networking is broken? - What does CTRL-T show for the hung SSH or route processes? - What does "procstat -kk" on the same processes show? - Does going to single user mode ("init 1" and killing off any leftover processes) cause the machine to start working again? If so, what's the output of "netstat -m" afterwards? If you look to be hitting some of the limits shown by "netstat -m", try logging the date, "netstat -m" and "vmstat -m" to a file every 30 seconds or similar so that we can see if it is a memory leak, and what may be leaking. Gavin