From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 3 10:49:58 2005 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 09D4016A41F for ; Mon, 3 Oct 2005 10:49:58 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [204.156.12.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id B0FA043D45 for ; Mon, 3 Oct 2005 10:49:57 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by cyrus.watson.org (Postfix) with ESMTP id 209FB46B08; Mon, 3 Oct 2005 06:49:57 -0400 (EDT) Date: Mon, 3 Oct 2005 11:49:57 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: "David S. Madole" In-Reply-To: <049901c5c78b$d94163d0$c3e7a8c0@david> Message-ID: <20051003114458.D71864@fledge.watson.org> References: <049901c5c78b$d94163d0$c3e7a8c0@david> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-hackers@freebsd.org Subject: Re: How to troubleshoot solid freeze-up? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Oct 2005 10:49:58 -0000 On Sun, 2 Oct 2005, David S. Madole wrote: > I'm looking for some tips on how to troubleshoot a possible driver > problem. Here is the scenario: > > 1. Using a Pentium II 333Mhz mobile processor, 82443BX motherboard, and > Intel i82559 NIC (fxp driver). > > 2. A combination of heavy disk I/O, high CPU utilization, and high > network traffic causes a solid machine freeze-up sometime between 10 > minutes and 3 hours of running. > > 3. Replacing the NIC with a DP83815-based card (sis driver) seems to > solve the problem. I have run the problem load for up to 8 hours without > issue on this NIC. > > 4. The problem is reproducable on multiple identical machines with > multiple identical NICs. Also reproducable on an i82558 NIC integrated > on the motherboard. > > How can I go about collecting useful information to troubleshoot this > when the machine locks solid? How can a get a core under this scenario? > > Switching to another NIC permanently is not a great solution because > this is a semi-embedded application and I need to use the NIC on the > motherboard. The normal method is to use a break signal to get into the debugger. Depending on your hardware and software configuration, this may be more or less easy. First, you'll need to configure options BREAK_TO_DEBUGGER into your kernel. You can break into the debugger in one of three ways: (1) Ctrl-alt-esc on a syscons console. Note that because the syscons console is under the Giant lock, reliability of this mechanism to get into the debugger on FreeBSD 5.x is reduced. It is quite a bit better in 6.x, and will continue to get better as the use of Giant is reduced. If this doesn't work for you, try (2). (2) Serial break on the first serial console port. Because the sio driver uses a fast interrupt handler, this is quite a reliable way to get into the debugger unless interrupts are disabled, in which case the serial port can't interrupt the CPU to drop into the debugger. If this doesn't work for you, try (3). (3) Break to debugger using an NMI. Some hardware, especially evaluation hardware, comes with an NMI button, frob, or other way to initiate a drop to the debugger despite interrupts being disabled. Hardware watchdogs are often also able to generate an NMI. I find that, except in pretty exceptional circumstances, (2) works quite well. You can find a section on kernel debugging in the FreeBSD handbook; my general advice is to compile in KDB, DDB, BREAK_TO_DEBUGGER, WITNESS, and INVARIANTS, and see where that gets you using a serial console. DDB is pretty easy to use for basic debugging -- i.e., checking thread state, checking lock state, generating stack traces, etc. Depending on the bug, you might also need/want to use kgdb via serial or on a core dump. Robert N M Watson