From owner-freebsd-scsi@FreeBSD.ORG Fri Aug 29 01:31:15 2003 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DD63416A4BF for ; Fri, 29 Aug 2003 01:31:14 -0700 (PDT) Received: from mail.messagingengine.com (out1.smtp.messagingengine.com [66.111.4.25]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1046D43F3F for ; Fri, 29 Aug 2003 01:31:14 -0700 (PDT) (envelope-from freebsd@soith.com) Received: from www.fastmail.fm (localhost [127.0.0.1]) by localhost.localdomain (Postfix) with ESMTP id 8727614D9A2; Fri, 29 Aug 2003 04:30:51 -0400 (EDT) Received: from 10.202.2.132 ([10.202.2.132] helo=www.fastmail.fm) by messagingengine.com with SMTP; Fri, 29 Aug 2003 04:30:51 -0400 Received: by www.fastmail.fm (Postfix, from userid 99) id 4ACFC3A1A5; Fri, 29 Aug 2003 04:30:51 -0400 (EDT) Content-Disposition: inline Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="ISO-8859-1" MIME-Version: 1.0 X-Mailer: MIME::Lite 1.2 (F2.71; T1.001; A1.51; B2.12; Q2.03) From: "Aaron Wohl" To: "Sheldon Hearn" , freebsd-scsi@FreeBSD.org Date: Fri, 29 Aug 2003 02:30:51 -0600 X-Epoch: 1062145851 X-Sasl-enc: wKhRn0HlDx90lDVYkZmwFg References: <20030829041017.GL93028@starjuice.net> In-Reply-To: <20030829041017.GL93028@starjuice.net> Message-Id: <20030829083051.4ACFC3A1A5@www.fastmail.fm> Subject: Re: SMP, the aac driver and command timeouts X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Aug 2003 08:31:15 -0000 Yeah im getting 2-3 aac driver related crashes a day now with -current on a 5400s. I was seeing that "aac0 ... COMMAND 0x...... TIMEOUT AFTER ... seconds" as well. I did a cvsup and rebuild/install yesterday. Im not getting that now but still geting "command not in queue" panics. from an adaptic 5400S. AAC0> controller details Executing: controller details Controller Information ---------------------- Remote Computer: S Device Name: S Controller Type: Adaptec 5400S Access Mode: READ-WRITE Controller Serial Number: Last Six Digits = 6B1825 Number of Buses: 4 Devices per Bus: 15 Controller CPU: Strong Arm 110 Controller CPU Speed: 233 Mhz Controller Memory: 144 Mbytes Battery State: Ok Component Revisions ------------------- CLI: 1.0-0 (Build #5263) API: 1.0-0 (Build #5263) Miniport Driver: 1.0-0 (Build #5262) Controller Software: 1.0-0 (Build #5262) Controller BIOS: 1.0-0 (Build #5262) Controller Firmware: (Build #5262) Controller Hardware: 3.3 AAC0> uname -a (hostname edited) FreeBSD xxx 5.1-CURRENT FreeBSD 5.1-CURRENT #34: Wed Aug 27 17:26:58 EDT 2003 xxx:/usr/obj/usr/src/sys/PASODOBLE i386 *** email I sent this morning to the vendor we got our SMP hardware from ** We are getting 2-3 crashes a day in the aac driver on the machine we thought about replacing the processor on. Ive read all the goings on on the -current lists etc and trieed asking there about it. The crashes happen when doing heavy scsi io. Either with disk intensive mysql jobs or using the tape drive (amanda). Each time the panic is "panic: command not in queue" from the aac driver. The other server we got from you is not having these crashes. But we havent updated the OS on it since Fri Aug 1 19:50:58 EDT 2003. Its interesting that the stack backtrace for this crash ALWAYS has fork_exit in the stack backtrace. Its trying to remove a command from the response queue, but the item in the response queue has a santity check that says which queue its on and its not listed as being on the that queue. I think you mentioned you where shipping 5.x on your server now? Do you get -current or is there a specific date/time for cvs checkout of the operating system sources. Id read the stuff on the -current list about having INVARIANTs on pissing off the scsi driver due to new restrictions on doing INVARIANT checks from drivers. I tried building a kernel with INVARIANT off but it didnt help. panic: command not in queue panic messages: --- dmesg: kvm_read: --- Reading symbols from /usr/obj/usr/src/sys/PASODOBLE/modules/usr/src/sys/modules/acpi/acpi.ko.debug...done. Loaded symbols for /usr/obj/usr/src/sys/PASODOBLE/modules/usr/src/sys/modules/acpi/acpi.ko.debug Reading symbols from /boot/kernel/green_saver.ko...done. Loaded symbols for /boot/kernel/green_saver.ko #0 doadump () at /usr/src/sys/kern/kern_shutdown.c:240 240 dumping++; (kgdb) where #0 doadump () at /usr/src/sys/kern/kern_shutdown.c:240 #1 0xc0332b41 in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:372 #2 0xc0332f98 in panic () at /usr/src/sys/kern/kern_shutdown.c:550 #3 0xc01676b4 in aac_complete (context=0xcb918000, pending=1) at /usr/src/sys/dev/aac/aacvar.h:535 #4 0xc03599ed in taskqueue_run (queue=0xc6768780) at /usr/src/sys/kern/subr_taskqueue.c:205 #5 0xc0359ac3 in taskqueue_swi_run (dummy=0x0) at /usr/src/sys/kern/subr_taskqueue.c:221 #6 0xc031c8d8 in ithread_loop (arg=0xc6768700) at /usr/src/sys/kern/kern_intr.c:534 #7 0xc031b511 in fork_exit (callout=0xc031c700 , arg=0x0, frame=0x0) at /usr/src/sys/kern/kern_fork.c:796 (kgdb)