From owner-freebsd-stable@FreeBSD.ORG Wed Feb 4 13:13:04 2009 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BBF1B1065672 for ; Wed, 4 Feb 2009 13:13:04 +0000 (UTC) (envelope-from mattblists@icritical.com) Received: from mail2.icritical.com (mail2.icritical.com [212.57.248.50]) by mx1.freebsd.org (Postfix) with SMTP id 0E08B8FC12 for ; Wed, 4 Feb 2009 13:13:03 +0000 (UTC) (envelope-from mattblists@icritical.com) Received: (qmail 4087 invoked from network); 4 Feb 2009 12:47:24 -0000 Received: from localhost (127.0.0.1) by mail2.icritical.com with SMTP; 4 Feb 2009 12:47:24 -0000 Received: (qmail 4080 invoked by uid 599); 4 Feb 2009 12:47:20 -0000 Received: from unknown (HELO icritical.com) (87.127.43.249) by mail2.icritical.com (qpsmtpd/0.28) with ESMTP; Wed, 04 Feb 2009 12:47:20 +0000 Message-ID: <49898E3D.7030609@icritical.com> Date: Wed, 04 Feb 2009 12:46:53 +0000 From: Matt Burke User-Agent: Thunderbird 2.0.0.16 (X11/20080707) MIME-Version: 1.0 To: stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 04 Feb 2009 12:47:01.0655 (UTC) FILETIME=[ACBE0270:01C986C6] X-Virus-Scanned: by iCritical at mail2.icritical.com Cc: Subject: 7.1-RELEASE I/O hang X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Feb 2009 13:13:05 -0000 I have a machine with a PERC6/e controller. Attached to that are 3 disk shelves, each configured as individual 14-disk RAID10 arrays (the PERC annoyingly only lets you use 8 spans per array) I can run bonnie++ on the arrays individually with no problem. I can also run it across a gstripe of the arrays with no problem. However running it over the 3 arrays in parallel causes something I/O related in the kernel to hang. To define 'hang' better: It appears anything which needs disk io, even on a different controller (albeit the same mfi driver), will hang. A command like 'ps' cached in ram will work but bash hangs after execution, presumably while trying to write ~/.bash_history 'sysctl -a' works but trying to run 'sysctl kern.msgbuf' also hangs I've done some research and it seems the usual cause of bonnie++ crashing a system is due to overflowing TCQ. camcontrol doesn't see any disks, so I've tried setting hw.mfi.max_cmds=32 in /boot/loader.conf but it hadn't made any difference. The bonnie++ invocation is this: (newfs devices mfid[2-3], mount) bonnie++ -s 64g -u root -p3 bonnie++ -d /data/2 -s 64g -u root -y s >b2 2>&1 & bonnie++ -d /data/3 -s 64g -u root -y s >b3 2>&1 & bonnie++ -d /data/4 -s 64g -u root -y s >b4 2>&1 & and it always hangs on "Rewriting...". It's a fresh 7.1-RELEASE with nothing else running (devd, sshd, syslogd, etc) Any ideas? --