From owner-freebsd-hardware@FreeBSD.ORG Mon Jul 7 23:42:45 2008 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8BA7D1065689 for ; Mon, 7 Jul 2008 23:42:45 +0000 (UTC) (envelope-from mandrews@bit0.com) Received: from magnum.bit0.com (magnum.bit0.com [207.246.88.226]) by mx1.freebsd.org (Postfix) with ESMTP id 656558FC13 for ; Mon, 7 Jul 2008 23:42:45 +0000 (UTC) (envelope-from mandrews@bit0.com) Received: from localhost (localhost [127.0.0.1]) by magnum.bit0.com (Postfix) with ESMTP id 2AC335641F for ; Mon, 7 Jul 2008 19:26:59 -0400 (EDT) X-Virus-Scanned: amavisd-new at bit0.com Received: from magnum.bit0.com ([127.0.0.1]) by localhost (magnum.int.bit0.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id K0VBiBeB8xJd for ; Mon, 7 Jul 2008 19:26:58 -0400 (EDT) Received: from beast.int.bit0.com (nat.bit0.com [207.246.88.210]) by magnum.bit0.com (Postfix) with ESMTP for ; Mon, 7 Jul 2008 19:26:58 -0400 (EDT) Date: Mon, 7 Jul 2008 19:26:54 -0400 (EDT) From: Mike Andrews X-X-Sender: mandrews@beast.int.bit0.com To: freebsd-hardware@freebsd.org Message-ID: <20080707190237.K70038@beast.int.bit0.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Subject: Debugging 3Ware 9000 series hangs under load X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jul 2008 23:42:45 -0000 I've occasionally had problems with 3Ware 9550SX, 9650SE, and now 9690SA cards hanging under load. By "hanging" I mean "swap_pager: indefinite wait buffer" messages on the console, machine still pingable, etc -- basically a livelock situation. "Heavy load" usually involves a busy MySQL instance combined with, say, copying some multi-gigabyte files, maybe also combined with rsync, all running concurrently. The problem is the hangs are extremely sporadic and not easily reproducible on demand. It seems to happen with both ZFS and UFS2; I've been sticking to UFS2 for this for now due to some weird ZFS+MySQL issues (which are annoying but not relevant to this particular problem)... On a FreeBSD/amd64 7-STABLE system built from less-than-one-week-old source, with serial console and the kernel debugger compiled in, but being a complete idiot on how to use KDB, can someone tell me what would be the most useful info I could gather from KDB the next time this happens? Given the state of the controller at that point, I doubt forcing a crash dump from KDB is going to work unless I was able to do it to a disk on another controller... so I will work on getting that set up in the meantime. I suspect this is either a twa driver or kernel issue rather than hardware, as it's happened with multiple cards, cables, etc, but again, hard to tell if I don't know what to ask KDB for. :) On the off chance that it is driver related and takes a while to fix, anyone have recommendations for good 8-port SAS controllers for FreeBSD 7-STABLE amd64 that can take a heavy beating and play nice with Intel 3000 chipset-based boards? I'm eyeing the LSI Megaraid 8888ELP as a possible alternative... but don't have a great deal of LSI experience. The 3ware SAS setup I'm beating on now is replacing an old Adaptec 2120S SCSI setup. From owner-freebsd-hardware@FreeBSD.ORG Tue Jul 8 00:41:06 2008 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2400F1065673 for ; Tue, 8 Jul 2008 00:41:06 +0000 (UTC) (envelope-from aradford@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.176]) by mx1.freebsd.org (Postfix) with ESMTP id D04B68FC19 for ; Tue, 8 Jul 2008 00:41:05 +0000 (UTC) (envelope-from aradford@gmail.com) Received: by py-out-1112.google.com with SMTP id p76so1057655pyb.10 for ; Mon, 07 Jul 2008 17:41:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=ejBQnfLIL9nz5GIRs/PFze+zgibf0zHaM+pqwnFIhcA=; b=GfiCGipdmaw6goe7hN8hXPuZQrV+DLQXqRLG1XuNaPjJsUuA9mi971B+mOJoDsLris chlB87zkZCqJnMyjcTdzz49uoTdO0QZv8BY3Iy+i0sIqtGsGc6QzQrbIpmuYd75xbCjh +J4GrDUg8AazFDH4lhgu0D0h03mglaP9fGNTg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=IJ0EAj1EGwGXx5D+Wj2OYcJndC0EtPQJ2VHgFycGmNmSknYvBCRBDSK+J4rJ5kQkia XEOHUU9+Z+/35WXqunW6lzE3KpA6Dm/7vhUMRJ+qqYq9XvdrUxMEoKedcPS/ryPzUVdh f31qRQqrEgCi0gUvBtJPPv1zQZ3/jYxlibP0c= Received: by 10.115.32.8 with SMTP id k8mr7217520waj.89.1215477664840; Mon, 07 Jul 2008 17:41:04 -0700 (PDT) Received: by 10.115.79.9 with HTTP; Mon, 7 Jul 2008 17:41:04 -0700 (PDT) Message-ID: Date: Mon, 7 Jul 2008 17:41:04 -0700 From: "adam radford" To: "Mike Andrews" In-Reply-To: <20080707190237.K70038@beast.int.bit0.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20080707190237.K70038@beast.int.bit0.com> Cc: Manjunath Ranganathaiah , freebsd-hardware@freebsd.org Subject: Re: Debugging 3Ware 9000 series hangs under load X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 00:41:06 -0000 On Mon, Jul 7, 2008 at 4:26 PM, Mike Andrews wrote: > I've occasionally had problems with 3Ware 9550SX, 9650SE, and now 9690SA > cards hanging under load. By "hanging" I mean "swap_pager: indefinite wait > buffer" messages on the console, machine still pingable, etc -- basically a > livelock situation. "Heavy load" usually involves a busy MySQL instance > combined with, say, copying some multi-gigabyte files, maybe also combined > with rsync, all running concurrently. The problem is the hangs are > extremely sporadic and not easily reproducible on demand. It seems to happen > with both ZFS and UFS2; I've been sticking to UFS2 for this for now due to > some weird ZFS+MySQL issues (which are annoying but not relevant to this > particular problem)... > > On a FreeBSD/amd64 7-STABLE system built from less-than-one-week-old source, > with serial console and the kernel debugger compiled in, but being a > complete idiot on how to use KDB, can someone tell me what would be the most > useful info I could gather from KDB the next time this happens? Mike, Can you first run the 3ware command line utility (tw_cli) and post the results of: tw_cli info c0 diag -Adam From owner-freebsd-hardware@FreeBSD.ORG Wed Jul 9 05:22:41 2008 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4BEE71065681 for ; Wed, 9 Jul 2008 05:22:41 +0000 (UTC) (envelope-from mike503@gmail.com) Received: from wf-out-1314.google.com (wf-out-1314.google.com [209.85.200.169]) by mx1.freebsd.org (Postfix) with ESMTP id 23FDE8FC20 for ; Wed, 9 Jul 2008 05:22:41 +0000 (UTC) (envelope-from mike503@gmail.com) Received: by wf-out-1314.google.com with SMTP id 24so2589275wfg.7 for ; Tue, 08 Jul 2008 22:22:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type:content-transfer-encoding :content-disposition; bh=1HSn0jJNBWCVKuHwQST6VuES8qFcHtqgS1t2kKWriA8=; b=oe0WMjYQVUVRLUZhAFD9qlppwmfhxPu/PsQ3FZ3diKsZugxuj7+HK+Vg+UYj/edEoS yIlcbrrNWm7OX8qOia1A35+rRq2AhcqwnFZ8JpNlj/5L86P9QlAyv/jBwpzDFOs9PsJD F7cj5/QH7hDvh58zt8jHUtcGZhrdjA9PDgSoo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type :content-transfer-encoding:content-disposition; b=BUFJ++jY9UOX1qar6dgy6NQEHD++3ALCJGa9ZBlssHwWV1WR+bv+JvOlH9XrCYV9es MxSHIKVVGnkQAqcerCwPRLb528rWfxyY/9nXFekwIuqAnrClSGWsOy/HvgSM+gsD6t6m cC+F1HM/2S/uQMuIBfSlijmBW+m2DWaJm3/3w= Received: by 10.142.165.14 with SMTP id n14mr2109508wfe.77.1215579224533; Tue, 08 Jul 2008 21:53:44 -0700 (PDT) Received: by 10.142.102.11 with HTTP; Tue, 8 Jul 2008 21:53:44 -0700 (PDT) Message-ID: Date: Tue, 8 Jul 2008 21:53:44 -0700 From: mike To: freebsd-hardware@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: Wanting X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jul 2008 05:22:41 -0000 I'm trying to build the quietest smallest 4 or 6 drive FreeBSD 7 server to use at home for ZFS snapshotting. From the -fs list, it sounds like my idea passes the sanity check. Next is looking for the right hardware. There is a local computer shop I'd prefer to buy things at - www.enuinc.com, if people want to browse that and help me find the best equipment for this I'd be much appreciated. Basically it just needs space for 6 disks, ideally. - Gigabit LAN (single is fine, dual isn't required) - Onboard graphics - Support at least 4GB RAM - Quiet, small, and hopefully power efficient Will be hosting ZFS RAIDZ with all 6 disks (or 5 disks, and one for boot, or something since I think ZFS boot is not stable yet), with 15 filesystems to start and doing rsyncs every so often from remote servers locally and then creating a snapshot (basically creating daily snapshots every day for each server I maintain/backup) here's the stuff i'm not that sure about: mobo - http://www.enuinc.com/mb-775-asu-218.html RAM - would it be better for 2x2GB or 4x1GB? here is the 2x2GB kit: http://www.enuinc.com/m-04g-066-sup-k2.html here is the 2x1GB kits (if the mobo changes and it has 4 slots, I'd buy two): http://www.enuinc.com/m-02g-080-kin.html this is up for debate, too, for something smaller and quieter if possible. i love shuttles, but none of them can do maybe more than 4 disks that i know of... chassis - http://www.enuinc.com/cas-antec-056.html i'm sure of these: optical - chapest dvdrom http://www.enuinc.com/o-cdrd-son-001-black.html cpu - intel core2, whatever makes the most cost effective disks - between 4-6 1TB seagates Would this be good for a 95% dedicated to rsync+ZFS server? I doubt I will use it for anything else. Every hour it would run an rsync job, then snapshot it in ZFS. That's about it. Thanks in advance...