From owner-freebsd-current@FreeBSD.ORG Thu Jun 25 11:45:32 2009 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 537801065672; Thu, 25 Jun 2009 11:45:32 +0000 (UTC) (envelope-from ianjhart@ntlworld.com) Received: from mtaout02-winn.ispmail.ntl.com (mtaout02-winn.ispmail.ntl.com [81.103.221.48]) by mx1.freebsd.org (Postfix) with ESMTP id 35FC98FC27; Thu, 25 Jun 2009 11:45:30 +0000 (UTC) (envelope-from ianjhart@ntlworld.com) Received: from aamtaout03-winn.ispmail.ntl.com ([81.103.221.35]) by mtaout02-winn.ispmail.ntl.com (InterMail vM.7.08.04.00 201-2186-134-20080326) with ESMTP id <20090625114530.XQYU6611.mtaout02-winn.ispmail.ntl.com@aamtaout03-winn.ispmail.ntl.com>; Thu, 25 Jun 2009 12:45:30 +0100 Received: from cpc1-cove3-0-0-cust909.sol2.cable.ntl.com ([86.20.31.142]) by aamtaout03-winn.ispmail.ntl.com (InterMail vG.2.02.00.01 201-2161-120-102-20060912) with ESMTP id <20090625114529.OGOX2093.aamtaout03-winn.ispmail.ntl.com@cpc1-cove3-0-0-cust909.sol2.cable.ntl.com>; Thu, 25 Jun 2009 12:45:29 +0100 X-Virus-Scanned: amavisd-new at cpc2-cove3-0-0-cust311.sol2.cable.ntl.com Received: from localhost (localhost [127.0.0.1]) by cpc1-cove3-0-0-cust909.sol2.cable.ntl.com (8.14.3/8.14.3) with ESMTP id n5PBjHt5072017; Thu, 25 Jun 2009 12:45:17 +0100 (BST) (envelope-from ianjhart@cpc1-cove3-0-0-cust909.sol2.cable.ntl.com) Received: from localhost (localhost [127.0.0.1]) by 10.248.192.16 (Horde Framework) with HTTP; Thu, 25 Jun 2009 12:45:16 +0100 Message-ID: <20090625124516.25615sdf5qk0meko@10.248.192.16> Date: Thu, 25 Jun 2009 12:45:16 +0100 From: ianjhart@ntlworld.com To: Kip Macy References: <200906132311.15359.ianjhart@ntlworld.com> <200906141427.08397.ianjhart@ntlworld.com> <200906150844.07051.ianjhart@ntlworld.com> <3c1674c90906231449g3d583ed9o98dd4dabb57999e0@mail.gmail.com> In-Reply-To: <3c1674c90906231449g3d583ed9o98dd4dabb57999e0@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) 4.3.3 / FreeBSD-7.2 X-Spam-Status: No, score=-1.4 required=5.0 tests=ALL_TRUSTED autolearn=failed version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on cpc1-cove3-0-0-cust909.sol2.cable.ntl.com X-Cloudmark-Analysis: v=1.0 c=1 a=ERehf_AEJYYA:10 a=6I5d2MoRAAAA:8 a=-lcJ2IEj15Jw0AfMYLgA:9 a=7RQCiPjMSwhiCYHdXp8A:7 a=mjc8DEKdwOAPTAhzftT5fpaQaX8A:4 a=SV7veod9ZcQA:10 Cc: freebsd-current@freebsd.org, Freddie Cash , ian j hart , current@freebsd.org Subject: Re: zpool scrub errors on 3ware 9550SXU X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jun 2009 11:45:33 -0000 Quoting Kip Macy : >> >> As usual scrubs cleanly on 7.2. Started throwing errors within a >> few minutes under 8. Then it paniced, possibly due to scrub -s. >> >> It's sat at the DB prompt if there's anything I can do. I'll need >> idiots guide level instruction. I have a screen dump if someone >> want to step up. Off list? >> >> Highlight seems to be... >> >> Memory modified after free 0xffffff0004da0c00(248) val=3000000 @ >> 0xffffff0004dc00 >> Panic: most recently used by none > > Can you test with recent 7-STABLE? That would tell me whether or not > your hitting a general HEAD issues or problems with the v13 import. > > Thanks, > Kip Updated to a recent STABLE. A single scrub on the same v6 zpool completed without error.I've started another scrub, but I've only had one scrub complete under CURRENT and that was with a different data set (smaller and less random, i.e. more compressable).This data now included with the original data. I'm not sure this is down to zfs but it's the only thing which I've found to cause this issue. As I said before I'm open to suggestions. Is it significant that it seems to fail ~40m in? More sample messages Jun 24 09:15:14 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da5 offset=291125194752 size=21504 Jun 24 09:15:14 data kernel: twa0: ERROR: (0x16: 0x1302): Unexpected status bit(s): status reg = 0xe00000 Unexpected bits: [MC_ERR,Q_ERR,PCI_PERR] Jun 24 09:15:14 data kernel: twa0: rror: clearing... Re-seat/move/replace card: (0x65207974: 0x1303): PCI parity error: clearing... Re-seat/move/replace card: status reg = 0x2ae175bb [CMD_Q_EMPTY,MC_RDY,RESP_Q_EMPTY,RESP_INTR,MC_ERR,Q_ERR,PCI_PERR] Jun 24 09:15:14 data kernel: twa0: ueue error: clearing... : (0x71207265: 0x1305): Controller queue error: clearing... : status reg = 0x2ae175bb [CMD_Q_EMPTY,MC_RDY,RESP_Q_EMPTY,RESP_INTR,MC_ERR,Q_ERR,PCI_PERR] Jun 24 09:15:14 data kernel: twa0: ller error! : (0x6F72746E: 0x1307): Micro-controller error! : status reg = 0x2ae175bb [CMD_Q_EMPTY,MC_RDY,RESP_Q_EMPTY,RESP_INTR,MC_ERR,Q_ERR,PCI_PERR] Jun 24 09:15:14 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da11 offset=291124226560 size=22016 Jun 24 09:15:14 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da10 offset=291124248576 size=22016 Jun 24 09:15:14 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da11 offset=291135879168 size=22016 Jun 24 09:15:14 data kernel: twa0: ERROR: (0x16: 0x1302): Unexpected status bit(s): status reg = 0xf00000 Unexpected bits: [PCI_ABRT,MC_ERR,Q_ERR,PCI_PERR] Jun 24 09:15:14 data kernel: twa0: RT,MC_ERR,Q_ERR,PCI_PERR]: (0x42415F49: 0x1303): PCI parity error: clearing... Re-seat/move/replace card: status reg = 0xefff7351 [CMD_Q_EMPTY,MC_RDY,RESP_Q_EMPTY,RESP_INTR,CMD_INTR,ATTN_INTR,HOST_INTR,PCI_ABRT,MC_ERR,Q_ERR,PCI_PERR] Jun 24 09:15:14 data kernel: twa0: RT,MC_ERR,Q_ERR,PCI_PERR]: (0x42415F49: 0x1304): PCI abort: clearing... : status reg = 0xefff7351 [CMD_Q_EMPTY,MC_RDY,RESP_Q_EMPTY,RESP_INTR,CMD_INTR,ATTN_INTR,HOST_INTR,PCI_ABRT,MC_ERR,Q_ERR,PCI_PERR] Jun 24 09:15:14 data kernel: twa0: RT,MC_ERR,Q_ERR,PCI_PERR]: (0x42415F49: 0x1305): Controller queue error: clearing... : status reg = 0xefff7351 [CMD_Q_EMPTY,MC_RDY,RESP_Q_EMPTY,RESP_INTR,CMD_INTR,ATTN_INTR,HOST_INTR,PCI_ABRT,MC_ERR,Q_ERR,PCI_PERR] Jun 24 09:15:14 data kernel: twa0: RT,MC_ERR,Q_ERR,PCI_PERR]: (0x42415F49: 0x1307): Micro-controller error! : status reg = 0xefff7351 [CMD_Q_EMPTY,MC_RDY,RESP_Q_EMPTY,RESP_INTR,CMD_INTR,ATTN_INTR,HOST_INTR,PCI_ABRT,MC_ERR,Q_ERR,PCI_PERR] Jun 24 09:15:14 data kernel: twa0: ERROR: (0x16: 0x1301): Missing expected status bit(s): status reg = 0x35bc9ffc; Missing bits: [MC_RDY,] Jun 24 09:15:14 data kernel: twa0: ERROR: (0x16: 0x1302): Unexpected status bit(s): status reg = 0xb00000 Unexpected bits: [PCI_ABRT,MC_ERR,PCI_PERR] Jun 24 09:15:14 data kernel: twa0: ty error: clearing... Re-seat/move/replace card: (0x69726170: 0x1303): PCI parity error: clearing... Re-seat/move/replace card: status reg = 0x35bc9ffc [CMD_Q_EMPTY,CMD_Q_FULL,ATTN_INTR,HOST_INTR,PCI_ABRT,MC_ERR,PCI_PERR] Jun 24 09:15:14 data kernel: twa0: t: clearing... : (0x726F6261: 0x1304): PCI abort: clearing... : status reg = 0x35bc9ffc [CMD_Q_EMPTY,CMD_Q_FULL,ATTN_INTR,HOST_INTR,PCI_ABRT,MC_ERR,PCI_PERR] Jun 24 09:15:14 data kernel: twa0: ntroller error! : (0x6F632D6F: 0x1307): Micro-controller error! : status reg = 0x35bc9ffc [CMD_Q_EMPTY,CMD_Q_FULL,ATTN_INTR,HOST_INTR,PCI_ABRT,MC_ERR,PCI_PERR] Jun 24 09:15:14 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da12 offset=291135901184 size=22016 Jun 24 09:15:14 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da13 offset=291135769600 size=22016 Jun 24 09:15:14 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da3 offset=291137000448 size=22016 Jun 24 09:15:14 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da1 offset=291136803840 size=22016 Jun 24 09:15:15 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da7 offset=291137503744 size=22016 Jun 24 09:15:15 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da12 offset=291136601600 size=22016 Jun 24 09:15:15 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da15 offset=291136820224 size=22016 Jun 24 09:15:15 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da4 offset=291137635328 size=22016 Jun 24 09:15:15 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da4 offset=291137722880 size=22016 Jun 24 09:15:15 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da15 offset=291136929792 size=22016 Jun 24 09:15:15 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da4 offset=291137875968 size=22016 Jun 24 09:15:15 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da11 offset=291136667136 size=22016 Jun 24 09:15:15 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da15 offset=291136710656 size=22016 Jun 24 09:15:15 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da11 offset=291137295872 size=22016 Jun 24 09:15:15 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da13 offset=291137295872 size=22016 Jun 24 09:15:15 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da2 offset=291138679808 size=22016 Jun 24 09:15:15 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da0 offset=291138461184 size=22016 Jun 24 09:15:15 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da3 offset=291185550336 size=9728 ...and so on ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.