Date: Sat, 13 Jun 2009 23:11:15 +0100 From: ian j hart <ianjhart@ntlworld.com> To: freebsd-current@freebsd.org Subject: zpool scrub errors on 3ware 9550SXU Message-ID: <200906132311.15359.ianjhart@ntlworld.com>
next in thread | raw e-mail | index | archive | help
[long post with long lines, sorry] I have the following old hardware which I'm trying to make into a storage=20 server (back story elided). Tyan Thunder K8WE with dual Opteron 270 8GB REG ECC RAM 3ware/AMCC 9550SXU-16 SATA controller Adaptec 29160 SCSI card -> Quantum LTO3 tape ChenBro case and backplanes. 'don't remember' PSU. I do remember paying =A398 3 years ago, so not cheap! floppy Some Seagate Barracuda drives. Two old 500GB for the O/S and 14 new 1.5TB f= or=20 data (plus some spares). Astute readers will know that the 1.5TB units have a chequered history. I went to considerable effort to avoid being stuck with a bricked unit, so= =20 imagine my dismay when, just before I was about to post this, I discovered= =20 there's a new issue with these drives where they reallocate sectors, from=20 new. I don't want to get sucked into a discussion about whether these disks are= =20 faulty or not. I want to examine what seems to be a regression between=20 7.2-RELEASE and 8-CURRENT. If you can't resist, start a thread in chat and = CC=20 me. Anyway, here's the full story (from memory I'm afraid). All disks exported as single drives (no JBOD anymore). Install current snapshot on da0 and gmirror with da1, both 500GB disks. Create a pool with the 14 1.5TB disks. Raidz2. Create a dataset. Tried various settings while benchmarking. Currently usin= g=20 compression=3Don copies=3D2. Copy ~1TB of data. Large files, dumps from the backup server. [50MB/s for 8-10h.] zpool scrub At some point (40m?) checksum errors start along with dire warnings about t= he=20 card. Break the gmirror and install 7.2-RELEASE on da1. Delete the pool. Create an identical pool under 7.2 (version 6) and then=20 scrub. Works okay, no errors. Export the pool, boot 8, import pool. Copy the data again. Scrub again. Same checksum throwing result. Cancel scrub, export pool, reboot 7.2, import pool. Scrubs okay. So it looks like I can scrub okay on 7.2, but not on 8. Data itself would s= eem to be okay. [I've shortened this a bit. For example at one point the 7.2-RELEASE stoppe= d=20 booting, had to reinstall. LORs come and go! Lagg has been in and out (I'm= =20 trying to benchmark the network as well). I did see 4 checksum errors on 7.= 2=20 but that was after 8 had 'fixed' the pool and was interrupted doing so. The= re were no card errors. Also subject to my memory. Must learn to write stuff down.] I also thought I'd try stressing the hardware without zfs envolved in case= =20 there was an issue with the drives/card/PCI bus or PSU. foreach x ( `jot -s " " -w "da%d" 14 2` ) dd if=3D/dev/$x of=3D/dev/null bs=3D512k& end It's crude and the load across the drives is uneven but it pulls ~45MB/s of= f=20 the 'slowest' drive. Ran this for an hour, no errors. Note to self: double= =20 check you did this under current not 7.2. Better tests welcome. If it's three lines or fewer that's a bonus. At this point I install smartmontools and find reallocated sectors. Armed w= ith=20 this keyword Google gives me lots of hits :(( So I started again, this time checking for reallocated sectors each time. T= he=20 numbers aren't going up, scrubs are still failing. Here's a sample of the messages. Scrub cancelled as soon as messages start. Jun 12 20:02:00 data kernel: lock order reversal: Jun 12 20:02:00 data kernel: 1st 0xffffff0004e00448 ufs (ufs) @ /usr/src.8/= sys/kern/vfs_mount.c:1197 Jun 12 20:02:00 data kernel: 2nd 0xffffff0004dd5d80 devfs (devfs) @ /usr/sr= c.8/sys/ufs/ffs/ffs_vfsops.c:1194 Jun 12 20:02:00 data kernel: KDB: stack backtrace: Jun 12 20:02:00 data kernel: db_trace_self_wrapper() at db_trace_self_wrapp= er+0x2a Jun 12 20:02:00 data kernel: _witness_debugger() at _witness_debugger+0x2e Jun 12 20:02:00 data kernel: witness_checkorder() at witness_checkorder+0x8= 1e Jun 12 20:02:00 data kernel: __lockmgr_args() at __lockmgr_args+0xc2a Jun 12 20:02:00 data kernel: vop_stdlock() at vop_stdlock+0x39 Jun 12 20:02:00 data kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b Jun 12 20:02:00 data kernel: _vn_lock() at _vn_lock+0x47 Jun 12 20:02:00 data kernel: ffs_flushfiles() at ffs_flushfiles+0xb5 Jun 12 20:02:00 data kernel: ffs_unmount() at ffs_unmount+0x67 Jun 12 20:02:00 data kernel: dounmount() at dounmount+0x2e6 Jun 12 20:02:00 data kernel: unmount() at unmount+0x250 Jun 12 20:02:00 data kernel: syscall() at syscall+0x1af Jun 12 20:02:00 data kernel: Xfast_syscall() at Xfast_syscall+0xd0 Jun 12 20:02:00 data kernel: --- syscall (22, FreeBSD ELF64, unmount), rip = =3D 0x8006a049c, rsp =3D 0x7fffffffe308, rbp =3D 0 --- Jun 12 20:19:30 data kernel: This module (opensolaris) contains code covere= d by the Jun 12 20:19:30 data kernel: Common Development and Distribution License (C= DDL) Jun 12 20:19:30 data kernel: see http://opensolaris.org/os/licensing/openso= laris_license/ Jun 12 20:19:30 data kernel: WARNING: ZFS is considered to be an experiment= al feature in FreeBSD. Jun 12 20:19:30 data kernel: ZFS filesystem version 13 Jun 12 20:19:30 data kernel: ZFS storage pool version 13 Jun 12 20:46:42 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da10 offset=3D8845411328 size=3D10752 Jun 12 20:46:42 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da8 offset=3D8845422080 size=3D11264 Jun 12 20:46:42 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da5 offset=3D8845433344 size=3D11264 Jun 12 20:46:42 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da13 offset=3D8845444096 size=3D10752 Jun 12 20:46:42 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da7 offset=3D8845455360 size=3D10752 Jun 12 20:46:43 data kernel: twa0: ERROR: (0x16: 0x1301): Missing expected = status bit(s): status reg =3D 0xde0d530c; Missing bits: [MC_RDY,] Jun 12 20:46:43 data kernel: twa0: ERROR: (0x03: 0x0107): Duplicate request= ID: RequestID=3D188 Jun 12 20:46:43 data kernel: (da3:twa0:0:3:0): READ(10). CDB: 28 0 21 ba 71= 8c 0 0 2b 0 Jun 12 20:46:43 data kernel: (da3:twa0:0:3:0): CAM Status: SCSI Status Error Jun 12 20:46:43 data kernel: (da3:twa0:0:3:0): SCSI Status: Check Condition Jun 12 20:46:43 data kernel: (da3:twa0:0:3:0): ILLEGAL REQUEST asc:24,0 Jun 12 20:46:43 data kernel: (da3:twa0:0:3:0): Invalid field in CDB Jun 12 20:46:43 data kernel: (da3:twa0:0:3:0): Unretryable error Jun 12 20:46:43 data kernel: twa0: ERROR: (0x15: 0x1201): Unposted command = completed!!: request =3D 0xffffff80003f4ac4, status =3D 3 Jun 12 20:46:43 data kernel: twa0: INFO: (0x16: 0x1108): Resetting controll= er...: Jun 12 20:46:43 data kernel: twa0: INFO: (0x04: 0x005E): Cache synchronizat= ion completed: unit=3D0 Jun 12 20:46:43 data kernel: twa0: INFO: (0x04: 0x0001): Controller reset o= ccurred: resets=3D1 Jun 12 20:46:43 data kernel: twa0: INFO: (0x16: 0x1107): Controller reset d= one!: Jun 12 20:46:43 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da8 offset=3D8849261056 size=3D10752 Jun 12 20:46:43 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da7 offset=3D289718205952 size=3D11264 Jun 12 20:46:43 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da4 offset=3D8850237440 size=3D11264 Jun 12 20:46:43 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da10 offset=3D8850237440 size=3D10752 Jun 12 20:46:43 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da8 offset=3D289718480384 size=3D10752 Jun 12 20:46:43 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da15 offset=3D289718831104 size=3D11264 Jun 12 20:46:43 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da2 offset=3D289718908416 size=3D10752 Jun 12 20:46:43 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da6 offset=3D289718282752 size=3D11264 Jun 12 20:46:43 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da9 offset=3D289718282752 size=3D11264 Jun 12 20:46:43 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da3 offset=3D289718875648 size=3D10752 Jun 12 20:46:43 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da2 offset=3D8849897472 size=3D11264 Jun 12 20:46:43 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da9 offset=3D8849897472 size=3D10752 Jun 12 20:46:43 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da12 offset=3D289718557184 size=3D10752 Jun 12 20:46:43 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da15 offset=3D289719237120 size=3D10752 Jun 12 20:46:43 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da15 offset=3D8850511360 size=3D11264 Jun 12 20:46:43 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da14 offset=3D289719423488 size=3D11264 Jun 12 20:46:43 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da5 offset=3D8849952256 size=3D11264 Jun 12 20:46:43 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da10 offset=3D8849952256 size=3D10752 Jun 12 20:46:43 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da6 offset=3D289718579200 size=3D10752 Jun 12 20:46:43 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da10 offset=3D289718579200 size=3D10752 Jun 12 20:46:47 data root: ZFS: vdev I/O failure, zpool=3Dtank path=3D/dev/= da3 offset=3D289723783168 size=3D87552 error=3D22 Jun 12 20:46:47 data root: ZFS: vdev I/O failure, zpool=3Dtank path=3D/dev/= da3 offset=3D289723783168 size=3D10752 error=3D22 Jun 12 20:46:47 data root: ZFS: vdev I/O failure, zpool=3Dtank path=3D/dev/= da3 offset=3D289723793920 size=3D10752 error=3D22 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da13 offset=3D8851531776 size=3D11264 Jun 12 20:46:47 data root: ZFS: vdev I/O failure, zpool=3Dtank path=3D/dev/= da3 offset=3D289723804672 size=3D11264 error=3D22 Jun 12 20:46:47 data root: ZFS: vdev I/O failure, zpool=3Dtank path=3D/dev/= da3 offset=3D289723815936 size=3D10752 error=3D22 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da10 offset=3D8851312640 size=3D10752 Jun 12 20:46:47 data root: ZFS: vdev I/O failure, zpool=3Dtank path=3D/dev/= da3 offset=3D289723826688 size=3D11264 error=3D22 Jun 12 20:46:47 data root: ZFS: vdev I/O failure, zpool=3Dtank path=3D/dev/= da3 offset=3D289723837952 size=3D10752 error=3D22 Jun 12 20:46:47 data root: ZFS: vdev I/O failure, zpool=3Dtank path=3D/dev/= da3 offset=3D289723848704 size=3D11264 error=3D22 Jun 12 20:46:47 data root: ZFS: vdev I/O failure, zpool=3Dtank path=3D/dev/= da3 offset=3D289723859968 size=3D10752 error=3D22 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da13 offset=3D8851543040 size=3D10752 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da10 offset=3D8851454976 size=3D11264 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da8 offset=3D8851565056 size=3D10752 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da2 offset=3D8851235840 size=3D11264 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da6 offset=3D8851575808 size=3D11264 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da13 offset=3D8851499008 size=3D10752 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da15 offset=3D8852255744 size=3D11264 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da11 offset=3D8851971072 size=3D10752 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da11 offset=3D8851718656 size=3D10752 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da12 offset=3D289720421888 size=3D11264 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da4 offset=3D289720817152 size=3D11264 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da8 offset=3D289720948736 size=3D11264 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da11 offset=3D289721003520 size=3D11264 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da8 offset=3D289720718336 size=3D11264 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da6 offset=3D289721212416 size=3D10752 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da8 offset=3D289721332736 size=3D11264 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da8 offset=3D289721344000 size=3D10752 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da14 offset=3D289720443904 size=3D11264 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da14 offset=3D289721409536 size=3D10752 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da14 offset=3D289721420288 size=3D11264 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da7 offset=3D289720509952 size=3D11264 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da12 offset=3D8851795456 size=3D10752 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da13 offset=3D8851795456 size=3D10752 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da12 offset=3D8851806208 size=3D10752 Jun 12 20:46:47 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da12 offset=3D8851850240 size=3D10752 Jun 12 20:46:48 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da3 offset=3D573848415232 size=3D10752 Jun 12 20:46:48 data kernel: twa0: ERROR: (0x16: 0x1301): Missing expected = status bit(s): status reg =3D 0x7f9bdff5; Missing bits: [MC_RDY,] Jun 12 20:46:48 data kernel: twa0: ERROR: (0x16: 0x1302): Unexpected status= bit(s): status reg =3D 0x900000 Unexpected bits: [PCI_ABRT,PCI_PERR] Jun 12 20:46:48 data kernel: twa0: _PERR]: (0x4943502C: 0x1303): PCI parity= error: clearing... Re-seat/move/replace card: status reg =3D 0x7f9bdff5 [C= MD_Q_EMPTY,RESP_Q_EMPTY,CMD_Q_FULL,RESP_INTR,CMD_INTR,HOST_INTR,PCI_ABRT,PC= I_PERR] Jun 12 20:46:48 data kernel: twa0: _PERR]: (0x4943502C: 0x1304): PCI abort:= clearing... : status reg =3D 0x7f9bdff5 [CMD_Q_EMPTY,RESP_Q_EMPTY,CMD_Q_FU= LL,RESP_INTR,CMD_INTR,HOST_INTR,PCI_ABRT,PCI_PERR] Jun 12 20:46:48 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da11 offset=3D573849005056 size=3D11264 Jun 12 20:46:48 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da14 offset=3D573848699904 size=3D11264 Jun 12 20:46:52 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da5 offset=3D289734832640 size=3D10752 Jun 12 20:46:52 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da5 offset=3D289734843392 size=3D10752 Jun 12 20:46:52 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da14 offset=3D289734963712 size=3D10752 Jun 12 20:46:52 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da10 offset=3D289735402496 size=3D11264 Jun 12 20:46:52 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da13 offset=3D9262030848 size=3D10752 Jun 12 20:46:52 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da12 offset=3D9262228480 size=3D10752 Jun 12 20:46:52 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da7 offset=3D9262502912 size=3D10752 Jun 12 20:46:52 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da9 offset=3D9262469632 size=3D11264 Jun 12 20:46:52 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da8 offset=3D289735830528 size=3D11264 Jun 12 20:46:52 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da11 offset=3D289735874560 size=3D10752 Jun 12 20:46:52 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da6 offset=3D289736028160 size=3D11264 Jun 12 20:46:52 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da10 offset=3D289735731712 size=3D11264 Jun 12 20:46:52 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da2 offset=3D289735479808 size=3D10752 Jun 12 20:46:52 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da8 offset=3D289735479808 size=3D10752 Jun 12 20:46:52 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da13 offset=3D289735589376 size=3D10752 Jun 12 20:46:52 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da15 offset=3D289735512064 size=3D11264 Jun 12 20:46:52 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da11 offset=3D289735523328 size=3D11264 Jun 12 20:46:52 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da7 offset=3D9267793408 size=3D10752 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da11 offset=3D9267540992 size=3D10752 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da4 offset=3D9267573760 size=3D11264 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da7 offset=3D9267836928 size=3D11264 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da4 offset=3D9267595776 size=3D11264 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da6 offset=3D9268056576 size=3D10752 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da6 offset=3D9267848192 size=3D10752 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da11 offset=3D9267474944 size=3D10752 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da11 offset=3D9267485696 size=3D11264 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da12 offset=3D9268122112 size=3D11264 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da7 offset=3D9267913728 size=3D11264 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da6 offset=3D9267935744 size=3D11264 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da4 offset=3D9267607040 size=3D10752 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da12 offset=3D9268111360 size=3D10752 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da2 offset=3D289741294592 size=3D11264 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da7 offset=3D289741185024 size=3D10752 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da7 offset=3D289741195776 size=3D10752 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da8 offset=3D9267990528 size=3D11264 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da9 offset=3D289741228544 size=3D11264 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da10 offset=3D289741272576 size=3D10752 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da10 offset=3D289741283328 size=3D11264 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da12 offset=3D289741634560 size=3D10752 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da15 offset=3D289741864960 size=3D10752 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da15 offset=3D289741875712 size=3D10752 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da3 offset=3D289741755392 size=3D11264 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da3 offset=3D289740998656 size=3D10752 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da11 offset=3D289741008896 size=3D11264 Jun 12 20:46:53 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da2 offset=3D9268254208 size=3D10752 Jun 12 20:46:57 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da15 offset=3D9396110848 size=3D10752 Jun 12 20:46:57 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da5 offset=3D9397246464 size=3D10752 Jun 12 20:46:57 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da10 offset=3D573881940992 size=3D11264 Jun 12 20:46:57 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da6 offset=3D573881985024 size=3D11264 Jun 12 20:46:57 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da9 offset=3D573882555392 size=3D11264 Jun 12 20:46:57 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da6 offset=3D573882007040 size=3D10752 Jun 12 20:46:57 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da8 offset=3D573882007040 size=3D10752 Jun 12 20:46:57 data kernel: twa0: ERROR: (0x16: 0x1302): Unexpected status= bit(s): status reg =3D 0x500000 Unexpected bits: [PCI_ABRT,Q_ERR,] Jun 12 20:46:57 data kernel: twa0: . : (0x2E2E676E: 0x1304): PCI abort: cle= aring... : status reg =3D 0x1c536664 [MC_RDY,RESP_Q_EMPTY,RESP_INTR,CMD_INT= R,PCI_ABRT,Q_ERR,] Jun 12 20:46:57 data kernel: twa0: r: clearing... : (0x6F727265: 0x1305): C= ontroller queue error: clearing... : status reg =3D 0x1c536664 [MC_RDY,RESP= _Q_EMPTY,RESP_INTR,CMD_INTR,PCI_ABRT,Q_ERR,] Jun 12 20:46:57 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da2 offset=3D573884749824 size=3D11264 Jun 12 20:46:57 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da14 offset=3D573884058624 size=3D10752 Jun 12 20:46:57 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da2 offset=3D573884761088 size=3D10752 Jun 12 20:46:57 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da6 offset=3D573885046272 size=3D10752 Jun 12 20:46:59 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da3 offset=3D289755986432 size=3D10752 Jun 12 20:46:59 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da5 offset=3D289756029952 size=3D11264 Jun 12 20:46:59 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da9 offset=3D9545516544 size=3D11264 Jun 12 20:46:59 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da7 offset=3D289756062720 size=3D11264 Jun 12 20:47:00 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da12 offset=3D289756117504 size=3D11264 Jun 12 20:47:01 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da10 offset=3D289764136448 size=3D10752 Jun 12 20:47:01 data kernel: twa0: ERROR: (0x16: 0x1301): Missing expected = status bit(s): status reg =3D 0x145994eb; Missing bits: [MC_RDY,] Jun 12 20:47:01 data kernel: twa0: ERROR: (0x16: 0x1302): Unexpected status= bit(s): status reg =3D 0x500000 Unexpected bits: [PCI_ABRT,Q_ERR,] Jun 12 20:47:01 data kernel: twa0: ng... : (0x69726165: 0x1304): PCI abort:= clearing... : status reg =3D 0x145994eb [CMD_Q_EMPTY,CMD_Q_FULL,RESP_INTR,= HOST_INTR,PCI_ABRT,Q_ERR,] Jun 12 20:47:01 data kernel: twa0: error: clearing... : (0x20657565: 0x1305= ): Controller queue error: clearing... : status reg =3D 0x145994eb [CMD_Q_E= MPTY,CMD_Q_FULL,RESP_INTR,HOST_INTR,PCI_ABRT,Q_ERR,] Jun 12 20:47:01 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da11 offset=3D9553024000 size=3D11264 Jun 12 20:47:01 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da12 offset=3D9553254400 size=3D11264 Jun 12 20:47:01 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da15 offset=3D9553112064 size=3D10752 Jun 12 20:47:01 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da8 offset=3D289764279296 size=3D10752 Jun 12 20:47:04 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da14 offset=3D289775233536 size=3D10752 Jun 12 20:47:04 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da6 offset=3D289775902720 size=3D11264 Jun 12 20:47:05 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da6 offset=3D9707735040 size=3D10752 Jun 12 20:47:05 data kernel: twa0: ERROR: (0x16: 0x1302): Unexpected status= bit(s): status reg =3D 0x600000 Unexpected bits: [MC_ERR,Q_ERR,] Jun 12 20:47:05 data kernel: twa0: : (0x202E2E2E: 0x1305): Controller queue= error: clearing... : status reg =3D 0x576a2c2e [MC_RDY,CMD_INTR,HOST_INTR,= MC_ERR,Q_ERR,] Jun 12 20:47:05 data kernel: twa0: ERROR: (0x16: 0x1307): Micro-controller = error! : status reg =3D 0x576a2c2e [MC_RDY,CMD_INTR,HOST_INTR,MC_ERR,Q_ERR,] Jun 12 20:47:05 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da7 offset=3D9708688384 size=3D10752 Jun 12 20:47:05 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da4 offset=3D289779238400 size=3D10752 Jun 12 20:47:07 data kernel: twa0: ERROR: (0x16: 0x1301): Missing expected = status bit(s): status reg =3D 0x79c41042; Missing bits: [MC_RDY,] Jun 12 20:47:07 data kernel: twa0: ERROR: (0x16: 0x1302): Unexpected status= bit(s): status reg =3D 0xc00000 Unexpected bits: [Q_ERR,PCI_PERR] Jun 12 20:47:07 data kernel: twa0: ove/replace card: (0x6D2F7461: 0x1303): = PCI parity error: clearing... Re-seat/move/replace card: status reg =3D 0x7= 9c41042 [CMD_Q_EMPTY,ATTN_INTR,Q_ERR,PCI_PERR] Jun 12 20:47:07 data kernel: twa0: ERROR: (0x20: 0x1305): Controller queue = error: clearing... : status reg =3D 0x79c41042 [CMD_Q_EMPTY,ATTN_INTR,Q_ERR= ,PCI_PERR] Jun 12 20:47:07 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da13 offset=3D206088704 size=3D11264 Jun 12 20:47:07 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da13 offset=3D206099968 size=3D10752 Jun 12 20:47:07 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da10 offset=3D206396416 size=3D10752 Jun 12 20:47:07 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da11 offset=3D573904253952 size=3D10752 Jun 12 20:47:07 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da5 offset=3D573904166400 size=3D10752 Jun 12 20:47:07 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da5 offset=3D573904177152 size=3D11264 Jun 12 20:47:07 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da3 offset=3D573904681984 size=3D11264 Jun 12 20:47:07 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da3 offset=3D573904693248 size=3D10752 Jun 12 20:47:07 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da9 offset=3D573905043968 size=3D10752 Jun 12 20:47:08 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da6 offset=3D289786880 size=3D11264 Jun 12 20:47:08 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da7 offset=3D289611264 size=3D11264 Jun 12 20:47:08 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da14 offset=3D289830400 size=3D11264 Jun 12 20:47:08 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da4 offset=3D289633280 size=3D11264 Jun 12 20:47:08 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da13 offset=3D289918464 size=3D10752 Jun 12 20:47:08 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da15 offset=3D289929216 size=3D11264 Jun 12 20:47:08 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da8 offset=3D573911089152 size=3D10752 Jun 12 20:47:08 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da14 offset=3D289841664 size=3D10752 Jun 12 20:47:08 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da8 offset=3D290094080 size=3D10752 Jun 12 20:47:08 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da8 offset=3D290104832 size=3D11264 Jun 12 20:47:08 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da2 offset=3D573911187968 size=3D11264 Jun 12 20:47:08 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da2 offset=3D573911440384 size=3D11264 Jun 12 20:47:08 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da4 offset=3D573910979584 size=3D10752 Jun 12 20:47:08 data root: ZFS: checksum mismatch, zpool=3Dtank path=3D/dev= /da9 offset=3D573911330304 size=3D11264 Cheers =2D-=20 ian j hart
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200906132311.15359.ianjhart>