From owner-freebsd-current@FreeBSD.ORG Thu Aug 30 17:47:29 2007 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CECE316A41B for ; Thu, 30 Aug 2007 17:47:29 +0000 (UTC) (envelope-from M.S.Powell@salford.ac.uk) Received: from abbe.salford.ac.uk (abbe.salford.ac.uk [146.87.0.10]) by mx1.freebsd.org (Postfix) with SMTP id 4808613C45B for ; Thu, 30 Aug 2007 17:47:28 +0000 (UTC) (envelope-from M.S.Powell@salford.ac.uk) Received: (qmail 14016 invoked by uid 98); 30 Aug 2007 18:47:11 +0100 Received: from 146.87.255.121 by abbe.salford.ac.uk (envelope-from , uid 401) with qmail-scanner-2.01 (clamdscan: 0.90/3838. spamassassin: 3.1.8. Clear:RC:1(146.87.255.121):. Processed in 0.044917 secs); 30 Aug 2007 17:47:11 -0000 Received: from rust.salford.ac.uk (HELO rust.salford.ac.uk) (146.87.255.121) by abbe.salford.ac.uk (qpsmtpd/0.3x.614) with SMTP; Thu, 30 Aug 2007 18:47:11 +0100 Received: (qmail 60919 invoked by uid 1002); 30 Aug 2007 17:47:09 -0000 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 30 Aug 2007 17:47:09 -0000 Date: Thu, 30 Aug 2007 18:47:09 +0100 (BST) From: "Mark Powell" To: freebsd-current@freebsd.org Message-ID: <20070830183305.X60345@rust.salford.ac.uk> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Subject: Another ZFS kernel panic on same block on every drive in raidz X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Aug 2007 17:47:29 -0000 Hi, I am testing a 3 drive raidz1 array which has been built with 3 new WD 500GB SATA drives /dev/ad1[468], bought from 2 different sources. I am being told that a dma error is occuring on the same block on all 3 drives at the same time: Aug 30 18:13:15 echo kernel: ad14: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=268435340 Aug 30 18:13:15 echo kernel: ad16: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=268435340 Aug 30 18:13:15 echo kernel: ad18: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=268435340 Aug 30 18:13:46 echo kernel: ad14: TIMEOUT - WRITE_DMA retrying (0 retries left) LBA=268435340 Aug 30 18:13:46 echo kernel: ad16: TIMEOUT - WRITE_DMA retrying (0 retries left) LBA=268435340 Aug 30 18:13:46 echo kernel: ad18: TIMEOUT - WRITE_DMA retrying (0 retries left) LBA=268435340 Aug 30 18:13:46 echo kernel: ad14: FAILURE - WRITE_DMA timed out LBA=268435340 Aug 30 18:13:46 echo kernel: ad16: FAILURE - WRITE_DMA timed out LBA=268435340 Aug 30 18:13:46 echo kernel: ad18: FAILURE - WRITE_DMA timed out LBA=268435340 Aug 30 18:13:46 echo kernel: ad18: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=268435340 Aug 30 18:13:46 echo kernel: ad14: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=268435340 Aug 30 18:13:46 echo kernel: ad16: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=268435340 Aug 30 18:13:46 echo kernel: ad18: TIMEOUT - WRITE_DMA retrying (0 retries left) LBA=268435340 Aug 30 18:13:46 echo kernel: ad14: TIMEOUT - WRITE_DMA retrying (0 retries left) LBA=268435340 Aug 30 18:13:46 echo kernel: ad16: TIMEOUT - WRITE_DMA retrying (0 retries left) LBA=268435340 Aug 30 18:13:46 echo kernel: ad18: FAILURE - WRITE_DMA timed out LBA=268435340 Aug 30 18:13:46 echo kernel: ad14: FAILURE - WRITE_DMA timed out LBA=268435340 Aug 30 18:13:46 echo kernel: ad16: FAILURE - WRITE_DMA timed out LBA=268435340 Aug 30 18:13:46 echo kernel: ad14: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=268435340 Aug 30 18:13:46 echo kernel: ad18: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=268435340 Aug 30 18:13:46 echo kernel: ad16: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=268435340 Aug 30 18:13:25 echo root: ZFS: vdev I/O failure, zpool=pool path=/dev/ad14s2 offset=132076011520 size=65536 error=5 Aug 30 18:13:25 echo root: ZFS: vdev I/O failure, zpool=pool path=/dev/ad16s2 offset=132076011520 size=65536 error=5 Aug 30 18:13:25 echo root: ZFS: vdev I/O failure, zpool=pool path=/dev/ad18s2 offset=132076011520 size=65536 error=5 Aug 30 18:13:41 echo root: ZFS: vdev I/O failure, zpool=pool path=/dev/ad18s2 offset=132076011520 size=65536 error=5 Aug 30 18:13:41 echo root: ZFS: vdev I/O failure, zpool=pool path=/dev/ad14s2 offset=132076011520 size=65536 error=5 Aug 30 18:13:41 echo root: ZFS: vdev I/O failure, zpool=pool path=/dev/ad16s2 offset=132076011520 size=65536 error=5 Aug 30 18:13:41 echo root: ZFS: vdev I/O failure, zpool=pool path= offset=396215451648 size=131072 error=5 And then the kernel panics: panic: ZFS: I/O failure (write on off 0: zio 0xffffff0013b0d000 [L0 ZFS plain file] 20000L/20000P DVA[0]=<5:5c40480000:30000> fletcher2 uncompressed LE contiguous birth=20167 fill=1 cksum=cfcfcfcfcfcfce00:cfcfcfcfcfcfce00:8a8a8a8a8a56e700:8a8a8a8a8a56e cpuid = 0 I think I saw someone else have a similar problem to this. There were told their hardware was probably flakey on to look for errors with geli. Just performing a scrub now to see what happens. Let me know if you need any further info. Cheers. -- Mark Powell - UNIX System Administrator - The University of Salford Information Services Division, Clifford Whitworth Building, Salford University, Manchester, M5 4WT, UK. Tel: +44 161 295 4837 Fax: +44 161 295 5888 www.pgp.com for PGP key