From owner-freebsd-bugs@FreeBSD.ORG Thu Aug 16 12:30:02 2007 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5D51716A473 for ; Thu, 16 Aug 2007 12:30:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 3B21A13C4F5 for ; Thu, 16 Aug 2007 12:30:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.1/8.14.1) with ESMTP id l7GCU2Uo037584 for ; Thu, 16 Aug 2007 12:30:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.1/8.14.1/Submit) id l7GCU2wU037579; Thu, 16 Aug 2007 12:30:02 GMT (envelope-from gnats) Resent-Date: Thu, 16 Aug 2007 12:30:02 GMT Resent-Message-Id: <200708161230.l7GCU2wU037579@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, graham bradley Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9950D16A419 for ; Thu, 16 Aug 2007 12:20:05 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id 8A1FC13C481 for ; Thu, 16 Aug 2007 12:20:05 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.1/8.14.1) with ESMTP id l7GCK5Bl072525 for ; Thu, 16 Aug 2007 12:20:05 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.1/8.14.1/Submit) id l7GCK5xZ072524; Thu, 16 Aug 2007 12:20:05 GMT (envelope-from nobody) Message-Id: <200708161220.l7GCK5xZ072524@www.freebsd.org> Date: Thu, 16 Aug 2007 12:20:05 GMT From: graham bradley To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/115572: gbde partitions fail at 28bit/48bit LBA addressing boundary X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Aug 2007 12:30:02 -0000 >Number: 115572 >Category: kern >Synopsis: gbde partitions fail at 28bit/48bit LBA addressing boundary >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Thu Aug 16 12:30:01 GMT 2007 >Closed-Date: >Last-Modified: >Originator: graham bradley >Release: FreeBSD 6.2-RELEASE >Organization: >Environment: FreeBSD rabbit.net 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Fri Jan 12 08:43:30 UTC 2007 root@portnoy.cse.buffalo.edu:/usr/obj/usr/src/sys/SMP amd64 >Description: # # to illustrate faulty read/write of a gbde disk drive around the 137gb boundary # prove the raw disk can be written everywhere (a few hours on a 500gb drive) dd if=/dev/zero of=/dev/ad4 bs=64k # or for smaller blocks.... dd if=/dev/zero of=/dev/ad4 bs=4k # set the entire /dev/ad4 drive up as a gbde partition gbde init ad4 -L /etc/gbde/ad4.lock #...... it wants passwords here ...... gbde attach ad4 -l /etc/gbde/ad4.lock #..... and password again ..... dd if=/dev/zero of=/dev/ad4 bs=64k oseek=1972156 count=1000 dd if=/dev/zero of=/dev/ad4 bs=4k oseek=31806314 count=1000 # which all works fine, but now if we write to the whole disk dd if=/dev/zero of=/dev/ad4 bs=64k # we see..... rabbit kernel: ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 # we have an error at around the (physical drive) block 268435434, # at the 28/48 bit LBA boundary. n.b. the gbde partition address is about 130gb, # as the block addresses are mapped in gbde to fit in the key blocks. # oddly, trying smaller block sizes down to 2k still fail, whilst 1k and 512b succeed. # This suggests that the earlier long dd write (somewhere) has trampled on (something)...... # to be sure the disk is still physically ok, we can do.... gbde detach ad4 dd if=/dev/zero of=/dev/ad4 bs=64k # and the disk drive appears ok, including across the 137gb boundary. #------------------------------------------------------------------------------------------- # in service, with the gbde partition mounted, this write error is repeating each time # the soft-update is triggered (every 30 seconds), as the disc is filled with data..... rabbit kernel: g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 rabbit kernel: ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 # exact details of the error vary according to file sizes, and start to appear after the disk # is filled beyond 60-100gb of files. # not using soft-update leaves us with the same problem, but it is less frequently flagged. # It is impossible after the errors start to achieve a snapshot, although fsck claims # the partition is fine! # Although fsck sanctions the disk as clean, the errors left behind slowly propagate # into the stored data in apparently random fashion, fsck then finds soft-update # inconsistencies, and sometimes files are silently left broken. >How-To-Repeat: re-initialise a large new drive with new keys and attach gbde no need for a newfs on the partition large dd write near/across the 137gb boundary seems ok attempt write /dev/zero from start to finish of the gbde partition, this fails as dd hits the (physical) 137gb boundary, at about 130gb in the gbde partition. detach gbde, and show that write is ok to the raw drive. check smartctl to show drive itself is error-free. >Fix: # n.b. geli does not map the partition on the physical drive, and apparently there's # no such problem. # so write across the entire disk..... geli init /dev/ad4 # .... password .... geli attach /dev/ad4 # .... password .... dd if=/dev/zero of=/dev/ad4.eli bs=64k # completes ok, so I am switching over to geli for big drives. Patch attached with submission follows: # # to illustrate faulty read/write of a gbde disk drive around the 137gb boundary # prove the raw disk can be written everywhere (a few hours on a 500gb drive) dd if=/dev/zero of=/dev/ad4 bs=64k # or for smaller blocks.... dd if=/dev/zero of=/dev/ad4 bs=4k # set the entire /dev/ad4 drive up as a gbde partition gbde init ad4 -L /etc/gbde/ad4.lock #...... it wants passwords here ...... gbde attach ad4 -l /etc/gbde/ad4.lock #..... and password again ..... dd if=/dev/zero of=/dev/ad4 bs=64k oseek=1972156 count=1000 dd if=/dev/zero of=/dev/ad4 bs=4k oseek=31806314 count=1000 # which all works fine, but now if we write to the whole disk dd if=/dev/zero of=/dev/ad4 bs=64k # we see..... rabbit kernel: ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 # we have an error at around the (physical drive) block 268435434, # at the 28/48 bit LBA boundary. n.b. the gbde partition address is about 130gb, # as the block addresses are mapped in gbde to fit in the key blocks. # oddly, trying smaller block sizes down to 2k still fail, whilst 1k and 512b succeed. # This suggests that the earlier long dd write (somewhere) has trampled on (something)...... # to be sure the disk is still physically ok, we can do.... gbde detach ad4 dd if=/dev/zero of=/dev/ad4 bs=64k # and the disk drive appears ok, including across the 137gb boundary. #------------------------------------------------------------------------------------------- # in service, with the gbde partition mounted, this write error is repeating each time # the soft-update is triggered (every 30 seconds), as the disc is filled with data..... rabbit kernel: g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 rabbit kernel: ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 # exact details of the error vary according to file sizes, and start to appear after the disk # is filled beyond 60-100gb of files. # not using soft-update leaves us with the same problem, but it is less often flagged. # It is impossible after the errors start to achieve a snapshot, although fsck claims # the partition is fine! # Although fsck sanctions the disk as clean, the errors left behind slowly propagate # into the stored data in apparently random fashion, fsck then finds soft-update # inconsistencies, and sometimes files are silently left broken. #------------------------------------------------------------------------------------------- # n.b. geli does not map the partition on the physical drive, and apparently there's # no such problem. # so write across the entire disk..... dd if=/dev/zero of=/dev/ad4.eli bs=64k # completes ok, so I am switching over to that method for big drives. #------------------------------------------------------------------------------------------- # DIAGNOSTICS # This is a new 500gb drive, and smartctl -a /dev/ad4 shows no drive errors afterwards..... rabbit# smartctl -a /dev/ad4 smartctl version 5.37 [amd64-portbld-freebsd6.2] Copyright (C) 2002-6 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: WDC WD5000AAKS-22TMA0 Serial Number: WD-WCAPW3258761 Firmware Version: 12.01C01 User Capacity: 500,107,862,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 7 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Thu Aug 16 11:11:53 2007 UTC SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (12000) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 150) minutes. Conveyance self-test routine recommended polling time: ( 6) minutes. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0003 187 187 021 Pre-fail Always - 5625 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 25 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x000e 200 200 051 Old_age Always - 0 9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 857 10 Spin_Retry_Count 0x0012 100 253 051 Old_age Always - 0 11 Calibration_Retry_Count 0x0012 100 253 051 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 23 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 25 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 28 194 Temperature_Celsius 0x0022 120 105 000 Old_age Always - 30 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0012 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 051 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. #------------------------------------------------------------------------------------------- # the output of dmesg when I do these tests ...... Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-RELEASE #0: Fri Jan 12 08:43:30 UTC 2007 root@portnoy.cse.buffalo.edu:/usr/obj/usr/src/sys/SMP ACPI APIC Table: Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Athlon(tm) 64 Processor 3500+ (2210.77-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x20ff2 Stepping = 2 Features=0x78bfbff Features2=0x1 AMD Features=0xe2500800 AMD Features2=0x1 real memory = 1073676288 (1023 MB) avail memory = 1023930368 (976 MB) ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 cpu0: on acpi0 acpi_button0: on acpi0 pcib0: port 0xcf8-0xcff,0xcf0-0xcf3 on acpi0 pci0: on pcib0 agp0: mem 0xf4000000-0xf5ffffff at device 0.0 on pci0 isab0: at device 1.0 on pci0 isa0: on isab0 pci0: at device 1.1 (no driver attached) ohci0: mem 0xfa003000-0xfa003fff irq 20 at device 2.0 on pci0 ohci0: [GIANT-LOCKED] usb0: OHCI version 1.0, legacy support usb0: on ohci0 usb0: USB revision 1.0 uhub0: nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 4 ports with 4 removable, self powered ohci1: mem 0xfa004000-0xfa004fff irq 21 at device 2.1 on pci0 ohci1: [GIANT-LOCKED] usb1: OHCI version 1.0, legacy support usb1: on ohci1 usb1: USB revision 1.0 uhub1: nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub1: 4 ports with 4 removable, self powered ehci0: mem 0xfa005000-0xfa0050ff irq 22 at device 2.2 on pci0 ehci0: [GIANT-LOCKED] usb2: EHCI version 1.0 usb2: companion controllers, 4 ports each: usb0 usb1 usb2: on ehci0 usb2: USB revision 2.0 uhub2: nVidia EHCI root hub, class 9/0, rev 2.00/1.00, addr 1 uhub2: 8 ports with 8 removable, self powered pci0: at device 6.0 (no driver attached) atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf000-0xf00f at device 8.0 on pci0 ata0: on atapci0 ata1: on atapci0 atapci1: port 0x9f0-0x9f7,0xbf0-0xbf3,0x970-0x977,0xb70-0xb73,0xdc00-0xdc0f,0xe000-0xe07f irq 21 at device 10.0 on pci0 ata2: on atapci1 ata3: on atapci1 pcib1: at device 11.0 on pci0 pci1: on pcib1 pci1: at device 0.0 (no driver attached) pcib2: at device 14.0 on pci0 pci2: on pcib2 skc0: port 0xa000-0xa0ff mem 0xf9000000-0xf9003fff irq 19 at device 11.0 on pci2 skc0: Marvell Yukon Lite Gigabit Ethernet rev. (0x9) sk0: on skc0 sk0: Ethernet address: 00:0f:ea:ff:33:86 miibus0: on sk0 e1000phy0: on miibus0 e1000phy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX-FDX, auto fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FAST] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A ppc0: port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppbus0: on ppc0 plip0: on ppbus0 lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: model IntelliMouse, device ID 3 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounter "TSC" frequency 2210769739 Hz quality 800 Timecounters tick every 1.000 msec ad0: 117800MB at ata0-master UDMA100 ad1: 117800MB at ata0-slave UDMA100 ad2: 238475MB at ata1-master UDMA100 ad3: 58644MB at ata1-slave UDMA100 ad4: 476940MB at ata2-master SATA150 ad6: 476940MB at ata3-master SATA150 Trying to mount root from ufs:/dev/ad0s1a ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding disabled, default to deny, logging disabled sk0: link state changed to UP sk0: link state changed to DOWN sk0: link state changed to UP ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435434 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435450 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435450 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435450 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435450 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435450 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435454 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435454 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435454 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435433 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435449 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435449 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435453 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435453 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435453 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435453 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435453 ad4: FAILURE - WRITE_DMA status=51 error=10 LBA=268435449 ad4: FAILURE - READ_DMA status=51 error=10 LBA=268435449 ad4: FAILURE - READ_DMA status=51 error=10 LBA=268435433 GEOM_ELI: Device ad4.eli created. GEOM_ELI: Encryption: AES-CBC 128 GEOM_ELI: Crypto: software >Release-Note: >Audit-Trail: >Unformatted: