From owner-freebsd-stable@freebsd.org Sat Sep 30 21:38:49 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 266A1E2FDA1 for ; Sat, 30 Sep 2017 21:38:49 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 975FF81AEF for ; Sat, 30 Sep 2017 21:38:47 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (mh0.gentlemail.de [78.138.80.135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id v8ULckOA055699 for ; Sat, 30 Sep 2017 23:38:46 +0200 (CEST) (envelope-from freebsd@omnilan.de) Received: from titan.inop.mo1.omnilan.net (s1.omnilan.de [217.91.127.234]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id BDB3FC51; Sat, 30 Sep 2017 23:38:45 +0200 (CEST) Message-ID: <59D00EE5.7090701@omnilan.de> Date: Sat, 30 Sep 2017 23:38:45 +0200 From: Harry Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Re: panic: Solaris(panic): blkptr invalid CHECKSUM1 References: <59CFC6A6.6030600@omnilan.de> <59CFD37A.8080009@omnilan.de> In-Reply-To: <59CFD37A.8080009@omnilan.de> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Greylist: ACL 129 matched, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [78.138.80.130]); Sat, 30 Sep 2017 23:38:46 +0200 (CEST) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: 78.138.80.135; Sender-helo: mh0.gentlemail.de; ) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Sep 2017 21:38:49 -0000 Bezüglich Harry Schmalzbauer's Nachricht vom 30.09.2017 19:25 (localtime): > Bezüglich Harry Schmalzbauer's Nachricht vom 30.09.2017 18:30 (localtime): >> Bad surprise. >> Most likely I forgot to stop a PCIe-Passthrough NIC before shutting down >> that (byhve(8)) guest – jhb@ helped my identifying this as the root >> cause for sever memory corruptions I regularly had (on stable-11). >> >> Now this time, corruption affected ZFS's RAM area, obviously. >> >> What I haven't expected is the panic. >> The machine has memory disk as root, so luckily I still can boot (from >> ZFS, –> mdpreload rootfs) into single user mode, but early rc stage >> (most likely mounting ZFS datasets) leads to the following panic: >> >> Trying to mount root from ufs:/dev/ufs/cetusROOT []... >> panic: Solaris(panic): blkptr at 0xfffffe0005b6b000 has invalid CHECKSUM 1 >> cpuid = 1 >> KDB: stack backtrace: >> #0 0xffffffff805e3837 at kdb_backtrace+0x67 >> #1 0xffffffff805a2286 at vpanic+0x186 >> #2 0xffffffff805a20f3 at panic+0x43 >> #3 0xffffffff81570192 at vcmn_err+0xc2 >> #4 0xffffffff812d7dda at zfs_panic_recover+0x5a >> #5 0xffffffff812ff49b at zfs_blkptr_verify+0x8b >> #6 0xffffffff812ff72c at zio_read+0x2c >> #7 0xffffffff812761de at arc_read+0x6de >> #8 0xffffffff81298b4d at traverse_prefetch_metadata+0xbd >> #9 0xffffffff812980ed at traverse_visitbp+0x39d >> #10 0xffffffff81298c27 at traverse_dnode+0xc7 >> #11 0xffffffff812984a3 at traverse_visitbp+0x753 >> #12 0xffffffff8129788b at traverse_impl+0x22b >> #13 0xffffffff81297afc at traverse_pool+0x5c >> #14 0xffffffff812cce06 at spa_load+0x1c06 >> #15 0xffffffff812cc302 at spa_load+0x1102 >> #16 0xffffffff812cac6e at spa_load_best+0x6e >> #17 0xffffffff812c73a1 at spa_open_common+0x101 >> Uptime: 37s >> Dumping 1082 out of 15733 MB:..2%..… >> Dump complete >> mps0: Sending StopUnit: path (xpt0:mps0:0:2:ffffffff): handle 12 >> mps0: Incrementing SSU count >> … >> >> Haven't done any scrub attempts yet – expectation is to get all datasets >> of the striped mirror pool back... >> >> Any hints highly appreciated. > Now it seems I'm in really big trouble. > Regular import doesn't work (also not if booted from cd9660). > I get all pools listed, but trying to import (unmounted) leads to the > same panic as initialy reported – because rc is just doning the same. > > I booted into single user mode (which works since the bootpool isn't > affected and root is a memory disk from the bootpool) > and set vfs.zfs.recover=1. > But this time I don't even get the list of pools to import 'zpool' > import instantaniously leads to that panic: > > Solaris: WARNING: blkptr at 0xfffffe0005a8e000 has invalid CHECKSUM 1 > Solaris: WARNING: blkptr at 0xfffffe0005a8e000 has invalid COMPRESS 0 > Solaris: WARNING: blkptr at 0xfffffe0005a8e000 DVA 0 has invalid VDEV > 2337865727 > Solaris: WARNING: blkptr at 0xfffffe0005a8e000 DVA 1 has invalid VDEV > 289407040 > Solaris: WARNING: blkptr at 0xfffffe0005a8e000 DVA 2 has invalid VDEV > 3959586324 > > > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0x50 > fault code = supervisor read data, page not present > instruction pointer = 0x20:0xffffffff812de904 > stack pointer = 0x28:0xfffffe043f6bcbc0 > frame pointer = 0x28:0xfffffe043f6bcbc0 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 44 (zpool) > trap number = 12 > panic: page fault > cpuid = 0 … OpenIndiana also panics at regular import. Unfortunately I don't know the aequivalent of vfs.zfs.recover in OI. panic[cpu1]/thread=ffffff06dafe8be0: blkptr at ffffff06dbe63000 has invalid CHECKSUM 1 Warning - stack not written to the dump buffer ffffff001f67f070 genunix:vcmn_err+42 () ffffff001f67f0e0 zfs:zfs_panic_recover+51 () ffffff001f67f140 zfs:zfs_blkptr_verify+8d () ffffff001f67f220 zfs:zio_read+55 () ffffff001f67f310 zfs:arc_read+662 () ffffff001f67f370 zfs:traverse_prefetch_metadata+b5 () ffffff001f67f450 zfs:traverse_visitbp+1c3 () ffffff001f67f4e0 zfs:traverse_dnode+af () ffffff001f67f5c0 zfs:traverse_visitbp+6dd () ffffff001f67f720 zfs:traverse_impl+1a6 () ffffff001f67f830 zfs:traverse_pool+9f () ffffff001f67f8a0 zfs:spa_load_verify+1e6 () ffffff001f67f990 zfs:spa_load_impl+e1c () ffffff001f67fa30 zfs:spa_load+14e () ffffff001f67fad0 zfs:spa_load_best+7a () ffffff001f67fb90 zfs:spa_import+1b0 () ffffff001f67fbe0 zfs:zfs_ioc_pool_import+10f () ffffff001f67fc80 zfs:zfsdev_ioctl+4b7 () ffffff001f67fcc0 genunix:cdev_ioctl+39 () ffffff001f67fd10 specfs:spec_ioctl+60 () ffffff001f67fda0 genunix:fop_ioctl+55 () ffffff001f67fec0 genunix:ioctl+9b () ffffff001f67ff10 unix:brand_sys_sysenter+1c9 () This is a important lesson. My impression was that it's not possible to corrupt a complete pool, but there's always a way to recover healthy/redundant data. Now my striped mirror has all 4 devices healthy available, but all datasets seem to be lost. No problem for 450G (99,9_%), but there's a 80M dataset which I'm really missing :-( Unfortunately I don't know the DVA and blkptr internals, so I won't write a zfs_fsck(8) soon ;-) Does it make sense to dump the disks for further analysis? I need to recreate the pool because I need the machine's resources... :-( Any help highly appreciated! -harry