From owner-freebsd-fs@FreeBSD.ORG Fri Mar 21 10:09:10 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C7869277 for ; Fri, 21 Mar 2014 10:09:10 +0000 (UTC) Received: from elf.hq.norma.perm.ru (mail.norma.perm.ru [IPv6:2001:470:1f09:14c0::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id DEFA22F2 for ; Fri, 21 Mar 2014 10:09:09 +0000 (UTC) Received: from bsdrookie.norma.com. (bsdrookie.norma.com [192.168.7.224]) by elf.hq.norma.perm.ru (8.14.5/8.14.5) with ESMTP id s2LA97tb097789 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Fri, 21 Mar 2014 16:09:07 +0600 (YEKT) (envelope-from emz@norma.perm.ru) Message-ID: <532C0FC3.5060108@norma.perm.ru> Date: Fri, 21 Mar 2014 16:09:07 +0600 From: "Eugene M. Zheganin" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: freebsd-fs Subject: Re: crash on zpool import - help get data back References: <532BEABC.5050808@norma.perm.ru> <532BF531.1050400@norma.perm.ru> In-Reply-To: <532BF531.1050400@norma.perm.ru> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (elf.hq.norma.perm.ru [192.168.3.10]); Fri, 21 Mar 2014 16:09:07 +0600 (YEKT) X-Spam-Status: No hits=-101.0 bayes=0.5 testhits ALL_TRUSTED=-1, USER_IN_WHITELIST=-100 autolearn=unavailable version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on elf.hq.norma.perm.ru X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Mar 2014 10:09:10 -0000 Hi. On 21.03.2014 14:15, Eugene M. Zheganin wrote: > > After some thinking (speeded up with the superiors running in circles) I > realized that the root cause is the same and I can apply the tricks > mentioned above. I did them, I'm able to import the pool and read some data, but when trying to read all of it I get panic on 43th gigabyte out of 1200 (attached below). Is there some way to get this hack-patched too ? Just to save data. ===Cut=== Fatal trap 12: page fault while in kernel mode cpuid = 3; apic id = 03 fault virtual address = 0x88 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff818a0454 stack pointer = 0x28:0xfffffe01214086e0 frame pointer = 0x28:0xfffffe0121408740 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 1282 (zfs) trap number = 12 panic: page fault cpuid = 3 KDB: stack backtrace: #0 0xffffffff808f01d0 at kdb_backtrace+0x60 #1 0xffffffff808b7ba5 at panic+0x155 #2 0xffffffff80c98f32 at trap_fatal+0x3a2 #3 0xffffffff80c99209 at trap_pfault+0x2c9 #4 0xffffffff80c9899b at trap+0x5bb #5 0xffffffff80c7fc52 at calltrap+0x8 #6 0xffffffff818aeed5 at zio_checksum_verify+0x65 #7 0xffffffff818abada at zio_execute+0x15a #8 0xffffffff818ab0b3 at zio_wait+0x23 #9 0xffffffff81839f83 at arc_read+0x783 #10 0xffffffff8184e04f at backup_cb+0x35f #11 0xffffffff8185243d at traverse_visitbp+0x20d #12 0xffffffff81852e6f at traverse_dnode+0xef #13 0xffffffff81852bb7 at traverse_visitbp+0x987 #14 0xffffffff81852613 at traverse_visitbp+0x3e3 #15 0xffffffff81852613 at traverse_visitbp+0x3e3 #16 0xffffffff81852613 at traverse_visitbp+0x3e3 #17 0xffffffff81852613 at traverse_visitbp+0x3e3 Uptime: 10m14s Dumping 426 out of 4043 MB:..4%..12%..23%..34%..42%..53%..64%..72%..83%..94% Reading symbols from /boot/kernel/zfs.ko.symbols...done. Loaded symbols for /boot/kernel/zfs.ko.symbols Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. Loaded symbols for /boot/kernel/opensolaris.ko.symbols Reading symbols from /boot/kernel/geom_mirror.ko.symbols...done. Loaded symbols for /boot/kernel/geom_mirror.ko.symbols #0 doadump (textdump=) at pcpu.h:219 219 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=) at pcpu.h:219 #1 0xffffffff808b7820 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:452 #2 0xffffffff808b7be4 in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:759 #3 0xffffffff80c98f32 in trap_fatal (frame=, eva=) at /usr/src/sys/amd64/amd64/trap.c:875 #4 0xffffffff80c99209 in trap_pfault (frame=0xfffffe0121408630, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:692 #5 0xffffffff80c9899b in trap (frame=0xfffffe0121408630) at /usr/src/sys/amd64/amd64/trap.c:456 #6 0xffffffff80c7fc52 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:232 #7 0xffffffff818a0454 in zfs_ereport_start_checksum (spa=0xfffff80099c0c000, vd=0x0, zio=0xfffff80117eed000, offset=0, length=512, arg=, info=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_fm.c:704 #8 0xffffffff818aeed5 in zio_checksum_verify (zio=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2858 #9 0xffffffff818abada in zio_execute (zio=0xfffff80117eed000) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1346 #10 0xffffffff818ab0b3 in zio_wait (zio=0xfffff80117eed000) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1370 #11 0xffffffff81839f83 in arc_read (pio=0x0, spa=0xfffff80099c0c000, bp=, done=, private=0x0, priority=ZIO_PRIORITY_ASYNC_READ, zio_flags=512, arc_flags=0xffffffff81838f02, zb=0x0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:3414 #12 0xffffffff8184e04f in backup_cb (spa=0xfffff80099c0c000, zilog=, bp=0xfffffe005c9de780, zb=0xfffffe0121408b08, dnp=, arg=0xfffff8006e402900) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c:422 #13 0xffffffff8185243d in traverse_visitbp (td=0xfffffe01214095f0, dnp=0xfffffe005c9de600, bp=0xfffffe005c9de780, zb=0xfffffe0121408b08) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:250 #14 0xffffffff81852e6f in traverse_dnode (td=0xfffffe01214095f0, dnp=0xfffffe005c9de600, objset=236, object=104707) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:417 #15 0xffffffff81852bb7 in traverse_visitbp (td=0xfffffe01214095f0, dnp=0xfffffe005c9de000, bp=0xfffffe0056cf9400, zb=0xfffffe0121408d28) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:309 #16 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01214095f0, dnp=0xfffff800a68a5000, bp=0xfffffe002c653c80, zb=0xfffffe0121408e58) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284 #17 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01214095f0, dnp=0xfffff800a68a5000, bp=0xfffffe002c74a000, zb=0xfffffe0121408f88) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284 #18 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01214095f0, dnp=0xfffff800a68a5000, bp=0xfffffe002c71a000, zb=0xfffffe01214090b8) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284 #19 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01214095f0, dnp=0xfffff800a68a5000, bp=0xfffffe002c37d000, zb=0xfffffe01214091e8) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284 #20 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01214095f0, dnp=0xfffff800a68a5000, bp=0xfffffe002c726000, zb=0xfffffe0121409318) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284 #21 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01214095f0, dnp=0xfffff800a68a5000, bp=0xfffff800a68a5040, zb=0xfffffe01214093d8) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284 #22 0xffffffff81852e04 in traverse_dnode (td=0xfffffe01214095f0, dnp=0xfffff800a68a5000, objset=236, object=0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:407 #23 0xffffffff818528c0 in traverse_visitbp (td=0xfffffe01214095f0, dnp=0x0, bp=0xfffff800a6b4ce80, zb=0xfffffe0121409588) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:338 #24 0xffffffff81851e5c in traverse_impl (spa=, ds=, objset=, rootbp=0xfffff800a6b4ce80, txg_start=, resume=, flags=, func=0xffffffff8184dcf0 , arg=0xfffff8006e402900) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:540 #25 0xffffffff81851bf3 in traverse_dataset (ds=0xfffffe01214087b0, txg_start=0, flags=, func=0x50, arg=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:563 #26 0xffffffff8184b701 in dmu_send_impl (tag=0xffffffff8192d305, dp=0xfffff80016151400, ds=0xfffff800a64d6c00, fromds=, outfd=-1504875032, fp=0xfffff800a64d6e08) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c:551 #27 0xffffffff8184b3a7 in dmu_send_obj (pool=, tosnap=, fromsnap=, outfd=1, fp=0xfffff80086e6e2d0, off=0xfffffe01214097c0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c:620 #28 0xffffffff818c2efa in zfs_ioc_send (zc=0xfffffe002c7a6000) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:4273 #29 0xffffffff818bf008 in zfsdev_ioctl (dev=, zcmd=, arg=, flag=, td=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:5960 #30 0xffffffff807b2f4f in devfs_ioctl_f (fp=0xfffff80086e6e370, com=3222821404, data=0xfffff80086cd19e0, cred=, td=0xfffff80026e1f000) at /usr/src/sys/fs/devfs/devfs_vnops.c:757 #31 0xffffffff8090680e in kern_ioctl (td=0xfffff80026e1f000, fd=, com=18446741879539140528) at file.h:319 #32 0xffffffff8090658f in sys_ioctl (td=0xfffff80026e1f000, uap=0xfffffe0121409a40) at /usr/src/sys/kern/sys_generic.c:702 #33 0xffffffff80c99827 in amd64_syscall (td=0xfffff80026e1f000, traced=0) at subr_syscall.c:134 #34 0xffffffff80c7ff3b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:391 #35 0x00000008019e413a in ?? () Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal (kgdb) ===Cut=== Thanks. Eugene.