From owner-freebsd-stable@freebsd.org Thu Aug 3 08:54:59 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4EEA1DD1A45 for ; Thu, 3 Aug 2017 08:54:59 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (mail.norma.perm.ru [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C45896E0C3 for ; Thu, 3 Aug 2017 08:54:58 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from bsdrookie.norma.com. (net206-94.perm.ertelecom.ru [46.146.206.94] (may be forged)) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTPS id v738soVu083466 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Thu, 3 Aug 2017 13:54:50 +0500 (YEKT) (envelope-from emz@norma.perm.ru) To: freebsd-stable@FreeBSD.org From: "Eugene M. Zheganin" Subject: panic: dva_get_dsize_sync(): bad DVA on 2016 11-STABLE. Message-ID: <392341d5-3bdc-cf18-6dcc-6e5f07f8e3df@norma.perm.ru> Date: Thu, 3 Aug 2017 13:54:50 +0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Spamd-Result: default: False [1.50 / 25.00] RBL_SPAMHAUS_PBL(2.00)[94.206.146.46.zen.spamhaus.org : 127.0.0.10] HFILTER_HOSTNAME_UNKNOWN(2.50)[] BAYES_HAM(-3.00)[99.99%] DMARC_NA(0.00)[norma.perm.ru] MIME_GOOD(-0.10)[text/plain] R_DKIM_NA(0.00)[] R_SPF_SOFTFAIL(0.00)[~all] RCPT_COUNT_1(0.00)[] MID_RHS_MATCH_FROM(0.00)[] RECEIVED_SPAMHAUS(0.00)[94.206.146.46.zen.spamhaus.org] TO_MATCH_ENVRCPT_ALL(0.00)[] FROM_HAS_DN(0.00)[] TO_DN_NONE(0.00)[] FROM_EQ_ENVFROM(0.00)[] RCVD_COUNT_1(0.00)[] ONCE_RECEIVED(0.10)[] X-Rspamd-Server: localhost X-Rspamd-Scan-Time: 5.11 X-Rspamd-Queue-ID: v738soVu083466 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Aug 2017 08:54:59 -0000 Hi, today I got the following panic on the December 2016 11-STABLE: FreeBSD san02.playkey.net 11.0-STABLE FreeBSD 11.0-STABLE #0 r310734M: Thu Dec 29 19:22:30 UTC 2016 emz@san02:/usr/obj/usr/src/sys/GENERIC amd64 panic: dva_get_dsize_sync(): bad DVA 4294967295:2086400 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: panic: dva_get_dsize_sync(): bad DVA 4294967295:2086400 cpuid = 2 KDB: stack backtrace: #0 0xffffffff80b023a7 at kdb_backtrace+0x67 #1 0xffffffff80ab88e6 at vpanic+0x186 #2 0xffffffff80ab8753 at panic+0x43 #3 0xffffffff8226a148 at bp_get_dsize+0x128 #4 0xffffffff8222cc29 at dmu_tx_count_write+0x589 #5 0xffffffff8222c675 at dmu_tx_hold_write+0x35 #6 0xffffffff822c573e at zvol_strategy+0x21e #7 0xffffffff809f6a10 at g_io_request+0x4a0 #8 0xffffffff8283a45f at ctl_be_block_dispatch_dev+0x20f #9 0xffffffff8283bddc at ctl_be_block_worker+0x6c #10 0xffffffff80b1484a at taskqueue_run_locked+0x14a #11 0xffffffff80b15a38 at taskqueue_thread_loop+0xe8 #12 0xffffffff80a70785 at fork_exit+0x85 #13 0xffffffff80f55f2e at fork_trampoline+0xe Uptime: 78d7h43m31s My question is (since I din't find much on this) what does this "bad DVA" mean ? I've read that this may indicate the on-disk zfs corrupton, but I'm not suer about it. Is this fixable in any way ? Do I have to prepare to recreate the pool (btw I have three pools) from scratch, and how do I determine which one has the corruption. Some [useless ?] zfs info: # zpool status pool: data state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM data ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 raidz1-1 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 da8 ONLINE 0 0 0 da9 ONLINE 0 0 0 raidz1-2 ONLINE 0 0 0 da10 ONLINE 0 0 0 da11 ONLINE 0 0 0 da12 ONLINE 0 0 0 da13 ONLINE 0 0 0 raidz1-3 ONLINE 0 0 0 da14 ONLINE 0 0 0 da15 ONLINE 0 0 0 da16 ONLINE 0 0 0 da17 ONLINE 0 0 0 errors: No known data errors pool: userdata state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM userdata ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/userdata0 ONLINE 0 0 0 gpt/userdata1 ONLINE 0 0 0 errors: No known data errors pool: zroot state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/zroot0 ONLINE 0 0 0 gpt/zroot1 ONLINE 0 0 0 errors: No known data errors Hardware: # camcontrol devlist at scbus4 target 0 lun 0 (pass0,ses0) at scbus11 target 0 lun 0 (pass1,ses1) at scbus12 target 4 lun 0 (pass2,da0) at scbus12 target 5 lun 0 (pass3,da1) at scbus12 target 8 lun 0 (pass4,da2) at scbus12 target 9 lun 0 (pass5,da3) at scbus12 target 10 lun 0 (pass6,da4) at scbus12 target 11 lun 0 (pass7,da5) at scbus12 target 12 lun 0 (pass8,da6) at scbus12 target 13 lun 0 (pass9,da7) at scbus12 target 14 lun 0 (pass10,da8) at scbus12 target 15 lun 0 (pass11,da9) at scbus12 target 16 lun 0 (pass12,da10) at scbus12 target 17 lun 0 (pass13,da11) at scbus12 target 18 lun 0 (pass14,da12) at scbus12 target 19 lun 0 (pass15,da13) at scbus12 target 20 lun 0 (pass16,da14) at scbus12 target 21 lun 0 (pass17,da15) at scbus12 target 22 lun 0 (pass18,da16) at scbus12 target 23 lun 0 (pass19,da17) at scbus12 target 24 lun 0 (pass20,da18) at scbus12 target 32 lun 0 (pass21,ses2) I also have the zdb -uuumdC for each pool, here they are in case someone needs them: https://enaza.ru/stub-data/zdb-uuumdC-data.txt https://enaza.ru/stub-data/zdb-uuumdC-userdata.txt https://enaza.ru/stub-data/zdb-uuumdC-zroot.txt Thanks. Eugene.