From owner-freebsd-fs@freebsd.org Sat Sep 2 17:17:19 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8225FE1D07C for ; Sat, 2 Sep 2017 17:17:19 +0000 (UTC) (envelope-from bsd@vink.pl) Received: from mail-qk0-x231.google.com (mail-qk0-x231.google.com [IPv6:2607:f8b0:400d:c09::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3E0B46A160 for ; Sat, 2 Sep 2017 17:17:19 +0000 (UTC) (envelope-from bsd@vink.pl) Received: by mail-qk0-x231.google.com with SMTP id a77so10865886qkb.1 for ; Sat, 02 Sep 2017 10:17:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vink-pl.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=vv4qsm6Ksdphx/Mmf/s0YoE5/kgaTFZbWwcLMktEk1I=; b=r1mIVyc7HBzyz0TObQ5Z37ZgQpuso2PfmGPJK0uYqY8PftgWjYHCKigfp05zmRDK9K xkzaTDQg5enypPDmi2VuJUmtuoj/z4P/n83pBfw92uiYikdt+BhmEoyOH08TAmIA7Eo1 Z8OVtE3tcLtsxEv73jTfZnsI9zOjcgRar6Khj4ZmxK9ekuWlgqCA7we3is3M10SnYl4+ RquUAI3u/T64rO1s0aRwnG/cZKTqqk7jB+CKMMUZKWDEeAJLH91db7rmIzEaQjhtUQ81 q8zj7Izxkeq7G9NZXiSaxqM4WmnpDlqOugd98PCGi/IAICAQDNutL/0h/n1iZog3eLF7 91tQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=vv4qsm6Ksdphx/Mmf/s0YoE5/kgaTFZbWwcLMktEk1I=; b=VwnX6NqiZqSfHfjhLiZe/UH4OLfH2RgBxCq9uHfczf8dlYSSR2p1r08WV12jEGFh9P /XR5EKWsFTT+aR3a+1o28HxvmiRr5z4nXJXzsc0cdROIMeBVN0jb3XdR4qY+TKQBz0Jh QqK24i7j29xxBVeSzgQSAHoAq4tun5nYZAQVVUi14Rz4NOJGKhXS1aKyPRsp0/W4Jcwk J/3r49RYqk86ICGW2xR3A5P9B6/0EcOVveKsFISQqUfszjQhf1Jz7sanU4Ls31H3gkmw jT6l4cAH4/W/t4g2xTrgCPFiWAaeXF6kPPwBy1+oZ+SwshPHF80FSqITYtS7FdESPlhD S8Mw== X-Gm-Message-State: AHPjjUjRTTLIGrwQKFA852NvnlJ7fXeAwBMJ5PnAaFNcZx6iujn1Eq9J HAXAAJZHdr9DvD/KXUU= X-Received: by 10.55.103.151 with SMTP id b145mr7336659qkc.78.1504372637948; Sat, 02 Sep 2017 10:17:17 -0700 (PDT) Received: from mail-qk0-f179.google.com (mail-qk0-f179.google.com. [209.85.220.179]) by smtp.gmail.com with ESMTPSA id s124sm1097862qkc.5.2017.09.02.10.17.17 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 02 Sep 2017 10:17:17 -0700 (PDT) Received: by mail-qk0-f179.google.com with SMTP id k126so10852912qkb.4 for ; Sat, 02 Sep 2017 10:17:17 -0700 (PDT) X-Google-Smtp-Source: ADKCNb6GmqCGzTeMVt1WfMwdrfmhegV21jekzf3hev3MqpJi+ILoFCUvEGwOLPcMVnUwykNmqbNpOjRZWp0r/xG1sis= X-Received: by 10.233.237.12 with SMTP id c12mr8238177qkg.268.1504372636993; Sat, 02 Sep 2017 10:17:16 -0700 (PDT) MIME-Version: 1.0 Received: by 10.12.156.197 with HTTP; Sat, 2 Sep 2017 10:17:16 -0700 (PDT) In-Reply-To: References: From: Wiktor Niesiobedzki Date: Sat, 2 Sep 2017 19:17:16 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Resolving errors with ZVOL-s To: freebsd-fs Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Sep 2017 17:17:19 -0000 Hi, I have recently encountered errors on my ZFS Pool on my 11.1-R: $ uname -a FreeBSD kadlubek 11.1-RELEASE-p1 FreeBSD 11.1-RELEASE-p1 #0: Wed Aug 9 11:55:48 UTC 2017 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 # zpool status -v tank pool: tank state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: scrub repaired 0 in 5h27m with 0 errors on Sat Sep 2 15:30:59 2017 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 98 mirror-0 ONLINE 0 0 196 gpt/tank1.eli ONLINE 0 0 196 gpt/tank2.eli ONLINE 0 0 196 errors: Permanent errors have been detected in the following files: dkr-test:<0x1> dkr-test is ZVOL that I use within bhyve and indeed - within bhyve I have noticed I/O errors on this volume. This ZVOL did not have any snapshots. Following the advice mentioned in action I tried to restore the ZVOL: # zfs desroy tank/dkr-test But still errors are mentioned in zpool status: errors: Permanent errors have been detected in the following files: <0x5095>:<0x1> I can't find any reference to this dataset in zdb: # zdb -d tank | grep 5095 # zdb -d tank | grep 20629 I tried also getting statistics about metadata in this pool: # zdb -b tank Traversing all blocks to verify nothing leaked ... loading space map for vdev 0 of 1, metaslab 159 of 174 ... No leaks (block sum matches space maps exactly) bp count: 24426601 ganged count: 0 bp logical: 1983127334912 avg: 81187 bp physical: 1817897247232 avg: 74422 compression: 1.09 bp allocated: 1820446928896 avg: 74527 compression: 1.09 bp deduped: 0 ref>1: 0 deduplication: 1.00 SPA allocated: 1820446928896 used: 60.90% additional, non-pointer bps of type 0: 57981 Dittoed blocks on same vdev: 296490 And zdb got stuck using 100% CPU And now to my questions: 1. Do I interpret correctly, that this situation is probably due to error during write, and both copies of the block got checksum mismatching their data? And if it is a hardware problem, it is probably something other than disk? (No, I don't use ECC RAM) 2. Is there any way to remove offending dataset and clean the pool of the errors? 3. Is my metadata OK? Or should I restore entire pool from backup? 4. I tried also running zdb -bc tank, but this resulted in kernel panic. I might try to get the stack trace once I get physical access to machine next week. Also - checksum verification slows down process from 1000MB/s to less than 1MB/s. Is this expected? 5. When I work with zdb (as as above) should I try to limit writes to the pool (e.g. by unmounting the datasets)? Cheers, Wiktor Niesiobedzki PS. Sorry for previous partial message.