From owner-freebsd-fs@freebsd.org Mon Nov 11 20:14:02 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 28BE81BD832 for ; Mon, 11 Nov 2019 20:14:02 +0000 (UTC) (envelope-from jonathan.anderson@mun.ca) Received: from esa07.ucs.mun.ca (esa07.ucs.mun.ca [134.153.136.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx.mun.ca", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47Bhsh2MhVz4cM1 for ; Mon, 11 Nov 2019 20:14:00 +0000 (UTC) (envelope-from jonathan.anderson@mun.ca) IronPort-SDR: Jn8I7aimyPFCiTiKnktTh4UVR7WUo/CmMb+p6Iaa4/2nuHot7TCQ3uXEArGNc6zxIHqsv+rxf/ xFBj5eAOZDDXE0H9pESkcFfoC16oHY6Oq1OexYWgxAy+LV5n0f5HtoLAyEXTJuB7YYsXOJhtFf +T+En1XylhU1QnT4HYg+9zYl1rmNYiwdXY0sroOszJHYWl3eFO7WqWAN4qPFx5qTTuxaoIo4gK Lmur9QcfUWD7pJAVJWkvJ8M4HA5ayK/CYNLVszUiSsBXzrPShxhaootrkODoIBsJg/KRt3yzPe mhs= X-IronPort-AV: E=Sophos;i="5.68,293,1569292200"; d="scan'208";a="19610839" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from mail-qk1-f198.google.com ([209.85.222.198]) by smtp07.ucs.mun.ca with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Nov 2019 16:43:50 -0430 Received: by mail-qk1-f198.google.com with SMTP id a186so8556587qkb.18 for ; Mon, 11 Nov 2019 12:13:49 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=amC9uFo3vRqZuj9QBGBMtACBJWbBy+kmYH+tHB4N5mI=; b=m2B13qCjuK8aSD+3+5jdQw/egByUXQCX4RScVcpmOI1tuH1mm+VA7N0Yu9R7BSCoq8 jrTjx8XX+vkRl3Fqx3FkDZERsP6yypNlqRAL7GiWGc3JLRvX+AFMFhNfa1hdBF0CziBR zMaLtToj1s/eRU4D0iog+idqiTfjk9FOF8DHDLsc/aWwxXia1ZpdGsszlu3Oc07DOPik NOO2hslkkM/MGW9t16PGcpDfZeJ3GBZ+v8oyx7oQDhGBRp2xx3j2LetoqUWdaOu1Zzdt DQROBpO1mkHgrySRELbKw05uzMm4IlHB+ukGEAn18g2dMUgCQ2i6nxq4S20PEZ1vvG2W MhAw== X-Gm-Message-State: APjAAAWJKnxgKrQR9EhDrmTH+tcdDBYGJxmVQFNo8MY76SBRR3qiBEa+ Uwt+wpGPcfjIQ9qVDrrkiefQeuWwh65P4N7FTbxPUdDzKhbcohH1TkLysKQy76fiktTswKpdUzD VCLTaUO35DdNFsj2j X-Received: by 2002:ac8:342b:: with SMTP id u40mr27383799qtb.87.1573503228534; Mon, 11 Nov 2019 12:13:48 -0800 (PST) X-Google-Smtp-Source: APXvYqy8WDOto0aHYpD0KiRTmyqNQfKhIkUrzcITCKlfTkSmGLCi0gnGQdsPYGQ+s0t5H+y8SPpT5w== X-Received: by 2002:ac8:342b:: with SMTP id u40mr27383770qtb.87.1573503228171; Mon, 11 Nov 2019 12:13:48 -0800 (PST) Received: from bagstock.jonandchrissy.ca (CPE00fc8db75533-CM00fc8db75530.cpe.net.cable.rogers.com. [174.117.179.42]) by smtp.gmail.com with ESMTPSA id r36sm10095298qta.27.2019.11.11.12.13.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Nov 2019 12:13:47 -0800 (PST) Date: Mon, 11 Nov 2019 16:43:45 -0330 From: Jonathan Anderson To: Andriy Gapon Cc: "freebsd-fs@FreeBSD.org" Subject: Re: Broken ZFS boot on upgrade Message-ID: <20191111201345.GA79423@bagstock.jonandchrissy.ca> References: <1cb4895b-c84d-6204-18fa-53eac7195ad6@FreeBSD.org> <20191111131021.GC70914@bagstock.jonandchrissy.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.12.2 (2019-09-21) X-Rspamd-Queue-Id: 47Bhsh2MhVz4cM1 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of jonathan.anderson@mun.ca has no SPF policy when checking 134.153.136.27) smtp.mailfrom=jonathan.anderson@mun.ca X-Spamd-Result: default: False [-1.25 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.991,0]; NEURAL_HAM_LONG(-1.00)[-0.998,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[mun.ca]; AUTH_NA(1.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[27.136.153.134.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6579, ipnet:134.153.0.0/16, country:CA]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-0.16)[ipnet: 134.153.0.0/16(-0.40), asn: 6579(-0.32), country: CA(-0.09)] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Nov 2019 20:14:02 -0000 On 11/11, Andriy Gapon wrote: > You need to find out the inode number first (ls -i or stat -s). > Then you can use zdb -ddddd That's very helpful, thanks! I've attempted to do this, but zdb is giving surprising errors. My root dataset is mounted at /mnt/zroot: zroot/ROOT/default on /mnt/zroot (zfs, local, noatime, nfsv4acls) and ls -i /mnt/zroot gives me inode numbers for things in / (including /boot), but when I execute zdb with that dataset and inode number I get an error: # zdb -ddddd zroot/ROOT/default 4495 failed to own dataset 'zroot/ROOT/default': No such file or directory zdb: can't open 'zroot/ROOT/default': No such file or directory This despite zpool status saying that everything is fine, me being able to run `find /boot` with no errors, etc. I get the same error whether the dataset is mounted or unmounted. At this point I'm probably best off flattening this system and restoring my most recent backup snapshot, but this behaviour is truly curious. Thanks to everyone for your help, Jon -- Jonathan Anderson jonathan@FreeBSD.org