From owner-freebsd-stable@freebsd.org Wed Apr 17 09:12:42 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 20AB4158B17D for ; Wed, 17 Apr 2019 09:12:42 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from smtp.fagskolen.gjovik.no (smtp.fagskolen.gjovik.no [IPv6:2001:700:1100:1:200:ff:fe00:b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7ECAE6F524 for ; Wed, 17 Apr 2019 09:12:41 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from mail.fig.ol.no (localhost [127.0.0.1]) by mail.fig.ol.no (8.15.2/8.15.2) with ESMTP id x3H9CYTX030479 for ; Wed, 17 Apr 2019 11:12:34 +0200 (CEST) (envelope-from trond@fagskolen.gjovik.no) Received: from localhost (trond@localhost) by mail.fig.ol.no (8.15.2/8.15.2/Submit) with ESMTP id x3H9CY0m030476 for ; Wed, 17 Apr 2019 11:12:34 +0200 (CEST) (envelope-from trond@fagskolen.gjovik.no) X-Authentication-Warning: mail.fig.ol.no: trond owned process doing -bs Date: Wed, 17 Apr 2019 11:12:34 +0200 (CEST) From: =?ISO-8859-1?Q?Trond_Endrest=F8l?= Sender: Trond.Endrestol@fagskolen.gjovik.no To: FreeBSD stable Subject: Re: ZFS parallel mounting gone wrong? In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21.9999 (BSF 287 2018-06-16) Organization: Fagskolen Innlandet OpenPGP: url=http://fig.ol.no/~trond/trond.key MIME-Version: 1.0 X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.fig.ol.no Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2019 09:12:42 -0000 On Mon, 15 Apr 2019 15:24+0200, Trond Endrestøl wrote: > I upgraded a non-critical system running amd64 stable/12 to r346220. > > During multiuser boot not all ZFS filesystems below zroot/usr/local > was mounted. Some more explaining is in order: This system has two 7 year old pools that complement each other. /usr/local comes mostly from the zroot pool, but other children comes from a zdata pool. The intermediary filesystems have their canmount property set to off and mountpoints are specified at the top level only. The same goes for other parts of the filesystem hierarchy, such as /var/db and /var/spool. I just upgraded to stable/12 global r346269, local r346268. During a routine "zfs mount -av" performed in single user mode, the kernel proceeded to mount a child filesystem (enterprise_zdata/var/db/mysql) without the parent filesystems being mounted first. I rebooted back to r345628 from March 28th, and this kernel has no problems correctly mounting the ZFS filesystems in parallel. That BE used LLVM 7.0.1 from base as its system compiler. Rebooting into r346220 (April 15th) or r346269 (April 17th) clearly shows problems mounting filesystems in the correct order. These BEs was compiled using LLVM 8.0.0 from base. Maybe the system compiler is irrelevant. The name of the pools might also be a factor, the zdata pool preceedes the zroot pool in alphanumerical order. Maybe there is a bug in the code, or the code breaks when parts of the filesystem hierarchy is being built from multiple pools. Here's an attempt at explaining how this fits together: zfs list -ro name,canmount,mountpoint enterprise_zroot/usr enterprise_zdata/usr enterprise_zroot/var enterprise_zdata/var [the list has been slightly edited, moving zdata below zroot and adding an empty line] NAME CANMOUNT MOUNTPOINT enterprise_zroot/usr off /usr enterprise_zroot/usr/compat on /usr/compat enterprise_zroot/usr/local on /usr/local enterprise_zroot/usr/local/certs on /usr/local/certs enterprise_zroot/usr/local/etc on /usr/local/etc enterprise_zroot/usr/local/etc/shellkonfig3 on /usr/local/etc/shellkonfig3 enterprise_zroot/usr/local/etc/shellkonfig3/head on /usr/local/etc/shellkonfig3/head enterprise_zroot/usr/local/etc/shellkonfig3/stable-10 on /usr/local/etc/shellkonfig3/stable-10 enterprise_zroot/usr/local/etc/shellkonfig3/stable-11 on /usr/local/etc/shellkonfig3/stable-11 enterprise_zroot/usr/local/etc/shellkonfig3/stable-8 on /usr/local/etc/shellkonfig3/stable-8 enterprise_zroot/usr/local/etc/shellkonfig3/stable-9 on /usr/local/etc/shellkonfig3/stable-9 enterprise_zroot/usr/local/info on /usr/local/info enterprise_zroot/usr/local/var on /usr/local/var enterprise_zroot/usr/obj on /usr/obj enterprise_zroot/usr/ports on /usr/ports enterprise_zroot/usr/ports/distfiles on /usr/ports/distfiles enterprise_zroot/usr/ports/local off /usr/ports/local enterprise_zroot/usr/ports/packages on /usr/ports/packages enterprise_zroot/usr/ports/workdirs on /usr/ports/workdirs enterprise_zroot/usr/src on /usr/src enterprise_zdata/usr off /usr enterprise_zdata/usr/local off /usr/local enterprise_zdata/usr/local/moodledata on /usr/local/moodledata enterprise_zdata/usr/local/pgsql on /usr/local/pgsql enterprise_zdata/usr/local/restaurering on /usr/local/restaurering enterprise_zdata/usr/local/www on /usr/local/www enterprise_zdata/usr/local/www/moodle on /usr/local/www/moodle enterprise_zroot/var off /var enterprise_zroot/var/Named on /var/Named enterprise_zroot/var/account on /var/account enterprise_zroot/var/audit on /var/audit enterprise_zroot/var/cache off /var/cache enterprise_zroot/var/cache/ccache on /var/cache/ccache enterprise_zroot/var/cache/synth on /var/cache/synth enterprise_zroot/var/crash on /var/crash enterprise_zroot/var/db on /var/db enterprise_zroot/var/db/darkstat on /var/db/darkstat enterprise_zroot/var/db/dkim on /var/db/dkim enterprise_zroot/var/db/etcupdate on /var/db/etcupdate enterprise_zroot/var/db/hyperv on /var/db/hyperv enterprise_zroot/var/db/ntp on /var/db/ntp enterprise_zroot/var/db/pkg on /var/db/pkg enterprise_zroot/var/db/ports on /var/db/ports enterprise_zroot/var/db/sup on /var/db/sup enterprise_zroot/var/empty on /var/empty enterprise_zroot/var/log on /var/log enterprise_zroot/var/mail on /var/mail enterprise_zroot/var/munin on /var/munin enterprise_zroot/var/run on /var/run enterprise_zroot/var/spool on /var/spool enterprise_zroot/var/spool/cvsup on /var/spool/cvsup enterprise_zroot/var/synth on /var/synth enterprise_zroot/var/synth/builders on /var/synth/builders enterprise_zroot/var/synth/live_packages on /var/synth/live_packages enterprise_zroot/var/tmp on /var/tmp enterprise_zroot/var/unbound on /var/unbound enterprise_zdata/var off /var enterprise_zdata/var/db off /var/db enterprise_zdata/var/db/mysql on /var/db/mysql enterprise_zdata/var/db/mysql_secure on /var/db/mysql_secure enterprise_zdata/var/db/mysql_tmpdir on /var/db/mysql_tmpdir enterprise_zdata/var/db/postgres on /var/db/postgres enterprise_zdata/var/db/postgres/data11 on /var/db/postgres/data11 enterprise_zdata/var/db/postgres/data11/base on /var/db/postgres/data11/base enterprise_zdata/var/db/postgres/data11/pg_wal on /var/db/postgres/data11/pg_wal enterprise_zdata/var/db/postgres/data96 on /var/db/postgres/data96 enterprise_zdata/var/db/postgres/data96/base on /var/db/postgres/data96/base enterprise_zdata/var/db/postgres/data96/pg_xlog on /var/db/postgres/data96/pg_xlog enterprise_zdata/var/db/prometheus on /var/db/prometheus enterprise_zdata/var/db/prometheus/data on /var/db/prometheus/data enterprise_zdata/var/db/prometheus/data/wal on /var/db/prometheus/data/wal enterprise_zdata/var/spool off /var/spool enterprise_zdata/var/spool/bareos on /var/spool/bareos enterprise_zdata/var/spool/ftp on /var/spool/ftp Using this remount script in singleuser mode, brings order to chaos: #!/bin/sh # To be run while in singleuser mode, # preferably (re)booted directly to singleuser mode. PATH="/bin:/sbin:/usr/bin:/usr/sbin:/rescue" export PATH killall devd killall moused umount /usr/compat/linux/dev/fd umount /usr/compat/linux/dev umount /usr/compat/linux/proc umount /usr/compat/linux/sys zfs unmount -a for fs in `zfs list -Hro canmount,name enterprise_zroot | grep -v '^off' | grep -v 'enterprise_zroot$' | grep -v 'enterprise_zroot/ROOT' | grep -v 'enterprise_zroot/do-not-destroy' | awk '{print $2}'`; do zfs mount ${fs} done for fs in `zfs list -Hro canmount,name enterprise_zdata | grep -v '^off' | grep -v 'enterprise_zdata$' | grep -v 'enterprise_zdata/do-not-destroy' | awk '{print $2}'`; do zfs mount ${fs} done mount -al echo "You may now attempt to exit to multiuser mode ..." # EOF -- Trond. From owner-freebsd-stable@freebsd.org Wed Apr 17 09:42:42 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8793A158C056 for ; Wed, 17 Apr 2019 09:42:42 +0000 (UTC) (envelope-from ae@FreeBSD.org) Received: from butcher-nb.yandex.net (unknown [IPv6:2a02:6b8:b010:d003::1:13]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "butcher-nb.yandex.net", Issuer "butcher-nb.yandex.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id BF2697091F for ; Wed, 17 Apr 2019 09:42:41 +0000 (UTC) (envelope-from ae@FreeBSD.org) Received: from butcher-nb.yandex.net (localhost [127.0.0.1]) by butcher-nb.yandex.net (8.15.2/8.15.2) with ESMTP id x3H9fGY9030277; Wed, 17 Apr 2019 12:41:22 +0300 (MSK) (envelope-from ae@FreeBSD.org) Subject: Re: Panic during reboot involving softclock_call_cc(), nd6_timer() and nd6_dad_start() To: =?UTF-8?Q?Trond_Endrest=c3=b8l?= , FreeBSD stable References: From: "Andrey V. Elsukov" Openpgp: id=E6591E1B41DA1516F0C9BC0001C5EA0410C8A17A Autocrypt: addr=ae@FreeBSD.org; prefer-encrypt=mutual; keydata= mQENBEwBF1kBCADB9sXFhBEUy8qQ4X63Y8eBatYMHGEFWN9ypS5lI3RE6qQW2EYbxNk7qUC5 21YIIS1mMFVBEfvR7J9uc7yaYgFCEb6Sce1RSO4ULN2mRKGHP3/Sl0ijZEjWHV91hY1YTHEF ZW/0GYinDf56sYpDDehaBF5wkWIo1+QK5nmj3vl0DIDCMNd7QEiWpyLVwECgLX2eOAXByT8B bCqVhJGcG6iFP7/B9Ll6uX5gb8thM9LM+ibwErDBVDGiOgvfxqidab7fdkh893IBCXa82H9N CNwnEtcgzh+BSKK5BgvPohFMgRwjti37TSxwLu63QejRGbZWSz3OK3jMOoF63tCgn7FvABEB AAG0IkFuZHJleSBWLiBFbHN1a292IDxhZUBmcmVlYnNkLm9yZz6JATsEEwECACUCGwMGCwkI BwMCBhUIAgkKCwQWAgMBAh4BAheABQJMB/ruAhkBAAoJEAHF6gQQyKF6MLwH/3Ri/TZl9uo0 SepYWXOnxL6EaDVXDA+dLb1eLKC4PRBBjX29ttQ0KaWapiE6y5/AfzOPmRtHLrHYHjd/aiHX GMLHcYRXD+5GvdkK8iMALrZ28X0JXyuuZa8rAxWIWmCbYHNSBy2unqWgTI04Erodk90IALgM 9JeHN9sFqTM6zalrMnTzlcmel4kcjT3lyYw3vOKgoYLtsLhKZSbJoVVVlvRlGBpHFJI5AoYJ SyfXoN0rcX6k9X7Isp2K50YjqxV4v78xluh1puhwZyC0p8IShPrmrp9Oy9JkMX90o6UAXdGU KfdExJuGJfUZOFBTtNIMNIAKfMTjhpRhxONIr0emxxC5AQ0ETAEXWQEIAJ2p6l9LBoqdH/0J PEFDY2t2gTvAuzz+8zs3R03dFuHcNbOwjvWCG0aOmVpAzkRa8egn5JB4sZaFUtKPYJEQ1Iu+ LUBwgvtXf4vWpzC67zs2dDuiW4LamH5p6xkTD61aHR7mCB3bg2TUjrDWn2Jt44cvoYxj3dz4 S49U1rc9ZPgD5axCNv45j72tggWlZvpefThP7xT1OlNTUqye2gAwQravXpZkl5JG4eOqJVIU X316iE3qso0iXRUtO7OseBf0PiVmk+wCahdreHOeOxK5jMhYkPKVn7z1sZiB7W2H2TojbmcK HZC22sz7Z/H36Lhg1+/RCnGzdEcjGc8oFHXHCxUAEQEAAYkBHwQYAQIACQUCTAEXWQIbDAAK CRABxeoEEMihegkYCAC3ivGYNe2taNm/4Nx5GPdzuaAJGKWksV+w9mo7dQvU+NmI2az5w8vw 98OmX7G0OV9snxMW+6cyNqBrVFTu33VVNzz9pnqNCHxGvj5dL5ltP160JV2zw2bUwJBYsgYQ WfyJJIM7l3gv5ZS3DGqaGIm9gOK1ANxfrR5PgPzvI9VxDhlr2juEVMZYAqPLEJe+SSxbwLoz BcFCNdDAyXcaAzXsx/E02YWm1hIWNRxanAe7Vlg7OL+gvLpdtrYCMg28PNqKNyrQ87LQ49O9 50IIZDOtNFeR0FGucjcLPdS9PiEqCoH7/waJxWp6ydJ+g4OYRBYNM0EmMgy1N85JJrV1mi5i Message-ID: Date: Wed, 17 Apr 2019 12:41:16 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="lKLtm7DNWRtYUeCc0Lw8PZEWgm4ew9agc" X-Rspamd-Queue-Id: BF2697091F X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.99 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; NEURAL_HAM_SHORT(-0.99)[-0.988,0]; ASN(0.00)[asn:13238, ipnet:2a02:6b8::/32, country:RU]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2019 09:42:42 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --lKLtm7DNWRtYUeCc0Lw8PZEWgm4ew9agc Content-Type: multipart/mixed; boundary="zDSNDJ1vDRQQMOQzD6czJNOhKP2EyxQmF"; protected-headers="v1" From: "Andrey V. Elsukov" To: =?UTF-8?Q?Trond_Endrest=c3=b8l?= , FreeBSD stable Message-ID: Subject: Re: Panic during reboot involving softclock_call_cc(), nd6_timer() and nd6_dad_start() References: In-Reply-To: --zDSNDJ1vDRQQMOQzD6czJNOhKP2EyxQmF Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 15.04.2019 16:31, Trond Endrest=C3=B8l wrote: > Has anyone else witnessed a panic during reboot involving=20 > softclock_call_cc(), nd6_timer(), and nd6_dad_start()? >=20 > The stack trace goes more or less like this: >=20 > db_trace_self_wrapper() > vpanic() > panic() > trap_fatal() > trap() > calltrap() > nd6_dad_start() > nd6_timer() > softclock_call_cc() > softclock() > ithread_loop() > fork_exit() > fork_trampoline() >=20 > This was last seen while transitioning from r345628 to r346220 on=20 > amd64 stable/12. Hi, do you have exact panic message and/or backtrace from core dump? It would be good to submit PR about such problems. --=20 WBR, Andrey V. Elsukov --zDSNDJ1vDRQQMOQzD6czJNOhKP2EyxQmF-- --lKLtm7DNWRtYUeCc0Lw8PZEWgm4ew9agc Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQEzBAEBCAAdFiEE5lkeG0HaFRbwybwAAcXqBBDIoXoFAly29LwACgkQAcXqBBDI oXraEgf+LdzH0ZILT9jNUW19NuSCLHH+TRvEMdAC5HrYODOxaNwW6rwGFpOBzjXG 3JdTwGzevnOj00aRkVaBfkt+gM49QFgXeVuRl4NxSMgG2RaXYpiB0kIpcc8Erx7R e2IBg4vpFlEqqOkB6ESKquE9cA4XR+1BqdMH05NrXoQlUi4vm2pUw8YERKhBAypb xJyU7IoALMS/4uA/fAlPXx5A7lzRuCZ+HmqjVrAMcmBkHTUfivEZkIafGAxypb5V Kg3UdVJMV8ttKFA0jBtLxwzbvaLa+h9ORUxtrsL2N1dOEXjHGfQpGOwWfrKw5GO6 BkIFfvGHfYZ7xe8CL4c8L1nfavIk0A== =4XV1 -----END PGP SIGNATURE----- --lKLtm7DNWRtYUeCc0Lw8PZEWgm4ew9agc--