From owner-freebsd-current@freebsd.org Wed Nov 18 08:47:31 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 919122E7475 for ; Wed, 18 Nov 2020 08:47:31 +0000 (UTC) (envelope-from mpp302@gmail.com) Received: from mail-ed1-f68.google.com (mail-ed1-f68.google.com [209.85.208.68]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CbbzQ0SzGz4qxg for ; Wed, 18 Nov 2020 08:47:29 +0000 (UTC) (envelope-from mpp302@gmail.com) Received: by mail-ed1-f68.google.com with SMTP id q16so1096290edv.10 for ; Wed, 18 Nov 2020 00:47:29 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=BTk/DNquzQzwsVw+Lw0GzcuYueoN5GHe8lItzohwFhw=; b=JTv3xM7hGBqHXwP+H+zHNwq4H8aui6h1rQUVQJj/f9YxYKQyyTfB2ErEif1+yv1t2m ancrZFv83C3L/NyO6MVOca1UcOPpCPBdB0AYxlW5bGwtXnJpQhhiSwmMT36Rv2IjWidy KaUUmfVUuVTbnmEwrCykgrK1ihNWew5ID4Xxzz8eZcYzKme5pRg6TO5g2ZGXT6kj31Bw 2+1gqLSYQO21R5l2ODEEmmSHo4qnmLy2TzrfyQMdJCeHLPMkc14UoPgXZb7WFxMGRW00 ZOcxAEvi8q1IS/z4FTM4otYuvzBw2l+1yLoo/x0VUG56Xpaan9ty0YABcf3SYYRjnZV4 sRBw== X-Gm-Message-State: AOAM5323MEXPaxa+s0H57t7tI074b0UWbsbiQA4i25XKTld9P5ckOvxF q+v+w+RiKTgtld+nfwLME3CCagoQe30= X-Google-Smtp-Source: ABdhPJxxYDPNWgeX7ad8iW17p4tkG+Zdr9qAmGElZgAIQFYepwuBwbUxsks9bDTZpooSiaLe532EcQ== X-Received: by 2002:a05:6402:154b:: with SMTP id p11mr25194430edx.217.1605689248013; Wed, 18 Nov 2020 00:47:28 -0800 (PST) Received: from ?IPv6:2a02:8109:98c0:1bc0:5e5f:67ff:fef4:ffd8? ([2a02:8109:98c0:1bc0:5e5f:67ff:fef4:ffd8]) by smtp.gmail.com with ESMTPSA id ks18sm1585657ejb.67.2020.11.18.00.47.26 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 18 Nov 2020 00:47:26 -0800 (PST) Subject: Re: Shutdown errors and timeout To: Johan Hendriks , freebsd-current@freebsd.org References: <65b1ff51-a946-61d0-79d9-104c1e053554@gmail.com> <20201113.200459.520180046556100070.yasu@utahime.org> <20201114091951.4888878c686d07ad73e55da8@dec.sakura.ne.jp> <7316979e-1a87-791a-075c-7f3d7a75f43f@FreeBSD.org> From: Mateusz Piotrowski <0mp@FreeBSD.org> Message-ID: Date: Wed, 18 Nov 2020 09:47:32 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.4.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4CbbzQ0SzGz4qxg X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; RCVD_COUNT_THREE(0.00)[3]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com,freebsd.org]; FORGED_SENDER(0.30)[0mp@FreeBSD.org,mpp302@gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; R_DKIM_NA(0.00)[]; FROM_NEQ_ENVFROM(0.00)[0mp@FreeBSD.org,mpp302@gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[209.85.208.68:from]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; SPAMHAUS_ZRD(0.00)[209.85.208.68:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[209.85.208.68:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.208.68:from]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-current] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Nov 2020 08:47:31 -0000 On 11/16/20 7:16 PM, Johan Hendriks wrote: > > On 14/11/2020 13:03, Mateusz Piotrowski wrote: >> On 11/14/20 1:19 AM, Tomoaki AOKI wrote: >>> On Fri, 13 Nov 2020 20:04:59 +0900 (JST) >>> Yasuhiro KIMURA wrote: >>> >>>> From: Johan Hendriks >>>> >>>>> Hello all, i have two FreeBSD 13 machines, one is a bare metal and one >>>>> is virtualbox machine which i both update about once a week. >>>>> >>>>> The vritual machine seems to fail stopping something and gives a >>>>> timeout after 90 sec. >>>>> >>>>> The console ends with >>>>> >>>>> Writing entropy file: . >>>>> Writing early boot entropy file: . >>>>> >>>>> 90 second watchdog timeout expired. Shutdown terminated. >>>>> Fri Nov13 11:20:40 CEST 2020 >>>>> Nov 13 11:20:40 test-head init[1]: /etc/rc.shutdown terminated >>>>> abnormally, going to single user mode >>>>> ... >>>>> >>>>> On the bare metal machine i see the following. >>>>> Writing entropy file: . >>>>> Writing early boot entropy file: . >>>>> cannot unmount '/var/run': umount failed >>>>> cannot unmount '/var/log': umount failed >>>>> cannot unmount '/var': umount failed >>>>> cannot unmount '/usr/home': umount failed >>>>> cannot unmount '/usr': umount failed >>>>> cannot unmount '/': umount failed >>>>> >>>> (snip) >>>>> The pools have not been upgraded after the latest openzfs import, >>>>> maybe that is related? >>>>> >>>>> FreeBSD test-freebsd-head 13.0-CURRENT FreeBSD 13.0-CURRENT #2 >>>>> r367585: >>>>> >>>>> First thing i noticed is about a week ago. >>>> I'm facing same problem with 13.0-CURRENT amd64 r367487 and >>>> virtualbox. In my case I use autofs to mount remote file system of >>>> 12.2-RELEASE amd64 server with NFSv4. When there is still filesystem >>>> mounted by autofs, then watchdog timeout happens while shutdown. The >>>> watchdog timeout can be worked around by executing `automount -fu` >>>> before shutting down. But 'cannot unmount ...' error messages are >>>> still displayed. >>>> >>>> I added 'rc_debug="YES"' to /etc/rc.conf and checked which rc script >>>> causes this message. Then it is displayed when following `zfs_stop` >>>> function of /etc/rc.d/zfs is executed. >>>> >>>> ---------------------------------------------------------------------- >>>> zfs_stop_main() >>>> { >>>>     zfs unshare -a >>>>     zfs unmount -a >>>> } >>>> ---------------------------------------------------------------------- >>>> >>>> At this point syslog process still running and it opens some files >>>> under /var/log. So it make sence that `zfs unmount -a` results in the >>>> message. >>>> >>>> Probably order of executing each rc script in shutdown time should be >>>> changed so `/etc/rc.d/zfs faststop` is executed after all processes >>>> other than `init' are exited. >>> This happens on stable/12, too. >>> As a workaround, reverting r367291 on head (r367546 on stable/12) >>> would stop the issue until this is really fixed. >>> >>> If you have shared dataset or jail(s) mounting dataset, the workaround >>> would be discouraged. Read commit message for detail. >> >> I've committed r367291 and r367546. >> >> I am not sure if I can think of a proper fix for the described issues, so I guess the best idea >> would be to revert those changes for now until we figure out how to do it properly. > > > I can tell that reverting the mentioned commit i do not have the symptoms when i reboot my servers. > Thank you all for your time, and no sorry needed ;-) I'll revert the commit then. I'm just waiting for an approval from a src committer. Best, Mateusz