Date: Tue, 3 Dec 2019 11:03:11 +0200 From: Andriy Gapon <avg@FreeBSD.org> To: Mark Johnston <markj@freebsd.org>, Peter Eriksson <pen@lysator.liu.se> Cc: freebsd-fs@freebsd.org Subject: Re: Slow reboots due to ZFS cleanup in kern_shutdown() .. zio_fini() Message-ID: <3b71fe37-c29f-e3e5-ff96-5dce15cc7553@FreeBSD.org> In-Reply-To: <20191202225424.GG43802@raichu> References: <AD17E454-6A51-436D-A853-07F04A406EC9@lysator.liu.se> <D2A11CE9-9B24-4E40-A51A-8D318E0288C9@lysator.liu.se> <20191202225424.GG43802@raichu>
next in thread | previous in thread | raw e-mail | index | archive | help
On 03/12/2019 00:54, Mark Johnston wrote: > On Mon, Dec 02, 2019 at 11:39:01PM +0100, Peter Eriksson wrote: >> Sigh. >> >> Slight correction, the output below should have said uma_zdestroy() and not uma_zfree_arg() (wrong printf text, but the right times). >> >> After an uptime of 7 hours, a reboot have these times (I removed the “uma” printf in this run): >> >> kmem_cache_destroy(zio_data_buf_cache[8]) took 2 seconds >> kmem_cache_destroy(zio_buf_cache[10]) took 6 seconds >> kmem_cache_destroy(zio_buf_cache[14]) took 2 seconds >> kmem_cache_destroy(zio_buf_cache[16]) took 136 seconds >> kmem_cache_destroy(zio_buf_cache[20]) took 31 seconds >> kmem_cache_destroy(zio_buf_cache[28]) took 303 seconds >> kmem_cache_destroy(zio_buf_cache[224]) took 89 seconds >> kmem_cache_destroy(zio_data_buf_cache[224]) took 31 seconds >> >> This is on a mostly idle server (well, apart from compiling the kernel code :-) and some snapshots being taken of all filesystems (once per hour). >> >> >> So now on to finding out why uma_destroy() is taking so long… :-). > > uma_destroy() frees all of the memory cached in the zone back to the > page allocator. This operation takes time proportional to the number of > cached items. I would expect most of the time to be spent in > zone_reclaim(), called by zone_dtor(). But spending *minutes* there is really unexpected. I have never seen anything like that. I wonder if there is anything untypical about the system's hardware (like a very big number of processors) or configuration. -- Andriy Gapon
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3b71fe37-c29f-e3e5-ff96-5dce15cc7553>