Date: Mon, 2 Dec 2019 17:54:24 -0500 From: Mark Johnston <markj@freebsd.org> To: Peter Eriksson <pen@lysator.liu.se> Cc: freebsd-fs@freebsd.org Subject: Re: Slow reboots due to ZFS cleanup in kern_shutdown() .. zio_fini() Message-ID: <20191202225424.GG43802@raichu> In-Reply-To: <D2A11CE9-9B24-4E40-A51A-8D318E0288C9@lysator.liu.se> References: <AD17E454-6A51-436D-A853-07F04A406EC9@lysator.liu.se> <D2A11CE9-9B24-4E40-A51A-8D318E0288C9@lysator.liu.se>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Dec 02, 2019 at 11:39:01PM +0100, Peter Eriksson wrote: > Sigh. > > Slight correction, the output below should have said uma_zdestroy() and not uma_zfree_arg() (wrong printf text, but the right times). > > After an uptime of 7 hours, a reboot have these times (I removed the “uma” printf in this run): > > kmem_cache_destroy(zio_data_buf_cache[8]) took 2 seconds > kmem_cache_destroy(zio_buf_cache[10]) took 6 seconds > kmem_cache_destroy(zio_buf_cache[14]) took 2 seconds > kmem_cache_destroy(zio_buf_cache[16]) took 136 seconds > kmem_cache_destroy(zio_buf_cache[20]) took 31 seconds > kmem_cache_destroy(zio_buf_cache[28]) took 303 seconds > kmem_cache_destroy(zio_buf_cache[224]) took 89 seconds > kmem_cache_destroy(zio_data_buf_cache[224]) took 31 seconds > > This is on a mostly idle server (well, apart from compiling the kernel code :-) and some snapshots being taken of all filesystems (once per hour). > > > So now on to finding out why uma_destroy() is taking so long… :-). uma_destroy() frees all of the memory cached in the zone back to the page allocator. This operation takes time proportional to the number of cached items. I would expect most of the time to be spent in zone_reclaim(), called by zone_dtor(). Before rebooting, could you try grabbing the output of vmstat -z?
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20191202225424.GG43802>