Date: Tue, 03 Sep 2013 20:35:56 +1000 From: Grant Gray <grant@gray.id.au> To: Andriy Gapon <avg@FreeBSD.org> Cc: freebsd-fs@FreeBSD.org, Grant Gray <grant@grantgray.id.au> Subject: Re: ZFS livelock / deadlock on pure SSD pool Message-ID: <5225BB8C.5050802@gray.id.au> In-Reply-To: <5225AB77.9020208@FreeBSD.org> References: <522599A9.9070107@grantgray.id.au> <5225AB77.9020208@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 3/09/2013 7:27 PM, Andriy Gapon wrote: > on 03/09/2013 11:11 Grant Gray said the following: >> I haven't yet enabled the kernel debugger to get a stack trace/lock status, but >> procstat -kk -a is here: >> http://pastebin.com/raw.php?i=SYhmyhGj > I believe that this another ARC deadlock triggered by low memory condition. > This time it seems to be FreeBSD-specific too: > > 6 100059 zfskern arc_reclaim_thre mi_switch+0x194 sleepq_wait+0x42 > _sx_xlock_hard+0x4d6 _sx_xlock+0x75 arc_buf_remove_ref+0x8a > dbuf_rele_and_unlock+0x132 dbuf_evict+0x11 dbuf_do_evict+0x53 > arc_do_user_evicts+0xe2 arc_reclaim_thread+0x264 fork_exit+0x11f fork_trampoline+0xe > > 5338 102410 vorbisgain - mi_switch+0x194 sleepq_wait+0x42 > _sx_xlock_hard+0x4d6 _sx_xlock+0x75 arc_lowmem+0x38 kmem_malloc+0xb0 > uma_large_malloc+0x4a malloc+0xd9 arc_get_data_buf+0x1f4 arc_read+0x225 > dbuf_read+0x445 dmu_buf_hold_array_by_dnode+0x168 dmu_buf_hold_array+0x67 > dmu_read_uio+0x3f zfs_freebsd_read+0x483 VOP_READ_APV+0x6e vn_read+0xed > vn_io_fault+0x90 > > Thread 100059 acquired arc_reclaim_thr_lock before calling arc_do_user_evicts > and now it wants to take a buf header hash lock. > Thread 102410 acquired the hash lock in arc_read, then it got into arc_lowmem > because of a memory allocation problem (and M_WAIT flag) and now it wants to > take arc_reclaim_thr_lock. > > A classic deadlock. Thanks for the feedback. Do you think it may be triggered when the ARC is evicting pages because it is full, or a genuine low-memory case? The system has 32GB of RAM, of which the ARC is typically about 24G (I think).
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5225BB8C.5050802>