From owner-freebsd-fs@freebsd.org Mon Oct 2 20:56:23 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 06D39E27927 for ; Mon, 2 Oct 2017 20:56:23 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citapm.icyb.net.ua (citapm.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 3410E7776C for ; Mon, 2 Oct 2017 20:56:21 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citapm.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id XAA10583; Mon, 02 Oct 2017 23:56:14 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1dz7l7-000JfZ-SK; Mon, 02 Oct 2017 23:56:13 +0300 Subject: Re: ZFS stalled after some mirror disks were lost To: Ben RUBSON , Freebsd fs References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <71d4416a-3454-df36-adae-34c0b70cd84e@multiplay.co.uk> <8A189756-028A-465E-9962-D0181FAEBB79@gmail.com> <5d3e1f0d-c618-afa4-7e52-819c9edf30c9@FreeBSD.org> <48D23270-1811-4E09-8AF2-5C0FEC2F9176@gmail.com> From: Andriy Gapon Message-ID: <9ff8ef2c-b445-dad3-d726-b84793c173ee@FreeBSD.org> Date: Mon, 2 Oct 2017 23:55:38 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <48D23270-1811-4E09-8AF2-5C0FEC2F9176@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Oct 2017 20:56:23 -0000 On 02/10/2017 22:13, Ben RUBSON wrote: >> On 02 Oct 2017, at 20:45, Andriy Gapon wrote: >> >> On 02/10/2017 21:17, Ben RUBSON wrote: >>> Unfortunately the command stalls / does not return :/ >> >> Try to take procstat -kk -a. > > Thank you Andriy for your answer. > > Here is the procstat output : > https://benrubson.github.io/zfs/procstat01.log First, it seems that there are some iscsi threads stuck on a lock like: 0 100291 kernel iscsimt mi_switch+0xd2 sleepq_wait+0x3a _sx_xlock_hard+0x592 iscsi_maintenance_thread+0x316 fork_exit+0x85 fork_trampoline+0xe or like 8580 102077 iscsictl - mi_switch+0xd2 sleepq_wait+0x3a _sx_slock_hard+0x325 iscsi_ioctl+0x7ea devfs_ioctl_f+0x13f kern_ioctl+0x2d4 sys_ioctl+0x171 amd64_syscall+0x4ce Xfast_syscall+0xfb Also, there is a thread in cam_sim_free(): 0 100986 kernel iscsimt mi_switch+0xd2 sleepq_wait+0x3a _sleep+0x2a1 cam_sim_free+0x48 iscsi_session_cleanup+0x1bd iscsi_maintenance_thread+0x388 fork_exit+0x85 fork_trampoline+0xe So, it looks like there could be a problem is the iscsi teardown path. Maybe that caused a domino effect in ZFS code. I see a lot of threads waiting either for spa_namespace_lock or a spa config lock (a highly specialized ZFS lock). But it is hard to untangle their inter-dependencies. Some of ZFS I/O threads are also affected, for example: 0 101538 kernel zio_write_issue_ mi_switch+0xd2 sleepq_wait+0x3a _cv_wait+0x194 spa_config_enter+0x9b zio_vdev_io_start+0x1c2 zio_execute+0x236 taskqueue_run_locked+0x14a taskqueue_thread_loop+0xe8 fork_exit+0x85 fork_trampoline+0xe 8716 101319 sshd - mi_switch+0xd2 sleepq_wait+0x3a _cv_wait+0x194 spa_config_enter+0x9b zio_vdev_io_start+0x1c2 zio_execute+0x236 zio_nowait+0x49 arc_read+0x8e4 dbuf_read+0x6c2 dmu_buf_hold_array_by_dnode+0x1d3 dmu_read_uio_dnode+0x41 dmu_read_uio_dbuf+0x3b zfs_freebsd_read+0x5fc VOP_READ_APV+0x89 vn_read+0x157 vn_io_fault1+0x1c2 vn_io_fault+0x197 dofileread+0x98 71181 101141 encfs - mi_switch+0xd2 sleepq_wait+0x3a _cv_wait+0x194 spa_config_enter+0x9b zio_vdev_io_start+0x1c2 zio_execute+0x236 zio_nowait+0x49 arc_read+0x8e4 dbuf_read+0x6c2 dmu_buf_hold+0x3d zap_lockdir+0x43 zap_cursor_retrieve+0x171 zfs_freebsd_readdir+0x3f3 VOP_READDIR_APV+0x8f kern_getdirentries+0x21b sys_getdirentries+0x28 amd64_syscall+0x4ce Xfast_syscall+0xfb 71181 101190 encfs - mi_switch+0xd2 sleepq_wait+0x3a _cv_wait+0x194 spa_config_enter+0x9b zio_vdev_io_start+0x1c2 zio_execute+0x236 zio_nowait+0x49 arc_read+0x8e4 dbuf_prefetch_indirect_done+0xcc arc_read+0x425 dbuf_prefetch+0x4f7 dmu_zfetch+0x418 dmu_buf_hold_array_by_dnode+0x34d dmu_read_uio_dnode+0x41 dmu_read_uio_dbuf+0x3b zfs_freebsd_read+0x5fc VOP_READ_APV+0x89 vn_read+0x157 Note that the first of these threads executes a write zio. -- Andriy Gapon