Date: Thu, 10 Jan 2019 16:56:52 -0500 From: Garrett Wollman <wollman@csail.mit.edu> To: freebsd-fs@freebsd.org Subject: Still having NFS issues with at-quota ZFS datasets Message-ID: <23607.49060.41947.512101@khavrinen.csail.mit.edu>
next in thread | raw e-mail | index | archive | help
I thought that this issue was resolved by r326070 (r326427 in 11-STABLE) but it seems not. When NFS clients write into a ZFS dataset which is at quota, all NFS server processing stops: existing client socket buffers fill, no new client TCP connections are accepted, and UDP RPCs are dropped. At the time this is happening, numerous NFS server threads are idle and no auto-scaling is taking place. Here's what the nfsd kernel stacks look like: # procstat -kk 1022 PID TID COMM TDNAME KSTACK 1022 101105 nfsd nfsd: service mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _cv_wait_sig+0x167 svc_run_internal+0x586 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 1022 101116 nfsd nfsd: service mi_switch+0xe6 sleepq_wait+0x2c _cv_wait+0x16e txg_wait_open+0xa5 dmu_tx_assign+0x48 zfs_freebsd_write+0x57a VOP_WRITE_APV+0x103 nfsvno_write+0x12b nfsrvd_write+0x4a5 nfsrvd_dorpc+0x621 nfssvc_program+0x557 svc_run_internal+0xe09 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 1022 101505 nfsd nfsd: service mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _cv_wait_sig+0x167 svc_run_internal+0x586 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 1022 101551 nfsd nfsd: service mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _cv_wait_sig+0x167 svc_run_internal+0x586 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 1022 101556 nfsd nfsd: service mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _cv_wait_sig+0x167 svc_run_internal+0x586 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 1022 101642 nfsd nfsd: service mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _cv_wait_sig+0x167 svc_run_internal+0x586 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 1022 101769 nfsd nfsd: master mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x18c svc_run_internal+0x3e1 svc_run+0x176 nfsrvd_nfsd+0x22b nfssvc_nfsd+0x1d6 sys_nfssvc+0xba amd64_syscall+0xa38 fast_syscall_common+0x101 1022 101882 nfsd nfsd: service mi_switch+0xe6 sleepq_wait+0x2c _cv_wait+0x16e txg_wait_open+0xa5 dmu_tx_assign+0x48 zfs_freebsd_write+0x57a VOP_WRITE_APV+0x103 nfsvno_write+0x12b nfsrvd_write+0x4a5 nfsrvd_dorpc+0x621 nfssvc_program+0x557 svc_run_internal+0xe09 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 1022 102010 nfsd nfsd: service mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _cv_wait_sig+0x167 svc_run_internal+0x586 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 1022 102058 nfsd nfsd: service mi_switch+0xe6 sleepq_wait+0x2c _cv_wait+0x16e txg_wait_open+0xa5 dmu_tx_assign+0x48 zfs_freebsd_write+0x57a VOP_WRITE_APV+0x103 nfsvno_write+0x12b nfsrvd_write+0x4a5 nfsrvd_dorpc+0x621 nfssvc_program+0x557 svc_run_internal+0xe09 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 1022 102073 nfsd nfsd: service mi_switch+0xe6 sleepq_wait+0x2c _cv_wait+0x16e txg_wait_open+0xa5 dmu_tx_assign+0x48 zfs_freebsd_write+0x57a VOP_WRITE_APV+0x103 nfsvno_write+0x12b nfsrvd_write+0x4a5 nfsrvd_dorpc+0x621 nfssvc_program+0x557 svc_run_internal+0xe09 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 1022 102080 nfsd nfsd: service mi_switch+0xe6 sleepq_wait+0x2c _cv_wait+0x16e txg_wait_open+0xa5 dmu_tx_assign+0x48 zfs_freebsd_write+0x57a VOP_WRITE_APV+0x103 nfsvno_write+0x12b nfsrvd_write+0x4a5 nfsrvd_dorpc+0x621 nfssvc_program+0x557 svc_run_internal+0xe09 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 1022 102092 nfsd nfsd: service mi_switch+0xe6 sleepq_wait+0x2c _cv_wait+0x16e txg_wait_open+0xa5 dmu_tx_assign+0x48 zfs_freebsd_write+0x57a VOP_WRITE_APV+0x103 nfsvno_write+0x12b nfsrvd_write+0x4a5 nfsrvd_dorpc+0x621 nfssvc_program+0x557 svc_run_internal+0xe09 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 1022 102353 nfsd nfsd: service mi_switch+0xe6 sleepq_wait+0x2c _cv_wait+0x16e txg_wait_open+0xa5 dmu_tx_assign+0x48 zfs_freebsd_write+0x57a VOP_WRITE_APV+0x103 nfsvno_write+0x12b nfsrvd_write+0x4a5 nfsrvd_dorpc+0x621 nfssvc_program+0x557 svc_run_internal+0xe09 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 1022 102388 nfsd nfsd: service mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _cv_wait_sig+0x167 svc_run_internal+0x586 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 1022 102439 nfsd nfsd: service mi_switch+0xe6 sleepq_wait+0x2c _cv_wait+0x16e txg_wait_open+0xa5 dmu_tx_assign+0x48 zfs_freebsd_write+0x57a VOP_WRITE_APV+0x103 nfsvno_write+0x12b nfsrvd_write+0x4a5 nfsrvd_dorpc+0x621 nfssvc_program+0x557 svc_run_internal+0xe09 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 1022 102570 nfsd nfsd: service mi_switch+0xe6 sleepq_wait+0x2c _cv_wait+0x16e txg_wait_open+0xa5 dmu_tx_assign+0x48 zfs_freebsd_write+0x57a VOP_WRITE_APV+0x103 nfsvno_write+0x12b nfsrvd_write+0x4a5 nfsrvd_dorpc+0x621 nfssvc_program+0x557 svc_run_internal+0xe09 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 1022 102606 nfsd nfsd: service mi_switch+0xe6 sleepq_wait+0x2c _cv_wait+0x16e txg_wait_open+0xa5 dmu_tx_assign+0x48 zfs_freebsd_write+0x57a VOP_WRITE_APV+0x103 nfsvno_write+0x12b nfsrvd_write+0x4a5 nfsrvd_dorpc+0x621 nfssvc_program+0x557 svc_run_internal+0xe09 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 1022 102613 nfsd nfsd: service mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _cv_wait_sig+0x167 svc_run_internal+0x586 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 1022 102683 nfsd nfsd: service mi_switch+0xe6 sleepq_wait+0x2c _cv_wait+0x16e zfs_range_lock+0xbc zfs_freebsd_write+0x30d VOP_WRITE_APV+0x103 nfsvno_write+0x12b nfsrvd_write+0x4a5 nfsrvd_dorpc+0x621 nfssvc_program+0x557 svc_run_internal+0xe09 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe You can see that most of the threads are sleeping in txg_wait_open() but one is in zfs_range_lock() and the rest are just waiting to service incoming RPCs (which are not getting dispatched). The situation immediately clears up when the quota on the dataset is raised sufficiently to permit the pending writes to complete. (Obviously this is only possible on a pool that has sufficient free space to do so, as this one did.) -GAWollman
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?23607.49060.41947.512101>