Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 10 Jan 2019 16:56:52 -0500
From:      Garrett Wollman <wollman@csail.mit.edu>
To:        freebsd-fs@freebsd.org
Subject:   Still having NFS issues with at-quota ZFS datasets
Message-ID:  <23607.49060.41947.512101@khavrinen.csail.mit.edu>

next in thread | raw e-mail | index | archive | help
I thought that this issue was resolved by r326070 (r326427 in
11-STABLE) but it seems not.

When NFS clients write into a ZFS dataset which is at quota, all NFS
server processing stops: existing client socket buffers fill, no new
client TCP connections are accepted, and UDP RPCs are dropped.  At the
time this is happening, numerous NFS server threads are idle and no
auto-scaling is taking place.  Here's what the nfsd kernel stacks look
like:

# procstat -kk 1022
  PID    TID COMM                TDNAME              KSTACK                     
 1022 101105 nfsd                nfsd: service       mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _cv_wait_sig+0x167 svc_run_internal+0x586 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 
 1022 101116 nfsd                nfsd: service       mi_switch+0xe6 sleepq_wait+0x2c _cv_wait+0x16e txg_wait_open+0xa5 dmu_tx_assign+0x48 zfs_freebsd_write+0x57a VOP_WRITE_APV+0x103 nfsvno_write+0x12b nfsrvd_write+0x4a5 nfsrvd_dorpc+0x621 nfssvc_program+0x557 svc_run_internal+0xe09 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 
 1022 101505 nfsd                nfsd: service       mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _cv_wait_sig+0x167 svc_run_internal+0x586 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 
 1022 101551 nfsd                nfsd: service       mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _cv_wait_sig+0x167 svc_run_internal+0x586 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 
 1022 101556 nfsd                nfsd: service       mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _cv_wait_sig+0x167 svc_run_internal+0x586 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 
 1022 101642 nfsd                nfsd: service       mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _cv_wait_sig+0x167 svc_run_internal+0x586 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 
 1022 101769 nfsd                nfsd: master        mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x18c svc_run_internal+0x3e1 svc_run+0x176 nfsrvd_nfsd+0x22b nfssvc_nfsd+0x1d6 sys_nfssvc+0xba amd64_syscall+0xa38 fast_syscall_common+0x101 
 1022 101882 nfsd                nfsd: service       mi_switch+0xe6 sleepq_wait+0x2c _cv_wait+0x16e txg_wait_open+0xa5 dmu_tx_assign+0x48 zfs_freebsd_write+0x57a VOP_WRITE_APV+0x103 nfsvno_write+0x12b nfsrvd_write+0x4a5 nfsrvd_dorpc+0x621 nfssvc_program+0x557 svc_run_internal+0xe09 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 
 1022 102010 nfsd                nfsd: service       mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _cv_wait_sig+0x167 svc_run_internal+0x586 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 
 1022 102058 nfsd                nfsd: service       mi_switch+0xe6 sleepq_wait+0x2c _cv_wait+0x16e txg_wait_open+0xa5 dmu_tx_assign+0x48 zfs_freebsd_write+0x57a VOP_WRITE_APV+0x103 nfsvno_write+0x12b nfsrvd_write+0x4a5 nfsrvd_dorpc+0x621 nfssvc_program+0x557 svc_run_internal+0xe09 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 
 1022 102073 nfsd                nfsd: service       mi_switch+0xe6 sleepq_wait+0x2c _cv_wait+0x16e txg_wait_open+0xa5 dmu_tx_assign+0x48 zfs_freebsd_write+0x57a VOP_WRITE_APV+0x103 nfsvno_write+0x12b nfsrvd_write+0x4a5 nfsrvd_dorpc+0x621 nfssvc_program+0x557 svc_run_internal+0xe09 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 
 1022 102080 nfsd                nfsd: service       mi_switch+0xe6 sleepq_wait+0x2c _cv_wait+0x16e txg_wait_open+0xa5 dmu_tx_assign+0x48 zfs_freebsd_write+0x57a VOP_WRITE_APV+0x103 nfsvno_write+0x12b nfsrvd_write+0x4a5 nfsrvd_dorpc+0x621 nfssvc_program+0x557 svc_run_internal+0xe09 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 
 1022 102092 nfsd                nfsd: service       mi_switch+0xe6 sleepq_wait+0x2c _cv_wait+0x16e txg_wait_open+0xa5 dmu_tx_assign+0x48 zfs_freebsd_write+0x57a VOP_WRITE_APV+0x103 nfsvno_write+0x12b nfsrvd_write+0x4a5 nfsrvd_dorpc+0x621 nfssvc_program+0x557 svc_run_internal+0xe09 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 
 1022 102353 nfsd                nfsd: service       mi_switch+0xe6 sleepq_wait+0x2c _cv_wait+0x16e txg_wait_open+0xa5 dmu_tx_assign+0x48 zfs_freebsd_write+0x57a VOP_WRITE_APV+0x103 nfsvno_write+0x12b nfsrvd_write+0x4a5 nfsrvd_dorpc+0x621 nfssvc_program+0x557 svc_run_internal+0xe09 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 
 1022 102388 nfsd                nfsd: service       mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _cv_wait_sig+0x167 svc_run_internal+0x586 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 
 1022 102439 nfsd                nfsd: service       mi_switch+0xe6 sleepq_wait+0x2c _cv_wait+0x16e txg_wait_open+0xa5 dmu_tx_assign+0x48 zfs_freebsd_write+0x57a VOP_WRITE_APV+0x103 nfsvno_write+0x12b nfsrvd_write+0x4a5 nfsrvd_dorpc+0x621 nfssvc_program+0x557 svc_run_internal+0xe09 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 
 1022 102570 nfsd                nfsd: service       mi_switch+0xe6 sleepq_wait+0x2c _cv_wait+0x16e txg_wait_open+0xa5 dmu_tx_assign+0x48 zfs_freebsd_write+0x57a VOP_WRITE_APV+0x103 nfsvno_write+0x12b nfsrvd_write+0x4a5 nfsrvd_dorpc+0x621 nfssvc_program+0x557 svc_run_internal+0xe09 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 
 1022 102606 nfsd                nfsd: service       mi_switch+0xe6 sleepq_wait+0x2c _cv_wait+0x16e txg_wait_open+0xa5 dmu_tx_assign+0x48 zfs_freebsd_write+0x57a VOP_WRITE_APV+0x103 nfsvno_write+0x12b nfsrvd_write+0x4a5 nfsrvd_dorpc+0x621 nfssvc_program+0x557 svc_run_internal+0xe09 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 
 1022 102613 nfsd                nfsd: service       mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _cv_wait_sig+0x167 svc_run_internal+0x586 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 
 1022 102683 nfsd                nfsd: service       mi_switch+0xe6 sleepq_wait+0x2c _cv_wait+0x16e zfs_range_lock+0xbc zfs_freebsd_write+0x30d VOP_WRITE_APV+0x103 nfsvno_write+0x12b nfsrvd_write+0x4a5 nfsrvd_dorpc+0x621 nfssvc_program+0x557 svc_run_internal+0xe09 svc_thread_start+0xb fork_exit+0x83 fork_trampoline+0xe 

You can see that most of the threads are sleeping in txg_wait_open()
but one is in zfs_range_lock() and the rest are just waiting to
service incoming RPCs (which are not getting dispatched).

The situation immediately clears up when the quota on the dataset is
raised sufficiently to permit the pending writes to complete.
(Obviously this is only possible on a pool that has sufficient free
space to do so, as this one did.)

-GAWollman




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?23607.49060.41947.512101>