Date: Thu, 18 Dec 2014 11:28:42 +0000 From: "=?utf-8?B?TG/Dr2MgQmxvdA==?=" <loic.blot@unix-experience.fr> To: "Rick Macklem" <rmacklem@uoguelph.ca> Cc: freebsd-fs@freebsd.org Subject: Re: ZFS vnode lock deadlock in zfs_fhtovp was: High Kernel Load with nfsv4 Message-ID: <0eaadfe31ac4b8bbdeaf0baff696dada@mail.unix-experience.fr> In-Reply-To: <1215617347.12668398.1418653073454.JavaMail.root@uoguelph.ca> References: <1215617347.12668398.1418653073454.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi rick,=0Ai tried to start a LXC container on Debian Squeeze from my fre=
ebsd ZFS+NFSv4 server and i also have a deadlock on nfsd (vfs.lookup_shar=
ed=3D0). Deadlock procs each time i launch a squeeze container, it seems =
(3 tries, 3 fails).=0A=0A 921 - D 0:00.02 nfsd: server (nfsd)=0A=
=0AHere is the procstat -kk=0A=0A PID TID COMM TDNAME =
KSTACK =0A 921 100538 nfsd nfsd=
: master mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args=
+0xc9e vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 nfsvno_advlock+0=
x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad nfsrvd_locku+0x283 nfsrvd_d=
orpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77 svc_run+0x1de nfsr=
vd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c =0A 921 100572 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_=
wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0=
xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100573 nfsd nf=
sd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0=
xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_ex=
it+0x9a fork_trampoline+0xe =0A 921 100574 nfsd nfsd: servic=
e mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wai=
t_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fo=
rk_trampoline+0xe =0A 921 100575 nfsd nfsd: service mi_sw=
itch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16=
a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampol=
ine+0xe =0A 921 100576 nfsd nfsd: service mi_switch+0xe1 =
sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_=
internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =
=0A 921 100577 nfsd nfsd: service mi_switch+0xe1 sleepq_c=
atch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal=
+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 1=
00578 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signa=
ls+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc=
_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100579 nfsd=
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sl=
eepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_st=
art+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100580 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_=
sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fo=
rk_exit+0x9a fork_trampoline+0xe =0A 921 100581 nfsd nfsd: s=
ervice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _c=
v_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x=
9a fork_trampoline+0xe =0A 921 100582 nfsd nfsd: service =
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig=
+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_tr=
ampoline+0xe =0A 921 100583 nfsd nfsd: service mi_switch+=
0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0=
xe =0A 921 100584 nfsd nfsd: service mi_switch+0xe1 sleep=
q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_inter=
nal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 92=
1 100585 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_si=
gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e =
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100586 n=
fsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab=
sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread=
_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100587 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wa=
it_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb=
fork_exit+0x9a fork_trampoline+0xe =0A 921 100588 nfsd nfsd=
: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf=
_cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit=
+0x9a fork_trampoline+0xe =0A 921 100589 nfsd nfsd: service =
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_=
sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork=
_trampoline+0xe =0A 921 100590 nfsd nfsd: service mi_swit=
ch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a =
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampolin=
e+0xe =0A 921 100591 nfsd nfsd: service mi_switch+0xe1 sl=
eepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_in=
ternal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A =
921 100592 nfsd nfsd: service mi_switch+0xe1 sleepq_catch=
_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x8=
7e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 10059=
3 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thr=
ead_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100594 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq=
_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+=
0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100595 nfsd n=
fsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_e=
xit+0x9a fork_trampoline+0xe =0A 921 100596 nfsd nfsd: servi=
ce mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wa=
it_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a f=
ork_trampoline+0xe =0A 921 100597 nfsd nfsd: service mi_s=
witch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1=
6a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampo=
line+0xe =0A 921 100598 nfsd nfsd: service mi_switch+0xe1=
sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run=
_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =
=0A 921 100599 nfsd nfsd: service mi_switch+0xe1 sleepq_c=
atch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal=
+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 1=
00600 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signa=
ls+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc=
_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100601 nfsd=
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sl=
eepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_st=
art+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100602 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_=
sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fo=
rk_exit+0x9a fork_trampoline+0xe =0A 921 100603 nfsd nfsd: s=
ervice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _c=
v_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x=
9a fork_trampoline+0xe =0A 921 100604 nfsd nfsd: service =
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig=
+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_tr=
ampoline+0xe =0A 921 100605 nfsd nfsd: service mi_switch+=
0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0=
xe =0A 921 100606 nfsd nfsd: service mi_switch+0xe1 sleep=
q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_inter=
nal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 92=
1 100607 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_si=
gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e =
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100608 n=
fsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab=
sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread=
_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100609 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wa=
it_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb=
fork_exit+0x9a fork_trampoline+0xe =0A 921 100610 nfsd nfsd=
: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf=
_cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit=
+0x9a fork_trampoline+0xe =0A 921 100611 nfsd nfsd: service =
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_=
sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork=
_trampoline+0xe =0A 921 100612 nfsd nfsd: service mi_swit=
ch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a =
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampolin=
e+0xe =0A 921 100613 nfsd nfsd: service mi_switch+0xe1 sl=
eepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_in=
ternal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A =
921 100614 nfsd nfsd: service mi_switch+0xe1 sleepq_catch=
_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x8=
7e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 10061=
5 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thr=
ead_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100616 nfsd =
nfsd: service mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 nf=
smsleep+0x66 nfsv4_lock+0x9b nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f=
nfsrvd_lock+0x5b1 nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_intern=
al+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921=
100617 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_sig=
nals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100618 nf=
sd nfsd: service mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x=
287 nfsmsleep+0x66 nfsv4_lock+0x9b nfsrvd_dorpc+0x316 nfssvc_program+0x55=
4 svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampol=
ine+0xe =0A 921 100619 nfsd nfsd: service mi_switch+0xe1 =
sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_=
internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =
=0A 921 100620 nfsd nfsd: service mi_switch+0xe1 sleepq_c=
atch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal=
+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 1=
00621 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signa=
ls+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc=
_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100622 nfsd=
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sl=
eepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_st=
art+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100623 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_=
sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fo=
rk_exit+0x9a fork_trampoline+0xe =0A 921 100624 nfsd nfsd: s=
ervice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _c=
v_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x=
9a fork_trampoline+0xe =0A 921 100625 nfsd nfsd: service =
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig=
+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_tr=
ampoline+0xe =0A 921 100626 nfsd nfsd: service mi_switch+=
0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0=
xe =0A 921 100627 nfsd nfsd: service mi_switch+0xe1 sleep=
q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_inter=
nal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 92=
1 100628 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_si=
gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e =
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100629 n=
fsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab=
sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread=
_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100630 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wa=
it_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb=
fork_exit+0x9a fork_trampoline+0xe =0A 921 100631 nfsd nfsd=
: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf=
_cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit=
+0x9a fork_trampoline+0xe =0A 921 100632 nfsd nfsd: service =
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_=
sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork=
_trampoline+0xe =0A 921 100633 nfsd nfsd: service mi_swit=
ch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a =
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampolin=
e+0xe =0A 921 100634 nfsd nfsd: service mi_switch+0xe1 sl=
eepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_in=
ternal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A =
921 100635 nfsd nfsd: service mi_switch+0xe1 sleepq_catch=
_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x8=
7e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 10063=
6 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thr=
ead_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100637 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq=
_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+=
0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100638 nfsd n=
fsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_e=
xit+0x9a fork_trampoline+0xe =0A 921 100639 nfsd nfsd: servi=
ce mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wa=
it_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a f=
ork_trampoline+0xe =0A 921 100640 nfsd nfsd: service mi_s=
witch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1=
6a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampo=
line+0xe =0A 921 100641 nfsd nfsd: service mi_switch+0xe1=
sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run=
_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =
=0A 921 100642 nfsd nfsd: service mi_switch+0xe1 sleepq_c=
atch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal=
+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 1=
00643 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signa=
ls+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc=
_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100644 nfsd=
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sl=
eepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_st=
art+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100645 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_=
sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fo=
rk_exit+0x9a fork_trampoline+0xe =0A 921 100646 nfsd nfsd: s=
ervice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _c=
v_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb=20fork_exit+=
0x9a fork_trampoline+0xe =0A 921 100647 nfsd nfsd: service =
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_s=
ig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_=
trampoline+0xe =0A 921 100648 nfsd nfsd: service mi_switc=
h+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a s=
vc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline=
+0xe =0A 921 100649 nfsd nfsd: service mi_switch+0xe1 sle=
epq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_int=
ernal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A =
921 100650 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_=
signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87=
e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100651=
nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0x=
ab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thre=
ad_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100652 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_=
wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0=
xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100653 nfsd nf=
sd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0=
xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_ex=
it+0x9a fork_trampoline+0xe =0A 921 100654 nfsd nfsd: servic=
e mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wai=
t_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fo=
rk_trampoline+0xe =0A 921 100655 nfsd nfsd: service mi_sw=
itch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16=
a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampol=
ine+0xe =0A 921 100656 nfsd nfsd: service mi_switch+0xe1 =
sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_=
internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =
=0A 921 100657 nfsd nfsd: service mi_switch+0xe1 sleepq_c=
atch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal=
+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 1=
00658 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signa=
ls+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc=
_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100659 nfsd=
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sl=
eepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_st=
art+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100660 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_=
sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fo=
rk_exit+0x9a fork_trampoline+0xe =0A 921 100661 nfsd nfsd: s=
ervice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _c=
v_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x=
9a fork_trampoline+0xe =0A 921 100662 nfsd nfsd: service =
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig=
+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_tr=
ampoline+0xe =0A 921 100663 nfsd nfsd: service mi_switch+=
0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0=
xe =0A 921 100664 nfsd nfsd: service mi_switch+0xe1 sleep=
q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_inter=
nal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 92=
1 100665 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_si=
gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e =
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 921 100666 n=
fsd nfsd: service mi_switch+0xe1 sleepq_wait+0x3a _sleep+0=
x287 nfsmsleep+0x66 nfsv4_lock+0x9b nfsrv_setclient+0xbd nfsrvd_setclient=
id+0x3c8 nfsrvd_dorpc+0xc76 nfssvc_program+0x554 svc_run_internal+0xc77 s=
vc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A=0A=0ARegards,=
=0A=0ALo=C3=AFc Blot,=0AUNIX Systems, Network and Security Engineer=0Ahtt=
p://www.unix-experience.fr=0A=0A15 d=C3=A9cembre 2014 15:18 "Rick Macklem=
" <rmacklem@uoguelph.ca> a =C3=A9crit: =0A> Loic Blot wrote:=0A> =0A>> Fo=
r more informations, here is procstat -kk on nfsd, if you need more=0A>> =
hot datas, tell me.=0A>> =0A>> Regards, PID TID COMM TDNA=
ME KSTACK=0A>> 918 100529 nfsd nfsd: master mi_=
switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>=
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> =
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_progra=
m+0x554 svc_run_internal+0xc77 svc_run+0x1de=0A>> nfsrvd_nfsd+0x1ca nfssv=
c_nfsd+0x107 sys_nfssvc+0x9c=0A>> amd64_syscall+0x351=0A> =0A> Well, most=
of the threads are stuck like this one, waiting for a vnode=0A> lock in =
ZFS. All of them appear to be in zfs_fhtovp().=0A> I`m not a ZFS guy, so =
I can`t help much. I`ll try changing the subject line=0A> to include ZFS =
vnode lock, so maybe the ZFS guys will take a look.=0A> =0A> The only thi=
ng I`ve seen suggested is trying:=0A> sysctl vfs.lookup_shared=3D0=0A> to=
disable shared vop_lookup()s. Apparently zfs_lookup() doesn`t=0A> obey t=
he vnode locking rules for lookup and rename, according to=0A> the postin=
g I saw.=0A> =0A> I`ve added a couple of comments about the other threads=
below, but=0A> they are all either waiting for an RPC request or waiting=
for the=0A> threads stuck on the ZFS vnode lock to complete.=0A> =0A> ri=
ck=0A> =0A>> 918 100564 nfsd nfsd: service mi_switch+0xe1=
=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=0A=
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>> fork_t=
rampoline+0xe=0A> =0A> Fyi, this thread is just waiting for an RPC to arr=
ive. (Normal)=0A> =0A>> 918 100565 nfsd nfsd: service mi_s=
witch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si=
g+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>> fork_trampoline+0xe=0A>> 918 100566 nfsd nfsd: service =
mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_=
wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit=
+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100567 nfsd nfsd: ser=
vice mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=
_cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb fork=
_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100568 nfsd nfsd=
: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wait_si=
g+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_start+0xb=
fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100569 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab sleepq_wa=
it_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread_star=
t+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100570 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_catch_signals+0xab slee=
pq_wait_sig+0xf _cv_wait_sig+0x16a=0A>> svc_run_internal+0x87e svc_thread=
_start+0xb fork_exit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100571 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a _sleep+0=
x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>> nfsrvd_dorpc+0x316 nfssvc_progra=
m+0x554 svc_run_internal+0xc77=0A>> svc_thread_start+0xb fork_exit+0x9a f=
ork_trampoline+0xe=0A>> 918 100572 nfsd nfsd: service mi_s=
witch+0xe1=0A>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0=
x9b=0A>> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8 nfsrvd_dorpc+0xc76=
=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=0A=
>> fork_exit+0x9a fork_trampoline+0xe=0A> =0A> This one (and a few others=
) are waiting for the nfsv4_lock. This happens=0A> because other threads =
are stuck with RPCs in progress. (ie. The ones=0A> waiting on the vnode l=
ock in zfs_fhtovp().)=0A> For these, the RPC needs to lock out other thre=
ads to do the operation,=0A> so it waits for the nfsv4_lock() which can e=
xclusively lock the=20NFSv4=0A> data structures once all other nfsd threa=
ds complete their RPCs in=0A> progress.=0A> =0A>> 918 100573 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a _sleep+0x287 n=
fsmsleep+0x66 nfsv4_lock+0x9b=0A>> nfsrvd_dorpc+0x316 nfssvc_program+0x55=
4 svc_run_internal+0xc77=0A>> svc_thread_start+0xb fork_exit+0x9a fork_tr=
ampoline+0xe=0A> =0A> Same as above.=0A> =0A>> 918 100574 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __l=
ockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 =
zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x=
917=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100575 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100576 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100577 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100578 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100579 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100580 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100581 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100582 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100583 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100584 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100585 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100586 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100587 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100588 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100589 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100590 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100591 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100592 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100593 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100594 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100595 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100596 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a=20sleeplk+0x15d __=
lockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=
zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0=
x917=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0x=
b=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100597 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __l=
ockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 =
zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x=
917=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100598 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100599 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100600 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100601 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100602 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100603 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100604 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100605 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100606 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100607 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 z=
fs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9=
17=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
=0A>> fork_exit+0x9a fork_trampoline+0xe=0A> =0A> Lots more waiting for t=
he ZFS vnode lock in zfs_fhtovp().=0A> =0A>> 918 100608 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a _sleep+0x287 nfsmsl=
eep+0x66 nfsv4_lock+0x9b=0A>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21=
f nfsrvd_lock+0x5b1=0A>> nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_=
internal+0xc77=0A>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0=
xe=0A>> 918 100609 nfsd nfsd: service mi_switch+0xe1=0A>> =
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c=
VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7=
c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_i=
nternal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0x=
e=0A>> 918 100610 nfsd nfsd: service mi_switch+0xe1=0A>> s=
leepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e=0A>> vop_stdlock+0x3c =
VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>> nfsvno_advlock+0x119 nfsrv_dolocal+=
0x84 nfsrv_lockctrl+0x14ad=0A>> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfs=
svc_program+0x554=0A>> svc_run_internal+0xc77 svc_thread_start+0xb fork_e=
xit+0x9a=0A>> fork_trampoline+0xe=0A>> 918 100611 nfsd nfsd: =
service mi_switch+0xe1=0A>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x=
66 nfsv4_lock+0x9b=0A>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_i=
nternal+0xc77=0A>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0x=
e=0A>> 918 100612 nfsd nfsd: service mi_switch+0xe1=0A>> s=
leepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>> nfsrvd_d=
orpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77=0A>> svc_thread_st=
art+0xb fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100613 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a _sleep+0x287 nfs=
msleep+0x66 nfsv4_lock+0x9b=0A>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 =
svc_run_internal+0xc77=0A>> svc_thread_start+0xb fork_exit+0x9a fork_tram=
poline+0xe=0A>> 918 100614 nfsd nfsd: service mi_switch+0x=
e1=0A>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>=
nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77=0A>> svc_=
thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100615 nfsd =
nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a _sleep+=
0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>> nfsrvd_dorpc+0x316 nfssvc_progr=
am+0x554 svc_run_internal+0xc77=0A>> svc_thread_start+0xb fork_exit+0x9a =
fork_trampoline+0xe=0A>> 918 100616 nfsd nfsd: service mi_=
switch+0xe1=0A>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+=
0x9b=0A>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77=
=0A>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>> 918 10=
0617 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_wait+0x=
3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>> nfsrvd_dorpc+0x316 nf=
ssvc_program+0x554 svc_run_internal+0xc77=0A>> svc_thread_start+0xb fork_=
exit+0x9a fork_trampoline+0xe=0A>> 918 100618 nfsd nfsd: serv=
ice mi_switch+0xe1=0A>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 n=
fsv4_lock+0x9b=0A>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_inter=
nal+0xc77=0A>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A=
>> 918 100619 nfsd nfsd: service mi_switch+0xe1=0A>> sleep=
q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_=
LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfs=
d_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_intern=
al+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>=
> 918 100620 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq=
_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c VOP_L=
OCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd=
_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_interna=
l+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=0A>>=
918 100621 nfsd nfsd: service mi_switch+0xe1=0A>> sleepq_=
wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>> nfsrvd_dorpc+0=
x316 nfssvc_program+0x554 svc_run_internal+0xc77=0A>> svc_thread_start+0x=
b fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100622 nfsd nfs=
d: service mi_switch+0xe1=0A>> sleepq_wait+0x3a sleeplk+0x15d __lockmg=
r_args+0x902=0A>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_f=
htovp+0x38d=0A>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=
=0A>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=0A=
>> fork_exit+0x9a fork_trampoline+0xe=0A>> 918 100623 nfsd nf=
sd: service mi_switch+0xe1=0A>> sleepq_wait+0x3a _sleep+0x287 nfsmslee=
p+0x66 nfsv4_lock+0x9b=0A>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_r=
un_internal+0xc77=0A>> svc_thread_start+0xb fork_exit+0x9a fork_trampolin=
e+0xe=0A>> 918 100624 nfsd nfsd: service mi_switch+0xe1=0A=
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0=
x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+=
0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_ru=
n_internal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline=
+0xe=0A>> 918 100625 nfsd nfsd: service mi_switch+0xe1=0A>=
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x=
3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0=
x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run=
_internal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+=
0xe=0A>> 918 100626 nfsd nfsd: service mi_switch+0xe1=0A>>=
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3=
c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x=
7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_=
internal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0=
xe=0A>> 918 100627 nfsd nfsd: service mi_switch+0xe1=0A>> =
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c=
VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7=
c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_i=
nternal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0x=
e=0A>> 918 100628 nfsd nfsd: service mi_switch+0xe1=0A>> s=
leepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c =
VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c=
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_in=
ternal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100629 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100630 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100631 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100632 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100633 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100634 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100635 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100636 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100637 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100638 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100639 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100640 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100641 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100642 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100643 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100644 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100645 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab=20_vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7=
c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_i=
nternal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0x=
e=0A>> 918 100646 nfsd nfsd: service mi_switch+0xe1=0A>> s=
leepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c =
VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c=
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_in=
ternal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100647 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100648 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100649 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100650 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100651 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100652 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100653 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100654 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100655 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100656 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100657 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> 918 100658 nfsd nfsd: service mi_switch+0xe1=0A>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=0A>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb=0A>> fork_exit+0x9a fork_trampoline+0xe=
=0A>> =0A>> Lo=C3=AFc Blot,=0A>> UNIX Systems, Network and Security Engin=
eer=0A>> http://www.unix-experience.fr=0A>>; =0A>> 15 d=C3=A9cembre 2014 1=
3:29 "Lo=C3=AFc Blot" <loic.blot@unix-experience.fr> a=0A>> =C3=A9crit:=
=0A>>> Hmmm...=0A>>> now i'm experiencing a deadlock.=0A>>> =0A>>> 0 918 =
915 0 21 0 12352 3372 zfs D - 1:48.64 nfsd: server (nfsd)=0A>>> =0A>>> th=
e only issue was to reboot the server, but after rebooting=0A>>> deadlock=
arrives a second time when i=0A>>> start my jails over NFS.=0A>>> =0A>>>=
Regards,=0A>>> =0A>>> Lo=C3=AFc Blot,=0A>>> UNIX Systems, Network and Se=
curity Engineer=0A>>> http://www.unix-experience.fr=0A>>>; =0A>>> 15 d=C3=
=A9cembre 2014 10:07 "Lo=C3=AFc Blot" <loic.blot@unix-experience.fr> a=0A=
>>> =C3=A9crit:=0A>>> =0A>>> Hi Rick,=0A>>> after talking with my N+1, NF=
Sv4 is required on our infrastructure.=0A>>> I tried to upgrade NFSv4+ZFS=
=0A>>> server from 9.3 to 10.1, i hope this will resolve some issues...=
=0A>>> =0A>>> Regards,=0A>>> =0A>>> Lo=C3=AFc Blot,=0A>>> UNIX Systems, N=
etwork and Security Engineer=0A>>> http://www.unix-experience.fr=0A>>>; =
=0A>>> 10 d=C3=A9cembre 2014 15:36 "Lo=C3=AFc Blot" <loic.blot@unix-exper=
ience.fr> a=0A>>> =C3=A9crit:=0A>>> =0A>>> Hi Rick,=0A>>> thanks for your=
suggestion.=0A>>> For my locking bug, rpc.lockd is stucked in rpcrecv st=
ate on the=0A>>> server. kill -9 doesn't affect the=0A>>> process, it's b=
locked.... (State: Ds)=0A>>> =0A>>> for the performances=0A>>> =0A>>> NFS=
v3: 60Mbps=0A>>> NFSv4: 45Mbps=0A>>> Regards,=0A>>> =0A>>> Lo=C3=AFc Blot=
,=0A>>> UNIX Systems, Network and Security Engineer=0A>>> http://www.unix=
-experience.fr=0A>>> =0A>>> 10 d=C3=A9cembre 2014 13:56 "Rick Macklem" <r=
macklem@uoguelph.ca> a=0A>>> =C3=A9crit:=0A>>> =0A>>>> Loic Blot wrote:=
=0A>>>> =0A>>>>> Hi Rick,=0A>>>>> I'm trying NFSv3.=0A>>>>> Some jails ar=
e starting very well but now i have an issue with=0A>>>>> lockd=0A>>>>> a=
fter some minutes:=0A>>>>> =0A>>>>> nfs server 10.10.X.8:/jails: lockd no=
t responding=0A>>>>> nfs server 10.10.X.8:/jails lockd is alive again=0A>=
>>>> =0A>>>>> I look at mbuf, but i seems there is no problem.=0A>>>> =0A=
>>>> Well, if you need locks to be visible across multiple clients,=0A>>>=
> then=0A>>>> I'm afraid you are stuck with using NFSv4 and the performan=
ce you=0A>>>> get=0A>>>> from it. (There is no way to do file handle affi=
nity for NFSv4=0A>>>> because=0A>>>> the read and write ops are buried in=
the compound RPC and not=0A>>>> easily=0A>>>> recognized.)=0A>>>> =0A>>>=
> If the locks don't need to be visible across multiple clients, I'd=0A>>=
>> suggest trying the "nolockd" option with nfsv3.=0A>>>> =0A>>>>> Here i=
s my rc.conf on server:=0A>>>>> =0A>>>>> nfs_server_enable=3D"YES"=0A>>>>=
> nfsv4_server_enable=3D"YES"=0A>>>>> nfsuserd_enable=3D"YES"=0A>>>>> nfs=
d_server_flags=3D"-u -t -n 256"=0A>>>>> mountd_enable=3D"YES"=0A>>>>> mou=
ntd_flags=3D"-r"=0A>>>>> nfsuserd_flags=3D"-usertimeout 0 -force 20"=0A>>=
>>> rpcbind_enable=3D"YES"=0A>>>>> rpc_lockd_enable=3D"YES"=0A>>>>> rpc_s=
tatd_enable=3D"YES"=0A>>>>> =0A>>>>> Here is the client:=0A>>>>> =0A>>>>>=
nfsuserd_enable=3D"YES"=0A>>>>> nfsuserd_flags=3D"-usertimeout 0 -force =
20"=0A>>>>> nfscbd_enable=3D"YES"=0A>>>>> rpc_lockd_enable=3D"YES"=0A>>>>=
> rpc_statd_enable=3D"YES"=0A>>>>> =0A>>>>> Have you got an idea ?=0A>>>>=
> =0A>>>>> Regards,=0A>>>>> =0A>>>>> Lo=C3=AFc Blot,=0A>>>>> UNIX Systems=
, Network and Security Engineer=0A>>>>> http://www.unix-experience.fr=0A>=
>>>> =0A>>>>> 9 d=C3=A9cembre 2014 04:31 "Rick Macklem" <rmacklem@uoguelp=
h.ca> a=0A>>>>> =C3=A9crit:=0A>>>>>> Loic Blot wrote:=0A>>>>>> =0A>>>>>>>=
Hi rick,=0A>>>>>>> =0A>>>>>>> I waited 3 hours (no lag at jail launch) a=
nd now I do: sysrc=0A>>>>>>> memcached_flags=3D"-v -m 512"=0A>>>>>>> Comm=
and was very very slow...=0A>>>>>>> =0A>>>>>>> Here is a dd=20over NFS:=
=0A>>>>>>> =0A>>>>>>> 601062912 bytes transferred in 21.060679 secs (2853=
9579=0A>>>>>>> bytes/sec)=0A>>>>>> =0A>>>>>> Can you try the same read us=
ing an NFSv3 mount?=0A>>>>>> (If it runs much faster, you have probably b=
een bitten by the=0A>>>>>> ZFS=0A>>>>>> "sequential vs random" read heuri=
stic which I've been told=0A>>>>>> things=0A>>>>>> NFS is doing "random" =
reads without file handle affinity. File=0A>>>>>> handle affinity is very=
hard to do for NFSv4, so it isn't done.)=0A>>>> =0A>>>> I was actually s=
uggesting that you try the "dd" over nfsv3 to see=0A>>>> how=0A>>>> the p=
erformance compared with nfsv4. If you do that, please post=0A>>>> the=0A=
>>>> comparable results.=0A>>>> =0A>>>> Someday I would like to try and g=
et ZFS's sequential vs random=0A>>>> read=0A>>>> heuristic modified and a=
ny info on what difference in performance=0A>>>> that=0A>>>> might make f=
or NFS would be useful.=0A>>>> =0A>>>> rick=0A>>>> =0A>>>>>> rick=0A>>>>>=
> =0A>>>>>>> This is quite slow...=0A>>>>>>> =0A>>>>>>> You can found som=
e nfsstat below (command isn't finished yet)=0A>>>>>>> =0A>>>>>>> nfsstat=
-c -w 1=0A>>>>>>> =0A>>>>>>> GtAttr Lookup Rdlink Read Write Rename Acce=
ss Rddir=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 4 0 0 0 0 0 16 0=0A>>>>>>> 2=
0 0 0 0 0 17 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>=
>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 4 0 0 0 0 4 0=
=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 =
0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 4 0 0 0 0 0 3 0=0A>>>>>>> 0 0 0=
0 0 0 3 0=0A>>>>>>> 37 10 0 8 0 0 14 1=0A>>>>>>> 18 16 0 4 1 2 4 0=0A>>>=
>>>> 78 91 0 82 6 12 30 0=0A>>>>>>> 19 18 0 2 2 4 2 0=0A>>>>>>> 0 0 0 0 2=
0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> GtAttr Lookup Rdlink Read Writ=
e Rename Access Rddir=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=
=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 1 0 0 0 0 1 0=0A>>>>>>> 4 6 0 0 6 =
0 3 0=0A>>>>>>> 2 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 1 0 0=
0 0 0 0 0=0A>>>>>>> 0 0 0 0 1 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> =
0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>=
>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=
=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 6 108 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 =
0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> GtAttr Lookup Rdlink Read Wri=
te Rename Access Rddir=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 =
0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0=
0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 98 5=
4 0 86 11 0 25 0=0A>>>>>>> 36 24 0 39 25 0 10 1=0A>>>>>>> 67 8 0 63 63 0 =
41 0=0A>>>>>>> 34 0 0 35 34 0 0 0=0A>>>>>>> 75 0 0 75 77 0 0 0=0A>>>>>>> =
34 0 0 35 35 0 0 0=0A>>>>>>> 75 0 0 74 76 0 0 0=0A>>>>>>> 33 0 0 34 33 0 =
0 0=0A>>>>>>> 0 0 0 0 5 0 0 0=0A>>>>>>> 0 0 0 0 0 0 6 0=0A>>>>>>> 11 0 0 =
0 0 0 11 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 17 0 0 0 0 1 0=0A>>>>>>>=
GtAttr Lookup Rdlink Read Write Rename Access Rddir=0A>>>>>>> 4 5 0 0 0 =
0 12 0=0A>>>>>>> 2 0 0 0 0 0 26 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0=
0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>=
> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 4 0 0 0 0 4 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>=
>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 4 0 0 0 0 0 2 =
0=0A>>>>>>> 2 0 0 0 0 0 24 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 =
0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0=
0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>=
> GtAttr Lookup Rdlink Read Write Rename Access Rddir=0A>>>>>>> 0 0 0 0 0=
0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 4 0 0 0 0 0 7 0=0A>>>>>>> 2 1 =
0 0 0 0 1 0=0A>>>>>>> 0 0 0 0 2 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>=
0 0 0 0 6 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>=
>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=
=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 4 6 0 0 0 0 3 0=0A>>>>>>> 0 0 0 0 0 =
0 0 0=0A>>>>>>> 2 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0=
0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> =
GtAttr Lookup Rdlink Read Write Rename Access Rddir=0A>>>>>>> 0 0 0 0 0 0=
0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 =
0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 4 71 0 0 0 0 0 0=0A>>>>>>> =
0 1 0 0 0 0 0 0=0A>>>>>>> 2 36 0 0 0 0 1 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>=
>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=
=0A>>>>>>> 1 0 0 0 0 0 1 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 =
0 0 0=0A>>>>>>> 79 6 0 79 79 0 2 0=0A>>>>>>> 25 0 0 25 26 0 6 0=0A>>>>>>>=
43 18 0 39 46 0 23 0=0A>>>>>>> 36 0 0 36 36 0 31 0=0A>>>>>>> 68 1 0 66 6=
8 0 0 0=0A>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir=0A>=
>>>>>> 36 0 0 36 36 0 0 0=0A>>>>>>> 48 0 0 48 49 0 0 0=0A>>>>>>> 20 0 0 2=
0 20 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 3 14 0 1 0 0 11 0=0A>>>>>>=
> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 0 4 0 0 0 0 4 0=0A>=
>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>> 4 22 0 0 0 0 16 0=0A>>>>>>> 2 0 0 0 0 0 =
23 0=0A>>>>>>> =0A>>>>>>> Regards,=0A>>>>>>> =0A>>>>>>> Lo=C3=AFc Blot,=
=0A>>>>>>> UNIX Systems, Network and Security Engineer=0A>>>>>>> http://w=
ww.unix-experience.fr=0A>>>>>>> =0A>>>>>>> 8 d=C3=A9cembre 2014 09:36 "Lo=
=C3=AFc Blot"=0A>>>>>>> <loic.blot@unix-experience.fr> a=0A>>>>>>> =C3=A9=
crit:=0A>>>>>>>> Hi Rick,=0A>>>>>>>> I stopped the jails this week-end an=
d started it this morning,=0A>>>>>>>> i'll=0A>>>>>>>> give you some stats=
this week.=0A>>>>>>>> =0A>>>>>>>> Here is my nfsstat -m output (with you=
r rsize/wsize tweaks)=0A>>> =0A>>> =0A>> =0A> nfsv4,tcp,resvport,hard,cto=
,sec=3Dsys,acdirmin=3D3,acdirmax=3D60,acregmin=3D5,acregmax=3D60,nametime=
o=3D60,negna=0A>>> =0A>>>>>>>> =0A>>> =0A>>> =0A>> =0A> etimeo=3D60,rsize=
=3D32768,wsize=3D32768,readdirsize=3D32768,readahead=3D1,wcommitsize=3D77=
3136,timeout=3D120,retra=0A>>> =0A>>> s=3D2147483647=0A>>> =0A>>> On serv=
er side my disks are on a raid controller which show a=0A>>> 512b=0A>>> v=
olume and write performances=0A>>> are very honest (dd if=3D/dev/zero of=
=3D/jails/test.dd bs=3D4096=0A>>> count=3D100000000 =3D> 450MBps)=0A>>> =
=0A>>> Regards,=0A>>> =0A>>> Lo=C3=AFc Blot,=0A>>> UNIX Systems, Network =
and Security Engineer=0A>>> http://www.unix-experience.fr=0A>>>; =0A>>> 5 =
d=C3=A9cembre 2014 15:14 "Rick Macklem" <rmacklem@uoguelph.ca> a=0A>>> =
=C3=A9crit:=0A>>> =0A>>>> Loic Blot wrote:=0A>>>> =0A>>>>> Hi,=0A>>>>> i'=
m trying to create a virtualisation environment based on=0A>>>>> jails.=
=0A>>>>> Those jails are stored under a big ZFS pool on a FreeBSD 9.3=0A>=
>>>> which=0A>>>>> export a NFSv4 volume. This NFSv4 volume was mounted o=
n a big=0A>>>>> hypervisor (2 Xeon E5v3 + 128GB memory and 8 ports (but o=
nly 1=0A>>>>> was=0A>>>>> used at this time).=0A>>>>> =0A>>>>> The proble=
m is simple, my hypervisors runs 6 jails (used 1% cpu=0A>>>>> and=0A>>>>>=
10GB RAM approximatively and less than 1MB bandwidth) and works=0A>>>>> =
fine at start but the system slows down and after 2-3 days=0A>>>>> become=
=0A>>>>> unusable. When i look at top command i see 80-100% on system=0A>=
>>>> and=0A>>>>> commands are very very slow. Many process are tagged wit=
h=0A>>>>> nfs_cl*.=0A>>>> =0A>>>> To be honest, I would expect the slowne=
ss to be because of slow=0A>>>> response=0A>>>> from the NFSv4 server, bu=
t if you do:=0A>>>> # ps axHl=0A>>>> on a client when it is slow and post=
that, it would give us some=0A>>>> more=0A>>>> information on where the =
client side processes are sitting.=0A>>>> If you also do something like:=
=0A>>>> # nfsstat -c -w 1=0A>>>> and let it run for a while, that should =
show you how many RPCs=0A>>>> are=0A>>>> being done and which ones.=0A>>>=
> =0A>>>> # nfsstat -m=0A>>>> will show you what your mount is actually u=
sing.=0A>>>> The only mount option I can suggest trying is=0A>>>> "rsize=
=3D32768,wsize=3D32768",=0A>>>> since some network environments have diff=
iculties with 64K.=0A>>>> =0A>>>> There are a few things you can try on t=
he NFSv4 server side, if=0A>>>> it=0A>>>> appears=0A>>>> that the clients=
are generating a large RPC load.=0A>>>> - disabling the DRC cache for TC=
P by setting vfs.nfsd.cachetcp=3D0=0A>>>> - If the server is seeing a lar=
ge write RPC load, then=0A>>>> "sync=3Ddisabled"=0A>>>> might help, altho=
ugh it does run a risk of data loss when the=0A>>>> server=0A>>>> crashes=
.=0A>>>> Then there are a couple of other ZFS related things (I'm not a=
=0A>>>> ZFS=0A>>>> guy,=0A>>>> but these have shown up on the mailing lis=
ts).=0A>>>> - make sure your volumes are 4K aligned and ashift=3D12 (in c=
ase a=0A>>>> drive=0A>>>> that uses 4K sectors is pretending to be 512byt=
e sectored)=0A>>>> - never run over 70-80% full if write performance is a=
n issue=0A>>>> - use a zil on an SSD with good write performance=0A>>>> =
=0A>>>> The only NFSv4 thing I can tell you is that it is known that=0A>>=
>> ZFS's=0A>>>> algorithm for determining sequential vs random I/O fails =
for=0A>>>> NFSv4=0A>>>> during writing and this can be a performance hit.=
The only=0A>>>> workaround=0A>>>> is to use NFSv3 mounts, since file han=
dle affinity apparently=0A>>>> fixes=0A>>>> the problem and this is only =
done for NFSv3.=0A>>>> =0A>>>> rick=0A>>>> =0A>>>>> I saw that there are =
TSO issues with igb then i'm trying to=0A>>>>> disable=0A>>>>> it with sy=
sctl but the situation wasn't solved.=0A>>>>> =0A>>>>> Someone has got id=
eas ? I can give you more informations if you=0A>>>>> need.=0A>>>>> =0A>>=
>>> Thanks in advance.=0A>>>>> Regards,=0A>>>>> =0A>>>>> Lo=C3=AFc Blot,=
=0A>>>>> UNIX Systems, Network and Security Engineer=0A>>>>> http://www.u=
nix-experience.fr=0A>>>>> _______________________________________________=
=0A>>>>> freebsd-fs@freebsd.org mailing list=0A>>>>> http://lists.freebsd=
.org/mailman/listinfo/freebsd-fs=0A>>>>> To unsubscribe, send any mail to=
=0A>>>>> "freebsd-fs-unsubscribe@freebsd.org"=0A>>> =0A>>> ______________=
_________________________________=0A>>> freebsd-fs@freebsd.org mailing li=
st=0A>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>>>; To un=
subscribe, send any mail to=0A>>> "freebsd-fs-unsubscribe@freebsd.org"=0A=
>>> =0A>>> _______________________________________________=0A>>> freebsd-=
fs@freebsd.org mailing list=0A>>> http://lists.freebsd.org/mailman/listin=
fo/freebsd-fs=0A>>> To unsubscribe, send any mail to=0A>>> "freebsd-fs-un=
subscribe@freebsd.org"=0A>>> =0A>>> _____________________________________=
__________=0A>>> freebsd-fs@freebsd.org mailing list=0A>>> http://lists.f=
reebsd.org/mailman/listinfo/freebsd-fs=0A>>> To unsubscribe, send any mai=
l to=0A>>> "freebsd-fs-unsubscribe@freebsd.org"=0A>>> ___________________=
____________________________=0A>>> freebsd-fs@freebsd.org mailing list=0A=
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>>>; To unsubsc=
ribe, send any mail to=0A>>> "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?0eaadfe31ac4b8bbdeaf0baff696dada>