Date: Mon, 22 Dec 2014 08:41:26 +0000 From: "=?utf-8?B?TG/Dr2MgQmxvdA==?=" <loic.blot@unix-experience.fr> To: "Rick Macklem" <rmacklem@uoguelph.ca> Cc: freebsd-fs@freebsd.org Subject: Re: ZFS vnode lock deadlock in zfs_fhtovp was: High Kernel Load with nfsv4 Message-ID: <9fcfcbfe720a9b56a995cd6e227b8f9f@mail.unix-experience.fr> In-Reply-To: <2087358136.248078.1419122007097.JavaMail.root@uoguelph.ca> References: <2087358136.248078.1419122007097.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Rick,=0Amy 5 jails runs this weekend and now i have some stats on this=
monday.=0A=0AHopefully deadlock was fixed, yeah, but everything isn't go=
od :(=0A=0AOn NFSv4 server (FreeBSD 10.1) system uses 35% CPU=0A=0AAs i c=
an see this is because of nfsd:=0A=0A918 root 96 20 0 1235=
2K 3372K rpcsvc 6 51.4H 273.68% nfsd: server (nfsd)=0A=0AIf i look at =
dmesg i see:=0Anfsd server cache flooded, try increasing vfs.nfsd.tcphigh=
water=0A=0Avfs.nfsd.tcphighwater was set to 10000, i increase it to 15000=
=0A=0AHere is 'nfsstat -s' output:=0A=0AServer Info:=0A Getattr Setatt=
r Lookup Readlink Read Write Create Remove=0A 12600652=
1812 2501097 156 1386423 1983729 123 162067=0A=
Rename Link Symlink Mkdir Rmdir Readdir RdirPlus =
Access=0A 36762 9 0 0 0 3147 =
0 623524=0A Mknod Fsstat Fsinfo PathConf Commit=0A =
0 0 0 0 328117=0AServer Ret-Failed=0A =
0=0AServer Faults=0A 0=0AServer Cache Stats:=0A =
Inprog Idem Non-idem Misses=0A 0 0 0 126=
35512=0AServer Write Gathering:=0A WriteOps WriteRPC Opsaved=0A 19837=
29 1983729 0=0A=0AAnd here is 'procstat -kk' for nfsd (server)=
=0A=0A 918 100528 nfsd nfsd: master mi_switch+0xe1 sleep=
q_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x18=
b svc_run_internal+0x4a1 svc_run+0x1de nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x10=
7 sys_nfssvc+0x9c amd64_syscall+0x351 Xfast_syscall+0xfb =0A 918 100568 =
nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xa=
b sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_threa=
d_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100569 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_w=
ait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0x=
b fork_exit+0x9a fork_trampoline+0xe =0A 918 100570 nfsd nfs=
d: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0x=
f _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exi=
t+0x9a fork_trampoline+0xe =0A 918 100571 nfsd nfsd: service=
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait=
_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a for=
k_trampoline+0xe =0A 918 100572 nfsd nfsd: service mi_swi=
tch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a=
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoli=
ne+0xe =0A 918 100573 nfsd nfsd: service mi_switch+0xe1 s=
leepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_i=
nternal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A=
918 100574 nfsd nfsd: service mi_switch+0xe1 sleepq_catc=
h_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x=
87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 1005=
75 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+=
0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_th=
read_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100576 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleep=
q_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start=
+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100577 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig=
+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_=
exit+0x9a fork_trampoline+0xe =0A 918 100578 nfsd nfsd: serv=
ice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_w=
ait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a =
fork_trampoline+0xe =0A 918 100579 nfsd nfsd: service mi_=
switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x=
16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_tramp=
oline+0xe =0A 918 100580 nfsd nfsd: service mi_switch+0xe=
1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_ru=
n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =
=0A 918 100581 nfsd nfsd: service mi_switch+0xe1 sleepq_c=
atch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal=
+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 1=
00582 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signa=
ls+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc=
_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100583 nfsd=
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sl=
eepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_st=
art+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100584 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_=
sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fo=
rk_exit+0x9a fork_trampoline+0xe =0A 918 100585 nfsd nfsd: s=
ervice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _c=
v_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x=
9a fork_trampoline+0xe =0A 918 100586 nfsd nfsd: service =
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig=
+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_tr=
ampoline+0xe =0A 918 100587 nfsd nfsd: service mi_switch+=
0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0=
xe =0A 918 100588 nfsd nfsd: service mi_switch+0xe1 sleep=
q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_inter=
nal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 91=
8 100589 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_si=
gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e =
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100590 n=
fsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab=
sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread=
_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100591 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wa=
it_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb=
fork_exit+0x9a fork_trampoline+0xe =0A 918 100592 nfsd nfsd=
: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf=
_cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit=
+0x9a fork_trampoline+0xe =0A 918 100593 nfsd nfsd: service =
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_=
sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork=
_trampoline+0xe =0A 918 100594 nfsd nfsd: service mi_swit=
ch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a =
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampolin=
e+0xe =0A 918 100595 nfsd nfsd: service mi_switch+0xe1 sl=
eepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_in=
ternal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A =
918 100596 nfsd nfsd: service mi_switch+0xe1 sleepq_catch=
_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x8=
7e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 10059=
7 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thr=
ead_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100598 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq=
_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+=
0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100599 nfsd n=
fsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_e=
xit+0x9a fork_trampoline+0xe =0A 918 100600 nfsd nfsd: servi=
ce mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wa=
it_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a f=
ork_trampoline+0xe =0A 918 100601 nfsd nfsd: service mi_s=
witch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1=
6a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampo=
line+0xe =0A 918 100602 nfsd nfsd: service mi_switch+0xe1=
sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run=
_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =
=0A 918 100603 nfsd nfsd: service mi_switch+0xe1 sleepq_c=
atch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal=
+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 1=
00604 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signa=
ls+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc=
_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100605 nfsd=
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sl=
eepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_st=
art+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100606 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_=
sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fo=
rk_exit+0x9a fork_trampoline+0xe =0A 918 100607 nfsd nfsd: s=
ervice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _c=
v_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x=
9a fork_trampoline+0xe =0A 918 100608 nfsd nfsd: service =
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig=
+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_tr=
ampoline+0xe =0A 918 100609 nfsd nfsd: service mi_switch+=
0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0=
xe =0A 918 100610 nfsd nfsd: service mi_switch+0xe1 sleep=
q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_inter=
nal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 91=
8 100611 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_si=
gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e =
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100612 n=
fsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab=
sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread=
_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100613 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wa=
it_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb=
fork_exit+0x9a fork_trampoline+0xe =0A 918 100614 nfsd nfsd=
: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf=
_cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit=
+0x9a fork_trampoline+0xe =0A 918 100615 nfsd nfsd: service =
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_=
sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork=
_trampoline+0xe =0A 918 100616 nfsd nfsd: service mi_swit=
ch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a =
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampolin=
e+0xe =0A 918 100617 nfsd nfsd: service mi_switch+0xe1 sl=
eepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_in=
ternal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A =
918 100618 nfsd nfsd: service mi_switch+0xe1 sleepq_catch=
_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x8=
7e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 10061=
9 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thr=
ead_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100620 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq=
_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+=
0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100621 nfsd n=
fsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_e=
xit+0x9a fork_trampoline+0xe =0A 918 100622 nfsd nfsd: servi=
ce mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wa=
it_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a f=
ork_trampoline+0xe =0A 918 100623 nfsd nfsd: service mi_s=
witch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1=
6a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampo=
line+0xe =0A 918 100624 nfsd nfsd: service mi_switch+0xe1=
sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run=
_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =
=0A 918 100625 nfsd nfsd: service mi_switch+0xe1 sleepq_c=
atch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal=
+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 1=
00626 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signa=
ls+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc=
_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100627 nfsd=
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sl=
eepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_st=
art+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100628 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_=
sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fo=
rk_exit+0x9a fork_trampoline+0xe =0A 918 100629 nfsd nfsd: s=
ervice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _c=
v_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x=
9a fork_trampoline+0xe =0A 918 100630 nfsd nfsd: service =
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig=
+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_tr=
ampoline+0xe =0A 918 100631 nfsd nfsd: service mi_switch+=
0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0=
xe =0A 918 100632 nfsd nfsd: service mi_switch+0xe1 sleep=
q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_inter=
nal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 91=
8 100633 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_si=
gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e =
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100634 n=
fsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab=
sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread=
_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100635 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wa=
it_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb=
fork_exit+0x9a fork_trampoline+0xe =0A 918 100636 nfsd nfsd=
: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf=
_cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit=
+0x9a fork_trampoline+0xe =0A 918 100637 nfsd nfsd: service =
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_=
sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork=
_trampoline+0xe =0A 918 100638 nfsd nfsd: service mi_swit=
ch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a =
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampolin=
e+0xe =0A 918 100639 nfsd nfsd: service mi_switch+0xe1 sl=
eepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_in=
ternal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A =
918 100640 nfsd nfsd: service mi_switch+0xe1 sleepq_catch=
_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x8=
7e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 10064=
1 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thr=
ead_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100642 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq=
_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+=
0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100643 nfsd n=
fsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_e=
xit+0x9a fork_trampoline+0xe =0A 918 100644 nfsd nfsd: servi=
ce mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wa=
it_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a f=
ork_trampoline+0xe =0A 918 100645 nfsd nfsd: service mi_s=
witch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1=
6a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampo=
line+0xe =0A 918 100646 nfsd nfsd: service mi_switch+0xe1=
sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run=
_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =
=0A 918 100647 nfsd nfsd: service mi_switch+0xe1 sleepq_c=
atch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal=
+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 1=
00648 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signa=
ls+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc=
_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100649 nfsd=
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sl=
eepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_st=
art+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100650 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_=
sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fo=
rk_exit+0x9a fork_trampoline+0xe =0A 918 100651 nfsd nfsd: s=
ervice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _c=
v_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x=
9a fork_trampoline+0xe =0A 918 100652 nfsd nfsd: service =
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig=
+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_tr=
ampoline+0xe =0A 918 100653 nfsd nfsd: service mi_switch+=
0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0=
xe =0A 918 100654 nfsd nfsd: service mi_switch+0xe1 sleep=
q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_inter=
nal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 91=
8 100655 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_si=
gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e =
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100656 n=
fsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab=
sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread=
_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100657 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wa=
it_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb=
fork_exit+0x9a fork_trampoline+0xe =0A 918 100658 nfsd nfsd=
: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf=
_cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit=
+0x9a fork_trampoline+0xe =0A 918 100659 nfsd nfsd: service =
mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_=
sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork=
_trampoline+0xe =0A 918 100660 nfsd nfsd: service mi_swit=
ch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a =
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampolin=
e+0xe =0A 918 100661 nfsd nfsd: service mi_switch+0xe1 sl=
eepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_in=
ternal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A =
918 100662 nfsd nfsd: service mi_switch+0xe1 sleepq_catch=
_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x8=
7e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A---=0A=0ANow=
if we look at client (FreeBSD 9.3)=0A=0AWe see system was very busy and =
do many and many interrupts=0A=0ACPU: 0.0% user, 0.0% nice, 37.8% syste=
m, 51.2% interrupt, 11.0% idle=0A=0AA look at process list shows that the=
re are many sendmail process in state nfstry=0A=0Anfstry 18 32:27 0.88=
% sendmail: Queue runner@00:30:00 for /var/spool/clientm=0A=0AHere is 'nf=
sstat -c' output:=0A=0AClient Info:=0ARpc Counts:=0A Getattr Setattr =
Lookup Readlink Read Write Create Remove=0A 1051347 =
1724 2494481 118 903902 1901285 162676 161899=0A =
Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Acc=
ess=0A 36744 2 0 114 40 3131 =
0 544136=0A Mknod Fsstat Fsinfo PathConf Commit=0A =
9 0 0 0 245821=0ARpc Info:=0A TimedOut Inv=
alid X Replies Retries Requests=0A 0 0 0 =
0 8356557=0ACache Info:=0AAttr Hits Misses Lkup Hits Misses Bio=
R Hits Misses BioW Hits Misses=0A108754455 491475 54229224 24=
37229 46814561 821723 5132123 1871871=0ABioRLHits Misses BioD =
Hits Misses DirE Hits Misses Accs Hits Misses=0A 144035 =
118 53736 2753 27813 1 57238839 544205=0A=0A=0AI=
f you need more things, tell me, i let the PoC in this state.=0A=0AThanks=
=0A=0ARegards,=0A=0ALo=C3=AFc Blot,=0AUNIX Systems, Network and Security =
Engineer=0Ahttp://www.unix-experience.fr=0A=0A21 d=C3=A9cembre 2014 01:33=
"Rick Macklem" <rmacklem@uoguelph.ca> a =C3=A9crit: =0A> Loic Blot wrote=
:=0A> =0A>> Hi Rick,=0A>> ok, i don't need locallocks, i haven't understa=
nd option was for that=0A>> usage, i removed it.=0A>> I do more tests on =
monday.=0A>> Thanks for the deadlock fix, for other people :)=0A> =0A> Go=
od. Please let us know if running with vfs.nfsd.enable_locallocks=3D0=0A>=
gets rid of the deadlocks? (I think it fixes the one you saw.)=0A> =0A> =
On the performance side, you might also want to try different values of=
=0A> readahead, if the Linux client has such a mount option. (With the=0A=
> NFSv4-ZFS sequential vs random I/O heuristic, I have no idea what the=
=0A> optimal readahead value would be.)=0A> =0A> Good luck with it and pl=
ease let us know how it goes, rick=0A> ps: I now have a patch to fix the =
deadlock when vfs.nfsd.enable_locallocks=3D1=0A> is set. I'll post it for=
anyone who is interested after I put it=0A> through some testing.=0A> =
=0A>> --=0A>> Best regards,=0A>> Lo=C3=AFc BLOT,=0A>> UNIX systems, secur=
ity and network engineer=0A>> http://www.unix-experience.fr=0A>>; =0A>> Le=
jeudi 18 d=C3=A9cembre 2014 =C3=A0 19:46 -0500, Rick Macklem a =C3=A9cri=
t :=0A>>> Loic Blot wrote:=0A>>>> Hi rick,=0A>>>> i tried to start a LXC =
container on Debian Squeeze from my=0A>>>> freebsd=0A>>>> ZFS+NFSv4 serve=
r and i also have a deadlock on nfsd=0A>>>> (vfs.lookup_shared=3D0). Dead=
lock procs each time i launch a=0A>>>> squeeze=0A>>>> container, it seems=
(3 tries, 3 fails).=0A>>>> =0A>>> Well, I`ll take a look at this `procst=
at -kk`, but the only thing=0A>>> I`ve seen posted w.r.t. avoiding deadlo=
cks in ZFS is to not use=0A>>> nullfs. (I have no idea if you are using a=
ny nullfs mounts, but=0A>>> if so, try getting rid of them.)=0A>>> =0A>>>=
Here`s a high level post about the ZFS and vnode locking problem,=0A>>> =
but there is no patch available, as far as I know.=0A>>> =0A>>> http://do=
cs.FreeBSD.org/cgi/mid.cgi?54739F41.8030407=0A>>> =0A>>> rick=0A>>> =0A>>=
>> 921 - D 0:00.02 nfsd: server (nfsd)=0A>>>> =0A>>>> Here is the =
procstat -kk=0A>>>> =0A>>>> PID TID COMM TDNAME =
KSTACK=0A>>>> 921 100538 nfsd nfsd: master mi_switch+0xe1=
=0A>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e=0A>>>> vop_st=
dlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>> nfsvno_advlock+0x119 n=
fsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad=0A>>>> nfsrvd_locku+0x283 nfsrvd_=
dorpc+0xec6 nfssvc_program+0x554=0A>>>> svc_run_internal+0xc77 svc_run+0x=
1de nfsrvd_nfsd+0x1ca=0A>>>> nfssvc_nfsd+0x107 sys_nfssvc+0x9c=0A>>>> 921=
100572 nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_ca=
tch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_tramp=
oline+0xe=0A>>>> 921 100573 nfsd nfsd: service mi_switch+0=
xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_=
sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x=
9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100574 nfsd nfsd: se=
rvice mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_st=
art+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100575 nfsd =
nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal=
+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>=
>>> 921 100576 nfsd nfsd: service mi_switch+0xe1=0A>>>> sl=
eepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>=
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> for=
k_trampoline+0xe=0A>>>> 921 100577 nfsd nfsd: service mi_s=
witch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _c=
v_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_=
exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100578 nfsd n=
fsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wa=
it_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_th=
read_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 10057=
9 nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_si=
gnals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_i=
nternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+=
0xe=0A>>>> 921 100580 nfsd nfsd: service mi_switch+0xe1=0A=
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x=
16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>=
>>> fork_trampoline+0xe=0A>>>> 921 100581 nfsd nfsd: service =
mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A=
>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0x=
b fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100582 nfsd =
nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sl=
eepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e=
svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 92=
1 100583 nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_c=
atch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> sv=
c_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_tram=
poline+0xe=0A>>>> 921 100584 nfsd nfsd: service mi_switch+=
0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait=
_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0=
x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100585 nfsd nfsd: s=
ervice mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig=
+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_s=
tart+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100586 nfsd=
nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+=
0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_interna=
l+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A=
>>>> 921 100587 nfsd nfsd: service mi_switch+0xe1=0A>>>> s=
leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A=
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fo=
rk_trampoline+0xe=0A>>>> 921 100588 nfsd nfsd: service mi_=
switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _=
cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork=
_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100589 nfsd =
nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_w=
ait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_t=
hread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 1005=
90 nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_s=
ignals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_=
internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline=
+0xe=0A>>>> 921 100591 nfsd nfsd: service mi_switch+0xe1=
=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig=
+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 921 100592 nfsd nfsd: serv=
ice mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0x=
f=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_star=
t+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100593 nfsd =
nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xa=
b sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0=
x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>=
> 921 100594 nfsd nfsd: service mi_switch+0xe1=0A>>>> slee=
pq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>=
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_=
trampoline+0xe=0A>>>> 921 100595 nfsd nfsd: service mi_swi=
tch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_=
wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_ex=
it+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100596 nfsd nfs=
d: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait=
_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thre=
ad_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100597 =
nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_sign=
als+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_int=
ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0x=
e=0A>>>> 921 100598 nfsd nfsd: service mi_switch+0xe1=0A>>=
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16=
a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>=
> fork_trampoline+0xe=0A>>>> 921 100599 nfsd nfsd: service =
mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>=
>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100600 nfsd =
nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab slee=
pq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 =
100601 nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_cat=
ch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_=
run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampo=
line+0xe=0A>>>> 921 100602 nfsd nfsd: service mi_switch+0x=
e1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_s=
ig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9=
a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100603 nfsd nfsd: ser=
vice mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0=
xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_sta=
rt+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100604 nfsd =
nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0x=
ab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+=
0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>=
>> 921 100605 nfsd nfsd: service mi_switch+0xe1=0A>>>> sle=
epq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>=
>> svc_run_internal+0x87e svc_thread_start+0xb=20fork_exit+0x9a=0A>>>> fo=
rk_trampoline+0xe=0A>>>> 921 100606 nfsd nfsd: service mi_=
switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _=
cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork=
_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100607 nfsd =
nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_w=
ait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_t=
hread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 1006=
08 nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_s=
ignals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_=
internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline=
+0xe=0A>>>> 921 100609 nfsd nfsd: service mi_switch+0xe1=
=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig=
+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 921 100610 nfsd nfsd: serv=
ice mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0x=
f=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_star=
t+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100611 nfsd =
nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xa=
b sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0=
x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>=
> 921 100612 nfsd nfsd: service mi_switch+0xe1=0A>>>> slee=
pq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>=
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_=
trampoline+0xe=0A>>>> 921 100613 nfsd nfsd: service mi_swi=
tch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_=
wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_ex=
it+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100614 nfsd nfs=
d: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait=
_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thre=
ad_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100615 =
nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_sign=
als+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_int=
ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0x=
e=0A>>>> 921 100616 nfsd nfsd: service mi_switch+0xe1=0A>>=
>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>> nf=
srv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1=0A>>>> nfsrv=
d_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>> svc_thre=
ad_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>> 921 100617 nfsd =
nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xa=
b sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0=
x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>=
> 921 100618 nfsd nfsd: service mi_switch+0xe1=0A>>>> slee=
pq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>> nfsrvd_do=
rpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>> svc_thread_s=
tart+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>> 921 100619 nfsd =
nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sl=
eepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e=
svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 92=
1 100620 nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_c=
atch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> sv=
c_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_tram=
poline+0xe=0A>>>> 921 100621 nfsd nfsd: service mi_switch+=
0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait=
_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0=
x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100622 nfsd nfsd: s=
ervice mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig=
+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_s=
tart+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100623 nfsd=
nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+=
0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_interna=
l+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A=
>>>> 921 100624 nfsd nfsd: service mi_switch+0xe1=0A>>>> s=
leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A=
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fo=
rk_trampoline+0xe=0A>>>> 921 100625 nfsd nfsd: service mi_=
switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _=
cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork=
_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100626 nfsd =
nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_w=
ait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_t=
hread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 1006=
27 nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_s=
ignals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_=
internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline=
+0xe=0A>>>> 921 100628 nfsd nfsd: service mi_switch+0xe1=
=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig=
+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 921 100629 nfsd nfsd: serv=
ice mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0x=
f=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_star=
t+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100630 nfsd =
nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xa=
b sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0=
x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>=
> 921 100631 nfsd nfsd: service mi_switch+0xe1=0A>>>> slee=
pq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>=
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_=
trampoline+0xe=0A>>>> 921 100632 nfsd nfsd: service mi_swi=
tch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_=
wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_ex=
it+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100633 nfsd nfs=
d: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait=
_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thre=
ad_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100634 =
nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_sign=
als+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_int=
ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0x=
e=0A>>>> 921 100635 nfsd nfsd: service mi_switch+0xe1=0A>>=
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16=
a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>=
> fork_trampoline+0xe=0A>>>> 921 100636 nfsd nfsd: service =
mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>=
>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100637 nfsd =
nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab slee=
pq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 =
100638 nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_cat=
ch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_=
run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampo=
line+0xe=0A>>>> 921 100639 nfsd nfsd: service mi_switch+0x=
e1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_s=
ig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9=
a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100640 nfsd nfsd: ser=
vice mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0=
xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_sta=
rt+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100641 nfsd =
nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0x=
ab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+=
0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>=
>> 921 100642 nfsd nfsd: service mi_switch+0xe1=0A>>>> sle=
epq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>=
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork=
_trampoline+0xe=0A>>>> 921 100643 nfsd nfsd: service mi_sw=
itch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv=
_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e=
xit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100644 nfsd nf=
sd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wai=
t_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thr=
ead_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100645=
nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_sig=
nals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_in=
ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0=
xe=0A>>>> 921 100646 nfsd nfsd: service mi_switch+0xe1=0A>=
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x1=
6a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>=
>> fork_trampoline+0xe=0A>>>> 921 100647 nfsd nfsd: service =
mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>=
>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb=
fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100648 nfsd =
nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sle=
epq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e =
svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921=
100649 nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_ca=
tch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_tramp=
oline+0xe=0A>>>> 921 100650 nfsd nfsd: service mi_switch+0=
xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_=
sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x=
9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100651 nfsd nfsd: se=
rvice mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+=
0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_st=
art+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100652 nfsd =
nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal=
+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>=
>>> 921 100653 nfsd nfsd: service mi_switch+0xe1=0A>>>> sl=
eepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>=
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> for=
k_trampoline+0xe=0A>>>> 921 100654 nfsd nfsd: service mi_s=
witch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _c=
v_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_=
exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100655 nfsd n=
fsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wa=
it_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_th=
read_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 10065=
6 nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_si=
gnals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_i=
nternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+=
0xe=0A>>>> 921 100657 nfsd nfsd: service mi_switch+0xe1=0A=
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x=
16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>=
>>> fork_trampoline+0xe=0A>>>> 921 100658 nfsd nfsd: service =
mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A=
>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0x=
b fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100659 nfsd =
nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sl=
eepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e=
svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 92=
1 100660 nfsd nfsd: service mi_switch+0xe1=0A>>>> sleepq_c=
atch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> sv=
c_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_tram=
poline+0xe=0A>>>> 921 100661 nfsd nfsd: service mi_switch+=
0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait=
_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0=
x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100662 nfsd nfsd: s=
ervice mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig=
+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_s=
tart+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100663 nfsd=
nfsd: service mi_switch+0xe1=0A>>>> sleepq_catch_signals+=
0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A>>>> svc_run_interna=
l+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fork_trampoline+0xe=0A=
>>>> 921 100664 nfsd nfsd: service mi_switch+0xe1=0A>>>> s=
leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _cv_wait_sig+0x16a=0A=
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>> fo=
rk_trampoline+0xe=0A>>>> 921 100665 nfsd nfsd: service mi_=
switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>> _=
cv_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork=
_exit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 921 100666 nfsd =
nfsd: service mi_switch+0xe1=0A>>>> sleepq_wait+0x3a _sleep+0x287 nfsm=
sleep+0x66 nfsv4_lock+0x9b=0A>>>> nfsrv_setclient+0xbd nfsrvd_setclientid=
+0x3c8=0A>>>> nfsrvd_dorpc+0xc76=0A>>>> nfssvc_program+0x554 svc_run_inte=
rnal+0xc77=0A>>>> svc_thread_start+0xb=0A>>>> fork_exit+0x9a fork_trampol=
ine+0xe=0A>>>> =0A>>>> =0A>>>> Regards,=0A>>>> =0A>>>> Lo=C3=AFc Blot,=0A=
>>>> UNIX Systems, Network and Security Engineer=0A>>>> http://www.unix-e=
xperience.fr=0A>>>> =0A>>>> 15 d=C3=A9cembre 2014 15:18 "Rick Macklem" <r=
macklem@uoguelph.ca> a=0A>>>> =C3=A9crit:=0A>>>>> Loic Blot wrote:=0A>>>>=
> =0A>>>>>> For more informations, here is procstat -kk on nfsd, if you=
=0A>>>>>> need=0A>>>>>> more=0A>>>>>> hot datas, tell me.=0A>>>>>> =0A>>>=
>>> Regards, PID TID COMM TDNAME KSTACK=0A>>>>>=
> 918 100529 nfsd nfsd: master mi_switch+0xe1=0A>>>>>> sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x=
3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nf=
svno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_prog=
ram+0x554 svc_run_internal+0xc77 svc_run+0x1de=0A>>>>>> nfsrvd_nfsd+0x1ca=
nfssvc_nfsd+0x107 sys_nfssvc+0x9c=0A>>>>>> amd64_syscall+0x351=0A>>>>> =
=0A>>>>> Well, most of the threads are stuck like this one, waiting for=
=0A>>>>> a=0A>>>>> vnode=0A>>>>> lock in ZFS. All of them appear to be in=
zfs_fhtovp().=0A>>>>> I`m not a ZFS guy, so I can`t help much. I`ll try =
changing the=0A>>>>> subject line=0A>>>>> to include ZFS vnode lock, so m=
aybe the ZFS guys will take a=0A>>>>> look.=0A>>>>> =0A>>>>> The only thi=
ng I`ve seen suggested is trying:=0A>>>>> sysctl vfs.lookup_shared=3D0=0A=
>>>>> to disable shared vop_lookup()s. Apparently zfs_lookup()=0A>>>>> do=
esn`t=0A>>>>> obey the vnode locking rules for lookup and rename, accordi=
ng=0A>>>>> to=0A>>>>> the posting I saw.=0A>>>>> =0A>>>>> I`ve added a co=
uple of comments about the other threads below,=0A>>>>> but=0A>>>>> they =
are all either waiting for an RPC request or waiting for=0A>>>>> the=0A>>=
>>> threads stuck on the ZFS vnode lock to complete.=0A>>>>> =0A>>>>> ric=
k=0A>>>>> =0A>>>>>> 918 100564 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>> _c=
v_wait_sig+0x16a=0A>>>>>> svc_run_internal+0x87e svc_thread_start+0xb for=
k_exit+0x9a=0A>>>>>> fork_trampoline+0xe=0A>>>>> =0A>>>>> Fyi, this threa=
d is just waiting for an RPC to arrive. (Normal)=0A>>>>> =0A>>>>>> 918 10=
0565 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_cat=
ch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>> _cv_wait_sig+0x16a=0A>>>>>> =
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>> fork_=
trampoline+0xe=0A>>>>>> 918 100566 nfsd nfsd: service mi_s=
witch+0xe1=0A>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>=
> _cv_wait_sig+0x16a=0A>>>>>> svc_run_internal+0x87e svc_thread_start+0xb=
fork_exit+0x9a=0A>>>>>> fork_trampoline+0xe=0A>>>>>> 918 100567 nfsd =
nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_catch_signals+0x=
ab sleepq_wait_sig+0xf=0A>>>>>> _cv_wait_sig+0x16a=0A>>>>>> svc_run_inter=
nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>> fork_trampoline+0x=
e=0A>>>>>> 918 100568 nfsd nfsd: service mi_switch+0xe1=0A=
>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>> _cv_wait_si=
g+0x16a=0A>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x=
9a=0A>>>>>> fork_trampoline+0xe=0A>>>>>> 918 100569 nfsd nfsd=
: service mi_switch+0xe1=0A>>>>>> sleepq_catch_signals+0xab sleepq_wai=
t_sig+0xf=0A>>>>>> _cv_wait_sig+0x16a=0A>>>>>> svc_run_internal+0x87e svc=
_thread_start+0xb fork_exit+0x9a=0A>>>>>> fork_trampoline+0xe=0A>>>>>> 91=
8 100570 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq=
_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>> _cv_wait_sig+0x16a=0A>>>=
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>> f=
ork_trampoline+0xe=0A>>>>>> 918 100571 nfsd nfsd: service =
mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv=
4_lock+0x9b=0A>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_inte=
rnal+0xc77=0A>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0=
xe=0A>>>>>> 918 100572 nfsd nfsd: service mi_switch+0xe1=
=0A>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A=
>>>>>> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8=0A>>>>>> nfsrvd_dorp=
c+0xc76=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc=
_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> =0A=
>>>>> This one (and a few others) are waiting for the nfsv4_lock.=0A>>>>>=
This=0A>>>>> happens=0A>>>>> because other threads are stuck with RPCs i=
n progress. (ie. The=0A>>>>> ones=0A>>>>> waiting on the vnode lock in zf=
s_fhtovp().)=0A>>>>> For these, the RPC needs to lock out other threads t=
o do the=0A>>>>> operation,=0A>>>>> so it waits for the nfsv4_lock() whic=
h can exclusively lock the=0A>>>>> NFSv4=0A>>>>> data structures once all=
other nfsd threads complete their RPCs=0A>>>>> in=0A>>>>> progress.=0A>>=
>>> =0A>>>>>> 918 100573 nfsd nfsd: service mi_switch+0xe1=
=0A>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A=
>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77=0A>=
>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>> =0A=
>>>>> Same as above.=0A>>>>> =0A>>>>>> 918 100574 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockm=
gr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=
=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 n=
fsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A=
>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=
=0A>>>>>> 918 100575 nfsd nfsd: service mi_switch+0xe1=0A>=
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_st=
dlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>=
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfs=
svc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A=
>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100576 nfsd =
nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0=
x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _v=
n_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fh=
tovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_intern=
al+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_tramp=
oline+0xe=0A>>>>>> 918 100577 nfsd nfsd: service mi_switch=
+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>=
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+=
0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>=
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_sta=
rt+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100578 nf=
sd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a =
sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_AP=
V+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7=
c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_r=
un_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a f=
ork_trampoline+0xe=0A>>>>>> 918 100579 nfsd nfsd: service =
mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x9=
02=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zf=
s_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+=
0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_t=
hread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 =
100580 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_w=
ait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP=
_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_f=
htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x=
554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_ex=
it+0x9a fork_trampoline+0xe=0A>>>>>> 918 100581 nfsd nfsd: se=
rvice mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr=
_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A=
>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsr=
vd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>=
>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>=
>>>> 918 100582 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>=
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock=
+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>>=
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_p=
rogram+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>=
> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100583 nfsd =
nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d =
__lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_loc=
k+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+=
0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0x=
c77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline=
+0xe=0A>>>>>> 918 100584 nfsd nfsd: service mi_switch+0xe1=
=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vo=
p_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=
=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>=
nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0x=
b=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100585 nfsd =
nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleep=
lk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xa=
b _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfs=
d_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_in=
ternal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_t=
rampoline+0xe=0A>>>>>> 918 100586 nfsd nfsd: service mi_sw=
itch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A=
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fht=
ovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=
=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread=
_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 10058=
7 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0=
x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK=
1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp=
+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 s=
vc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x=
9a fork_trampoline+0xe=0A>>>>>> 918 100588 nfsd nfsd: service=
mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args=
+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>=
> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_do=
rpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> s=
vc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> =
918 100589 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> slee=
pq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c=
VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsv=
no_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_progra=
m+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> for=
k_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100590 nfsd nfsd=
: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __loc=
kmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x4=
3=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 =
nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=
=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0x=
e=0A>>>>>> 918 100591 nfsd nfsd: service mi_switch+0xe1=0A=
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_s=
tdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A=
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nf=
ssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=
=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100592 nfsd =
nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleepl=
k+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab=
_vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd=
_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_tr=
ampoline+0xe=0A>>>>>> 918 100593 nfsd nfsd: service mi_swi=
tch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>=
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhto=
vp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=
=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread=
_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 10059=
4 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0=
x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK=
1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp=
+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 s=
vc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x=
9a fork_trampoline+0xe=0A>>>>>> 918 100595 nfsd nfsd: service=
mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args=
+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>=
> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_do=
rpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> s=
vc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> =
918 100596 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> slee=
pq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c=
VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsv=
no_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_progra=
m+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> for=
k_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100597 nfsd nfsd=
: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __loc=
kmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x4=
3=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 =
nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=
=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0x=
e=0A>>>>>> 918 100598 nfsd nfsd: service mi_switch+0xe1=0A=
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_s=
tdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A=
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nf=
ssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=
=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100599 nfsd =
nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleepl=
k+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab=
_vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd=
_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_tr=
ampoline+0xe=0A>>>>>> 918 100600 nfsd nfsd: service mi_swi=
tch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>=
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhto=
vp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=
=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread=
_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 10060=
1 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0=
x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK=
1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp=
+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 s=
vc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x=
9a fork_trampoline+0xe=0A>>>>>> 918 100602 nfsd nfsd: service=
mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args=
+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>=
> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_do=
rpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> s=
vc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> =
918 100603 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> slee=
pq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c=
VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsv=
no_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_progra=
m+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> for=
k_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100604 nfsd nfsd=
: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __loc=
kmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x4=
3=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 =
nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=
=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0x=
e=0A>>>>>> 918 100605 nfsd nfsd: service mi_switch+0xe1=0A=
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_s=
tdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A=
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nf=
ssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=
=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100606 nfsd =
nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleepl=
k+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab=
_vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd=
_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_tr=
ampoline+0xe=0A>>>>>> 918 100607 nfsd nfsd: service mi_swi=
tch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>=
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhto=
vp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=
=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread=
_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>> =0A>>>>> L=
ots more waiting for the ZFS vnode lock in zfs_fhtovp().=0A>>>>> =0A>>>>>=
> 918 100608 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sl=
eepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>> nfsrv=
_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1=0A>>>>>> nfsrvd=
_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thr=
ead_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100609 nfsd=
nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sl=
eeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+=
0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run=
_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a for=
k_trampoline+0xe=0A>>>>>> 918 100610 nfsd nfsd: service mi=
_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e=
=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> nfsv=
no_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad=0A>>>>>> nfsrvd=
_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554=0A>>>>>> svc_run_int=
ernal+0xc77 svc_thread_start+0xb fork_exit+0x9a=0A>>>>>> fork_trampoline+=
0xe=0A>>>>>> 918 100611 nfsd nfsd: service mi_switch+0xe1=
=0A>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A=
>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77=0A>=
>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 91=
8 100612 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq=
_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>> nfsrvd_do=
rpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread=
_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100613 nfsd =
nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a _slee=
p+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>> nfsrvd_dorpc+0x316 nfssvc=
_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb fork_=
exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100614 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmslee=
p+0x66 nfsv4_lock+0x9b=0A>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 s=
vc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_t=
rampoline+0xe=0A>>>>>> 918 100615 nfsd nfsd: service mi_sw=
itch+0xe1=0A>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_loc=
k+0x9b=0A>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+=
0xc77=0A>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A=
>>>>>> 918 100616 nfsd nfsd: service mi_switch+0xe1=0A>>>>=
>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>> =
nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> s=
vc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 10061=
7 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0=
x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>> nfsrvd_dorpc+0x3=
16 nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+=
0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100618 nfsd =
nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a _sleep+0x287=
nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>> nfsrvd_dorpc+0x316 nfssvc_progra=
m+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb fork_exit+0x=
9a fork_trampoline+0xe=0A>>>>>> 918 100619 nfsd nfsd: service=
mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args=
+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>=
> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_do=
rpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> s=
vc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> =
918 100620 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> slee=
pq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c=
VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsv=
no_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_progra=
m+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> for=
k_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100621 nfsd nfsd=
: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsl=
eep+0x66 nfsv4_lock+0x9b=0A>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554=
svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb fork_exit+0x9a fork=
_trampoline+0xe=0A>>>>>> 918 100622 nfsd nfsd: service mi_=
switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=
=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_=
fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x=
917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thr=
ead_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 10=
0623 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wai=
t+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>> nfsrvd_dorpc+=
0x316 nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_sta=
rt+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100624 nfsd =
nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0=
x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _v=
n_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fh=
tovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_intern=
al+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_tramp=
oline+0xe=0A>>>>>> 918 100625 nfsd nfsd: service mi_switch=
+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>=
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+=
0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>=
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_sta=
rt+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100626 nf=
sd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a =
sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_AP=
V+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7=
c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_r=
un_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a f=
ork_trampoline+0xe=0A>>>>>> 918 100627 nfsd nfsd: service =
mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x9=
02=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zf=
s_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+=
0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_t=
hread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 =
100628 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_w=
ait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP=
_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_f=
htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x=
554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_ex=
it+0x9a fork_trampoline+0xe=0A>>>>>> 918 100629 nfsd nfsd: se=
rvice mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr=
_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A=
>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsr=
vd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>=
>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>=
>>>> 918 100630 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>=
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock=
+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>>=
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_p=
rogram+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>=
> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100631 nfsd =
nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d =
__lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_loc=
k+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+=
0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0x=
c77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline=
+0xe=0A>>>>>> 918 100632 nfsd nfsd: service mi_switch+0xe1=
=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vo=
p_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=
=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>=
nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0x=
b=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100633 nfsd=20=
=20=20=20=20 nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+=
0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOC=
K1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtov=
p+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 =
svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0=
x9a fork_trampoline+0xe=0A>>>>>> 918 100634 nfsd nfsd: servic=
e mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_arg=
s+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>=
>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d=
orpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> =
svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>=
918 100635 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sle=
epq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3=
c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfs=
vno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_progr=
am+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fo=
rk_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100636 nfsd nfs=
d: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lo=
ckmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x=
43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8=
nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=
=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0x=
e=0A>>>>>> 918 100637 nfsd nfsd: service mi_switch+0xe1=0A=
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_s=
tdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A=
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nf=
ssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=
=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100638 nfsd =
nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleepl=
k+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab=
_vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd=
_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_tr=
ampoline+0xe=0A>>>>>> 918 100639 nfsd nfsd: service mi_swi=
tch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>=
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhto=
vp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=
=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread=
_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 10064=
0 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0=
x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK=
1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp=
+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 s=
vc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x=
9a fork_trampoline+0xe=0A>>>>>> 918 100641 nfsd nfsd: service=
mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args=
+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>=
> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_do=
rpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> s=
vc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> =
918 100642 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> slee=
pq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c=
VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsv=
no_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_progra=
m+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> for=
k_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100643 nfsd nfsd=
: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __loc=
kmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x4=
3=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 =
nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=
=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0x=
e=0A>>>>>> 918 100644 nfsd nfsd: service mi_switch+0xe1=0A=
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_s=
tdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A=
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nf=
ssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=
=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100645 nfsd =
nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleepl=
k+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab=
_vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd=
_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_tr=
ampoline+0xe=0A>>>>>> 918 100646 nfsd nfsd: service mi_swi=
tch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>=
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhto=
vp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=
=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread=
_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 10064=
7 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0=
x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK=
1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp=
+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 s=
vc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x=
9a fork_trampoline+0xe=0A>>>>>> 918 100648 nfsd nfsd: service=
mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args=
+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>=
> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_do=
rpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> s=
vc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> =
918 100649 nfsd nfsd: service mi_switch+0xe1=0A>>>>>> slee=
pq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c=
VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsv=
no_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_progra=
m+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> for=
k_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100650 nfsd nfsd=
: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __loc=
kmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x4=
3=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 =
nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=
=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0x=
e=0A>>>>>> 918 100651 nfsd nfsd: service mi_switch+0xe1=0A=
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_s=
tdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A=
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nf=
ssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=
=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100652 nfsd =
nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleepl=
k+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab=
_vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd=
_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_int=
ernal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_tr=
ampoline+0xe=0A>>>>>> 918 100653 nfsd nfsd: service mi_swi=
tch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>=
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhto=
vp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=
=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread=
_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 10065=
4 nfsd=20=20=20=20=20=20=20 nfsd: service mi_switch+0xe1=0A>>>>>>=
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock=
+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>>=
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_p=
rogram+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>=
> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100655 nfsd =
nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d =
__lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_loc=
k+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+=
0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0x=
c77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline=
+0xe=0A>>>>>> 918 100656 nfsd nfsd: service mi_switch+0xe1=
=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>> vo=
p_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=
=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>=
nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread_start+0x=
b=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> 918 100657 nfsd =
nfsd: service mi_switch+0xe1=0A>>>>>> sleepq_wait+0x3a sleep=
lk+0x15d __lockmgr_args+0x902=0A>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xa=
b _vn_lock+0x43=0A>>>>>> zfs_fhtovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfs=
d_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>> nfssvc_program+0x554 svc_run_in=
ternal+0xc77=0A>>>>>> svc_thread_start+0xb=0A>>>>>> fork_exit+0x9a fork_t=
rampoline+0xe=0A>>>>>> 918 100658 nfsd nfsd: service mi_sw=
itch+0xe1=0A>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A=
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>> zfs_fht=
ovp+0x38d=0A>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=
=0A>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>> svc_thread=
_start+0xb=0A>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>> =0A>>>>>>=
Lo=C3=AFc Blot,=0A>>>>>> UNIX Systems, Network and Security Engineer=0A>=
>>>>> http://www.unix-experience.fr=0A>>>>>>; =0A>>>>>> 15 d=C3=A9cembre 2=
014 13:29 "Lo=C3=AFc Blot"=0A>>>>>> <loic.blot@unix-experience.fr>=0A>>>>=
>> a=0A>>>>>> =C3=A9crit:=0A>>>>>>> Hmmm...=0A>>>>>>> now i'm experiencin=
g a deadlock.=0A>>>>>>> =0A>>>>>>> 0 918 915 0 21 0 12352 3372 zfs D - 1:=
48.64 nfsd: server=0A>>>>>>> (nfsd)=0A>>>>>>> =0A>>>>>>> the only issue w=
as to reboot the server, but after rebooting=0A>>>>>>> deadlock arrives a=
second time when i=0A>>>>>>> start my jails over NFS.=0A>>>>>>> =0A>>>>>=
>> Regards,=0A>>>>>>> =0A>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>> UNIX Systems, =
Network and Security Engineer=0A>>>>>>> http://www.unix-experience.fr=0A>=
>>>>>> =0A>>>>>>> 15 d=C3=A9cembre 2014 10:07 "Lo=C3=AFc Blot"=0A>>>>>>> =
<loic.blot@unix-experience.fr>=0A>>>>>>> a=0A>>>>>>> =C3=A9crit:=0A>>>>>>=
> =0A>>>>>>> Hi Rick,=0A>>>>>>> after talking with my N+1, NFSv4 is requi=
red on our=0A>>>>>>> infrastructure.=0A>>>>>>> I tried to upgrade NFSv4+Z=
FS=0A>>>>>>> server from 9.3 to 10.1, i hope this will resolve some=0A>>>=
>>>> issues...=0A>>>>>>> =0A>>>>>>> Regards,=0A>>>>>>> =0A>>>>>>> Lo=C3=
=AFc Blot,=0A>>>>>>> UNIX Systems, Network and Security Engineer=0A>>>>>>=
> http://www.unix-experience.fr=0A>>>>>>>; =0A>>>>>>> 10 d=C3=A9cembre 201=
4 15:36 "Lo=C3=AFc Blot"=0A>>>>>>> <loic.blot@unix-experience.fr>=0A>>>>>=
>> a=0A>>>>>>> =C3=A9crit:=0A>>>>>>> =0A>>>>>>> Hi Rick,=0A>>>>>>> thanks=
for your suggestion.=0A>>>>>>> For my locking bug, rpc.lockd is stucked =
in rpcrecv state on=0A>>>>>>> the=0A>>>>>>> server. kill -9 doesn't affec=
t the=0A>>>>>>> process, it's blocked.... (State: Ds)=0A>>>>>>> =0A>>>>>>=
> for the performances=0A>>>>>>> =0A>>>>>>> NFSv3: 60Mbps=0A>>>>>>> NFSv4=
: 45Mbps=0A>>>>>>> Regards,=0A>>>>>>> =0A>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>=
> UNIX Systems, Network and Security Engineer=0A>>>>>>> http://www.unix-e=
xperience.fr=0A>>>>>>> =0A>>>>>>> 10 d=C3=A9cembre 2014 13:56 "Rick Mackl=
em" <rmacklem@uoguelph.ca>=0A>>>>>>> a=0A>>>>>>> =C3=A9crit:=0A>>>>>>> =
=0A>>>>>>>> Loic Blot wrote:=0A>>>>>>>> =0A>>>>>>>>> Hi Rick,=0A>>>>>>>>>=
I'm trying NFSv3.=0A>>>>>>>>> Some jails are starting very well but now =
i have an issue=0A>>>>>>>>> with=0A>>>>>>>>> lockd=0A>>>>>>>>> after some=
minutes:=0A>>>>>>>>> =0A>>>>>>>>> nfs server 10.10.X.8:/jails: lockd not=
responding=0A>>>>>>>>> nfs server 10.10.X.8:/jails lockd is alive again=
=0A>>>>>>>>> =0A>>>>>>>>> I look at mbuf, but i seems there is no problem=
.=0A>>>>>>>> =0A>>>>>>>> Well, if you need locks to be visible across mul=
tiple=0A>>>>>>>> clients,=0A>>>>>>>> then=0A>>>>>>>> I'm afraid you are s=
tuck with using NFSv4 and the=0A>>>>>>>> performance=0A>>>>>>>> you=0A>>>=
>>>>> get=0A>>>>>>>> from it. (There is no way to do file handle affinity=
for=0A>>>>>>>> NFSv4=0A>>>>>>>> because=0A>>>>>>>> the read and write op=
s are buried in the compound RPC and=0A>>>>>>>> not=0A>>>>>>>> easily=0A>=
>>>>>>> recognized.)=0A>>>>>>>> =0A>>>>>>>> If the locks don't need to be=
visible across multiple=0A>>>>>>>> clients,=0A>>>>>>>> I'd=0A>>>>>>>> su=
ggest trying the "nolockd" option with nfsv3.=0A>>>>>>>> =0A>>>>>>>>> Her=
e is my rc.conf on server:=0A>>>>>>>>> =0A>>>>>>>>> nfs_server_enable=3D"=
YES"=0A>>>>>>>>> nfsv4_server_enable=3D"YES"=0A>>>>>>>>> nfsuserd_enable=
=3D"YES"=0A>>>>>>>>> nfsd_server_flags=3D"-u -t -n 256"=0A>>>>>>>>> mount=
d_enable=3D"YES"=0A>>>>>>>>> mountd_flags=3D"-r"=0A>>>>>>>>> nfsuserd_fla=
gs=3D"-usertimeout 0 -force 20"=0A>>>>>>>>> rpcbind_enable=3D"YES"=0A>>>>=
>>>>> rpc_lockd_enable=3D"YES"=0A>>>>>>>>> rpc_statd_enable=3D"YES"=0A>>>=
>>>>>> =0A>>>>>>>>> Here is the client:=0A>>>>>>>>> =0A>>>>>>>>> nfsuserd=
_enable=3D"YES"=0A>>>>>>>>> nfsuserd_flags=3D"-usertimeout 0 -force 20"=
=0A>>>>>>>>> nfscbd_enable=3D"YES"=0A>>>>>>>>> rpc_lockd_enable=3D"YES"=
=0A>>>>>>>>> rpc_statd_enable=3D"YES"=0A>>>>>>>>> =0A>>>>>>>>> Have you g=
ot an idea ?=0A>>>>>>>>> =0A>>>>>>>>> Regards,=0A>>>>>>>>> =0A>>>>>>>>> L=
o=C3=AFc Blot,=0A>>>>>>>>> UNIX Systems, Network and Security Engineer=0A=
>>>>>>>>> http://www.unix-experience.fr=0A>>>>>>>>>; =0A>>>>>>>>> 9 d=C3=
=A9cembre 2014 04:31 "Rick Macklem" <rmacklem@uoguelph.ca>=0A>>>>>>>>> a=
=0A>>>>>>>>> =C3=A9crit:=0A>>>>>>>>>> Loic Blot wrote:=0A>>>>>>>>>> =0A>>=
>>>>>>>>> Hi rick,=0A>>>>>>>>>>> =0A>>>>>>>>>>> I waited 3 hours (no lag =
at jail launch) and now I do:=0A>>>>>>>>>>> sysrc=0A>>>>>>>>>>> memcached=
_flags=3D"-v -m 512"=0A>>>>>>>>>>> Command was very very slow...=0A>>>>>>=
>>>>> =0A>>>>>>>>>>> Here is a dd over NFS:=0A>>>>>>>>>>> =0A>>>>>>>>>>> =
601062912 bytes transferred in 21.060679 secs (28539579=0A>>>>>>>>>>> byt=
es/sec)=0A>>>>>>>>>> =0A>>>>>>>>>> Can you try the same read using an NFS=
v3 mount?=0A>>>>>>>>>> (If it runs much faster, you have probably been bi=
tten by=0A>>>>>>>>>> the=0A>>>>>>>>>> ZFS=0A>>>>>>>>>> "sequential vs ran=
dom" read heuristic which I've been told=0A>>>>>>>>>> things=0A>>>>>>>>>>=
NFS is doing "random" reads without file handle affinity.=0A>>>>>>>>>> F=
ile=0A>>>>>>>>>> handle affinity is very hard to do for NFSv4, so it isn'=
t=0A>>>>>>>>>> done.)=0A>>>>>>>> =0A>>>>>>>> I was actually suggesting th=
at you try the "dd" over nfsv3=0A>>>>>>>> to=0A>>>>>>>> see=0A>>>>>>>> ho=
w=0A>>>>>>>> the performance compared with nfsv4. If you do that, please=
=0A>>>>>>>> post=0A>>>>>>>> the=0A>>>>>>>> comparable results.=0A>>>>>>>>=
=0A>>>>>>>> Someday I would like to try and get ZFS's sequential vs=0A>>=
>>>>>> random=0A>>>>>>>> read=0A>>>>>>>> heuristic modified and any info =
on what difference in=0A>>>>>>>> performance=0A>>>>>>>> that=0A>>>>>>>> m=
ight make for NFS would be useful.=0A>>>>>>>> =0A>>>>>>>> rick=0A>>>>>>>>=
=0A>>>>>>>>>> rick=0A>>>>>>>>>> =0A>>>>>>>>>>> This is quite slow...=0A>=
>>>>>>>>>> =0A>>>>>>>>>>> You can found some nfsstat below (command isn't=
finished=0A>>>>>>>>>>> yet)=0A>>>>>>>>>>> =0A>>>>>>>>>>> nfsstat -c -w 1=
=0A>>>>>>>>>>> =0A>>>>>>>>>>> GtAttr Lookup Rdlink Read Write Rename Acce=
ss Rddir=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 4 0 0 0 0 0 16 0=0A>=
>>>>>>>>>> 2 0 0 0 0 0 17 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0=
0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 =
0=0A>>>>>>>>>>> 0 4 0 0 0 0 4 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>=
>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 =
0 0 0=0A>>>>>>>>>>> 4 0 0 0 0 0 3 0=0A>>>>>>>>>>> 0 0 0 0 0 0 3 0=0A>>>>>=
>>>>>> 37 10 0 8 0 0 14 1=0A>>>>>>>>>>> 18 16 0 4 1 2 4 0=0A>>>>>>>>>>> 7=
8 91 0 82 6 12 30 0=0A>>>>>>>>>>> 19 18 0 2 2 4 2 0=0A>>>>>>>>>>> 0 0 0 0=
2 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> GtAttr Lookup Rdlink=
Read Write Rename Access Rddir=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>=
>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 1 0 0 0 =
0 1 0=0A>>>>>>>>>>> 4 6 0 0 6 0 3 0=0A>>>>>>>>>>> 2 0 0 0 0 0 0 0=0A>>>>>=
>>>>>> 0 0 0 0=200 0 0 0=0A>>>>>>>>>>> 1 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 =
0 0 1 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=
=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>=
> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0=
0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 6 108 0 0 0 0 0 0=0A>>>>=
>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> GtAtt=
r Lookup Rdlink Read Write Rename Access Rddir=0A>>>>>>>>>>> 0 0 0 0 0 0 =
0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>=
>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 =
0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 98 54 0 86 11 0 25 0=
=0A>>>>>>>>>>> 36 24 0 39 25 0 10 1=0A>>>>>>>>>>> 67 8 0 63 63 0 41 0=0A>=
>>>>>>>>>> 34 0 0 35 34 0 0 0=0A>>>>>>>>>>> 75 0 0 75 77 0 0 0=0A>>>>>>>>=
>>> 34 0 0 35 35 0 0 0=0A>>>>>>>>>>> 75 0 0 74 76 0 0 0=0A>>>>>>>>>>> 33 =
0 0 34 33 0 0 0=0A>>>>>>>>>>> 0 0 0 0 5 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 6=
0=0A>>>>>>>>>>> 11 0 0 0 0 0 11 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>=
>>>>> 0 17 0 0 0 0 1 0=0A>>>>>>>>>>> GtAttr Lookup Rdlink Read Write Rena=
me Access Rddir=0A>>>>>>>>>>> 4 5 0 0 0 0 12 0=0A>>>>>>>>>>> 2 0 0 0 0 0 =
26 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>=
>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0=
0 0 0 0=0A>>>>>>>>>>> 0 4 0 0 0 0 4 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>=
>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 4 0=
0 0 0 0 2 0=0A>>>>>>>>>>> 2 0 0 0 0 0 24 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=
=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>=
> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0=
0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> GtAttr Lookup Rdlink Rea=
d Write Rename Access Rddir=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0=
0 0 0 0 0 0 0=0A>>>>>>>>>>> 4 0 0 0 0 0 7 0=0A>>>>>>>>>>> 2 1 0 0 0 0 1 =
0=0A>>>>>>>>>>> 0 0 0 0 2 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>=
>> 0 0 0 0 6 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 =
0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>=
>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 4 6 0 =
0 0 0 3 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 2 0 0 0 0 0 0 0=0A>=
>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 =
0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> GtAttr Lookup R=
dlink Read Write Rename Access Rddir=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>=
>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0=
0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 4 71 0 0 0 0 0 0=
=0A>>>>>>>>>>> 0 1 0 0 0 0 0 0=0A>>>>>>>>>>> 2 36 0 0 0 0 1 0=0A>>>>>>>>>=
>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 =
0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 1 0 0 0 0 0 1 0=0A>>>>>=
>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 79 6 0=
79 79 0 2 0=0A>>>>>>>>>>> 25 0 0 25 26 0 6 0=0A>>>>>>>>>>> 43 18 0 39 46=
0 23 0=0A>>>>>>>>>>> 36 0 0 36 36 0 31 0=0A>>>>>>>>>>> 68 1 0 66 68 0 0 =
0=0A>>>>>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir=0A>>>=
>>>>>>>> 36 0 0 36 36 0 0 0=0A>>>>>>>>>>> 48 0 0 48 49 0 0 0=0A>>>>>>>>>>=
> 20 0 0 20 20 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 3 14 0 1=
0 0 11 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>=
>>>>>>>>>> 0 4 0 0 0 0 4 0=0A>>>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>>> 4 =
22 0 0 0 0 16 0=0A>>>>>>>>>>> 2 0 0 0 0 0 23 0=0A>>>>>>>>>>> =0A>>>>>>>>>=
>> Regards,=0A>>>>>>>>>>> =0A>>>>>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>>>>>> UN=
IX Systems, Network and Security Engineer=0A>>>>>>>>>>> http://www.unix-e=
xperience.fr=0A>>>>>>>>>>> =0A>>>>>>>>>>> 8 d=C3=A9cembre 2014 09:36 "Lo=
=C3=AFc Blot"=0A>>>>>>>>>>> <loic.blot@unix-experience.fr> a=0A>>>>>>>>>>=
> =C3=A9crit:=0A>>>>>>>>>>>> Hi Rick,=0A>>>>>>>>>>>> I stopped the jails =
this week-end and started it this=0A>>>>>>>>>>>> morning,=0A>>>>>>>>>>>> =
i'll=0A>>>>>>>>>>>> give you some stats this week.=0A>>>>>>>>>>>> =0A>>>>=
>>>>>>>> Here is my nfsstat -m output (with your rsize/wsize=0A>>>>>>>>>>=
>> tweaks)=0A>>>>>>> =0A>>>>>>> =0A>>>>>> =0A>>>>> =0A>> =0A> nfsv4,tcp,r=
esvport,hard,cto,sec=3Dsys,acdirmin=3D3,acdirmax=3D60,acregmin=3D5,acregm=
ax=3D60,nametimeo=3D60,negna=0A>>>>>>> =0A>>>>>>>>>>>> =0A>>>>>>> =0A>>>>=
>>> =0A>>>>>> =0A>>>>> =0A>> =0A> etimeo=3D60,rsize=3D32768,wsize=3D32768=
,readdirsize=3D32768,readahead=3D1,wcommitsize=3D773136,timeout=3D120,ret=
ra=0A>>>>>>> =0A>>>>>>> s=3D2147483647=0A>>>>>>> =0A>>>>>>> On server sid=
e my disks are on a raid controller which show a=0A>>>>>>> 512b=0A>>>>>>>=
volume and write performances=0A>>>>>>> are very honest (dd if=3D/dev/ze=
ro of=3D/jails/test.dd bs=3D4096=0A>>>>>>> count=3D100000000 =3D> 450MBps=
)=0A>>>>>>> =0A>>>>>>> Regards,=0A>>>>>>> =0A>>>>>>> Lo=C3=AFc Blot,=0A>>=
>>>>> UNIX Systems, Network and Security Engineer=0A>>>>>>> http://www.un=
ix-experience.fr=0A>>>>>>> =0A>>>>>>> 5 d=C3=A9cembre 2014 15:14 "Rick Ma=
cklem" <rmacklem@uoguelph.ca> a=0A>>>>>>> =C3=A9crit:=0A>>>>>>> =0A>>>>>>=
>> Loic Blot wrote:=0A>>>>>>>> =0A>>>>>>>>> Hi,=0A>>>>>>>>> i'm trying to=
create a virtualisation environment based on=0A>>>>>>>>> jails.=0A>>>>>>=
>>> Those jails are stored under a big ZFS pool on a FreeBSD=0A>>>>>>>>> =
9.3=0A>>>>>>>>> which=0A>>>>>>>>> export a NFSv4 volume. This NFSv4 volum=
e was mounted on a=0A>>>>>>>>> big=0A>>>>>>>>> hypervisor (2 Xeon E5v3 + =
128GB memory and 8 ports (but=0A>>>>>>>>> only 1=0A>>>>>>>>> was=0A>>>>>>=
>>> used at this time).=0A>>>>>>>>> =0A>>>>>>>>> The problem is simple, m=
y hypervisors runs 6 jails (used 1%=0A>>>>>>>>> cpu=0A>>>>>>>>> and=0A>>>=
>>>>>> 10GB RAM approximatively and less than 1MB bandwidth) and=0A>>>>>>=
>>> works=0A>>>>>>>>> fine at start but the system slows down and after 2=
-3 days=0A>>>>>>>>> become=0A>>>>>>>>> unusable. When i look at top comma=
nd i see 80-100% on=0A>>>>>>>>> system=0A>>>>>>>>> and=0A>>>>>>>>> comman=
ds are very very slow. Many process are tagged with=0A>>>>>>>>> nfs_cl*.=
=0A>>>>>>>> =0A>>>>>>>> To be honest, I would expect the slowness to be b=
ecause of=0A>>>>>>>> slow=0A>>>>>>>> response=0A>>>>>>>> from the NFSv4 s=
erver, but if you do:=0A>>>>>>>> # ps axHl=0A>>>>>>>> on a client when it=
is slow and post that, it would give us=0A>>>>>>>> some=0A>>>>>>>> more=
=0A>>>>>>>> information on where the client side processes are sitting.=
=0A>>>>>>>> If you also do something like:=0A>>>>>>>> # nfsstat -c -w 1=
=0A>>>>>>>> and let it run for a while, that should show you how many=0A>=
>>>>>>> RPCs=0A>>>>>>>> are=0A>>>>>>>> being done and which ones.=0A>>>>>=
>>> =0A>>>>>>>> # nfsstat -m=0A>>>>>>>> will show you what your mount is =
actually using.=0A>>>>>>>> The only mount option I can suggest trying is=
=0A>>>>>>>> "rsize=3D32768,wsize=3D32768",=0A>>>>>>>> since some network =
environments have difficulties with 64K.=0A>>>>>>>> =0A>>>>>>>> There are=
a few things you can try on the NFSv4 server side,=0A>>>>>>>> if=0A>>>>>=
>>> it=0A>>>>>>>> appears=0A>>>>>>>> that the clients are generating a la=
rge RPC load.=0A>>>>>>>> - disabling the DRC cache for TCP by setting=0A>=
>>>>>>> vfs.nfsd.cachetcp=3D0=0A>>>>>>>> - If the server is seeing a larg=
e write RPC load, then=0A>>>>>>>> "sync=3Ddisabled"=0A>>>>>>>> might help=
, although it does run a risk of data loss when=0A>>>>>>>> the=0A>>>>>>>>=
server=0A>>>>>>>> crashes.=0A>>>>>>>> Then there are a couple of other Z=
FS related things (I'm not=0A>>>>>>>> a=0A>>>>>>>> ZFS=0A>>>>>>>> guy,=0A=
>>>>>>>> but these have shown up on the mailing lists).=0A>>>>>>>> - make=
sure your volumes are 4K aligned and ashift=3D12 (in=0A>>>>>>>> case a=
=0A>>>>>>>> drive=0A>>>>>>>> that uses 4K sectors is pretending to be 512=
byte sectored)=0A>>>>>>>> - never run over 70-80% full if write performan=
ce is an=0A>>>>>>>> issue=0A>>>>>>>> - use a zil on an SSD with good writ=
e performance=0A>>>>>>>> =0A>>>>>>>> The only NFSv4 thing I can tell you =
is that it is known that=0A>>>>>>>> ZFS's=0A>>>>>>>> algorithm for determ=
ining sequential vs random I/O fails for=0A>>>>>>>> NFSv4=0A>>>>>>>> duri=
ng writing and this can be a performance hit. The only=0A>>>>>>>> workaro=
und=0A>>>>>>>> is to use NFSv3 mounts, since file handle affinity=0A>>>>>=
>>> apparently=0A>>>>>>>> fixes=0A>>>>>>>> the problem and this is only d=
one for NFSv3.=0A>>>>>>>> =0A>>>>>>>> rick=0A>>>>>>>> =0A>>>>>>>>> I saw =
that there are TSO issues with igb then i'm trying to=0A>>>>>>>>> disable=
=0A>>>>>>>>> it with sysctl but the situation wasn't solved.=0A>>>>>>>>> =
=0A>>>>>>>>> Someone has got ideas ? I can give you more informations if=
=0A>>>>>>>>> you=0A>>>>>>>>> need.=0A>>>>>>>>> =0A>>>>>>>>> Thanks in adv=
ance.=0A>>>>>>>>> Regards,=0A>>>>>>>>> =0A>>>>>>>>> Lo=C3=AFc Blot,=0A>>>=
>>>>>> UNIX Systems, Network and Security Engineer=0A>>>>>>>>> http://www=
.unix-experience.fr=0A>>>>>>>>> _________________________________________=
______=0A>>>>>>>>> freebsd-fs@freebsd.org mailing list=0A>>>>>>>>> http:/=
/lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>>>>>>>>> To unsubscribe=
, send any mail to=0A>>>>>>>>> "freebsd-fs-unsubscribe@freebsd.org"=0A>>>=
>>>> =0A>>>>>>> _______________________________________________=0A>>>>>>>=
freebsd-fs@freebsd.org mailing list=0A>>>>>>> http://lists.freebsd.org/m=
ailman/listinfo/freebsd-fs=0A>>>>>>> To unsubscribe, send any mail to=0A>=
>>>>>> "freebsd-fs-unsubscribe@freebsd.org"=0A>>>>>>> =0A>>>>>>> ________=
_______________________________________=0A>>>>>>> freebsd-fs@freebsd.org =
mailing list=0A>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-=
fs=0A>>>>>>> To unsubscribe, send any mail to=0A>>>>>>> "freebsd-fs-unsub=
scribe@freebsd.org"=0A>>>>>>> =0A>>>>>>> ________________________________=
_______________=0A>>>>>>> freebsd-fs@freebsd.org mailing list=0A>>>>>>> h=
ttp://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>>>>>>> To unsubscr=
ibe, send any mail to=0A>>>>>>> "freebsd-fs-unsubscribe@freebsd.org"=0A>>=
>>>>> _______________________________________________=0A>>>>>>> freebsd-f=
s@freebsd.org mailing list=0A>>>>>>> http://lists.freebsd.org/mailman/lis=
tinfo/freebsd-fs=0A>>>>>>> To unsubscribe, send any mail to=0A>>>>>>> "fr=
eebsd-fs-unsubscribe@freebsd.org"=0A>>>>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9fcfcbfe720a9b56a995cd6e227b8f9f>