Date: Mon, 15 Dec 2014 12:34:32 +0000 From: "=?utf-8?B?TG/Dr2MgQmxvdA==?=" <loic.blot@unix-experience.fr> To: "Rick Macklem" <rmacklem@uoguelph.ca> Cc: freebsd-fs@freebsd.org Subject: Re: High Kernel Load with nfsv4 Message-ID: <e731671ef5dd2998ca4bf4ee1ffa429a@mail.unix-experience.fr> In-Reply-To: <db7be16e523322eec76d281a9a9c5934@mail.unix-experience.fr> References: <db7be16e523322eec76d281a9a9c5934@mail.unix-experience.fr> <2efc29240b59eabfdea79fe29744178d@mail.unix-experience.fr> <fc9e829cf79a03cd72f21226d276eb78@mail.unix-experience.fr> <1280247055.9141285.1418216202088.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
For more informations, here is procstat -kk on nfsd, if you need more hot=
datas, tell me.=0A=0A=0ARegards, PID TID COMM TDNAME =
KSTACK =0A 918 100529 nfsd nfsd=
: master mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args=
+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x=
554 svc_run_internal+0xc77 svc_run+0x1de nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x=
107 sys_nfssvc+0x9c amd64_syscall+0x351 =0A 918 100564 nfsd =
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig=
+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_=
exit+0x9a fork_trampoline+0xe =0A 918 100565 nfsd nfsd: serv=
ice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_w=
ait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a =
fork_trampoline+0xe =0A 918 100566 nfsd nfsd: service mi_=
switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x=
16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_tramp=
oline+0xe =0A 918 100567 nfsd nfsd: service mi_switch+0xe=
1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_ru=
n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =
=0A 918 100568 nfsd nfsd: service mi_switch+0xe1 sleepq_c=
atch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal=
+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 1=
00569 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signa=
ls+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc=
_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100570 nfsd=
nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sl=
eepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_st=
art+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100571 nfsd =
nfsd: service mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 nfsmslee=
p+0x66 nfsv4_lock+0x9b nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_in=
ternal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A =
918 100572 nfsd nfsd: service mi_switch+0xe1 sleepq_wait+=
0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b nfsrv_setclient+0xbd nfs=
rvd_setclientid+0x3c8 nfsrvd_dorpc+0xc76 nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A =
918 100573 nfsd nfsd: service mi_switch+0xe1 sleepq_wait+0=
x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b nfsrvd_dorpc+0x316 nfssvc=
_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a=
fork_trampoline+0xe =0A 918 100574 nfsd nfsd: service mi=
_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdl=
ock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+=
0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A =
918 100575 nfsd nfsd: service mi_switch+0xe1 sleepq_wait+0=
x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xa=
b _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsr=
vd_dorpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_sta=
rt+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100576 nfsd =
nfsd: service mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmg=
r_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp=
+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_prog=
ram+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork=
_trampoline+0xe =0A 918 100577 nfsd nfsd: service mi_swit=
ch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0=
x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_internal=
+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 1=
00578 nfsd nfsd: service mi_switch+0xe1 sleepq_wait+0x3a s=
leeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn=
_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_do=
rpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0x=
b fork_exit+0x9a fork_trampoline+0xe =0A 918 100579 nfsd nfs=
d: service mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_arg=
s+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38=
d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0=
x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_tram=
poline+0xe =0A 918 100580 nfsd nfsd: service mi_switch+0x=
e1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_=
fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc7=
7 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100581=
nfsd nfsd: service mi_switch+0xe1 sleepq_wait+0x3a sleepl=
k+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock=
+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0=
x917 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb for=
k_exit+0x9a fork_trampoline+0xe =0A 918 100582 nfsd nfsd: se=
rvice mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x9=
02 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfs=
vno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 =
svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampolin=
e+0xe =0A 918 100583 nfsd nfsd: service mi_switch+0xe1 sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LO=
CK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtov=
p+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc77 svc=
_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100584 nfsd=
nfsd: service mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x1=
5d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=
zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 =
nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exi=
t+0x9a fork_trampoline+0xe =0A 918 100585 nfsd nfsd: service=
mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vo=
p_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_f=
htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_r=
un_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=
=0A 918 100586 nfsd nfsd: service mi_switch+0xe1 sleepq_=
wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_A=
PV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc=
8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thre=
ad_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100587 nfsd =
nfsd: service mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __=
lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_=
fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssv=
c_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9=
a fork_trampoline+0xe =0A 918 100588 nfsd nfsd: service m=
i_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_std=
lock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp=
+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_in=
ternal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A =
918 100589 nfsd nfsd: service mi_switch+0xe1 sleepq_wait+=
0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0x=
ab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfs=
rvd_dorpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_st=
art+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100590 nfsd =
nfsd: service mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockm=
gr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtov=
p+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_pro=
gram+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a for=
k_trampoline+0xe =0A 918 100591 nfsd nfsd: service mi_swi=
tch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+=
0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c=
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_interna=
l+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 =
100592 nfsd nfsd: service mi_switch+0xe1 sleepq_wait+0x3a =
sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _v=
n_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d=
orpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0=
xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100593 nfsd nf=
sd: service mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_ar=
gs+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x3=
8d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+=
0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_tra=
mpoline+0xe =0A 918 100594 nfsd nfsd: service mi_switch+0=
xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c =
VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd=
_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc=
77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 10059=
5 nfsd nfsd: service mi_switch+0xe1 sleepq_wait+0x3a sleep=
lk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_loc=
k+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+=
0x917 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fo=
rk_exit+0x9a fork_trampoline+0xe =0A 918 100596 nfsd nfsd: s=
ervice mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x=
902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nf=
svno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554=
svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoli=
ne+0xe =0A 918 100597 nfsd nfsd: service mi_switch+0xe1 s=
leepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_L=
OCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhto=
vp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc77 sv=
c_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100598 nfs=
d nfsd: service mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x=
15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x4=
3 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=
nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_ex=
it+0x9a fork_trampoline+0xe =0A 918 100599 nfsd nfsd: servic=
e mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 v=
op_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_=
fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_=
run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0x=
e =0A 918 100600 nfsd nfsd: service mi_switch+0xe1 sleepq=
_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_=
APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0x=
c8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thr=
ead_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100601 nfsd =
nfsd: service mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d _=
_lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs=
_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfss=
vc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x=
9a fork_trampoline+0xe =0A 918 100602 nfsd nfsd: service =
mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_st=
dlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtov=
p+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_i=
nternal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A=
918 100603 nfsd nfsd: service mi_switch+0xe1 sleepq_wait=
+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0=
xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nf=
srvd_dorpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_s=
tart+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100604 nfsd =
nfsd: service mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lock=
mgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhto=
vp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_pr=
ogram+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fo=
rk_trampoline+0xe =0A 918 100605 nfsd nfsd: service mi_sw=
itch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock=
+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7=
c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_intern=
al+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918=
100606 nfsd nfsd: service mi_switch+0xe1 sleepq_wait+0x3a=
sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _=
vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_=
dorpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+=
0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100607 nfsd n=
fsd: service mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_a=
rgs+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x=
38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program=
+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_tr=
ampoline+0xe =0A 918 100608 nfsd nfsd: service mi_switch+=
0xe1 sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b nfsrv_g=
etlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1 nfsrvd_dorpc+0xec=
6 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_e=
xit+0x9a fork_trampoline+0xe =0A 918 100609 nfsd nfsd: servi=
ce mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 =
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno=
_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc=
_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0=
xe =0A 918 100610 nfsd nfsd: service mi_switch+0xe1 sleep=
q_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e vop_stdlock+0x3c VOP_LOCK1=
_APV+0xab _vn_lock+0x43 nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_loc=
kctrl+0x14ad nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554 s=
vc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline=
+0xe =0A 918 100611 nfsd nfsd: service mi_switch+0xe1 sle=
epq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b nfsrvd_dorpc+0x=
316 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork=
_exit+0x9a fork_trampoline+0xe =0A 918 100612 nfsd nfsd: ser=
vice mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4=
_lock+0x9b nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77=
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100613 =
nfsd nfsd: service mi_switch+0xe1 sleepq_wait+0x3a _sleep+=
0x287 nfsmsleep+0x66 nfsv4_lock+0x9b nfsrvd_dorpc+0x316 nfssvc_program+0x=
554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_tramp=
oline+0xe =0A 918 100614 nfsd nfsd: service mi_switch+0xe=
1 sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b nfsrvd_dor=
pc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
fork_exit+0x9a fork_trampoline+0xe =0A 918 100615 nfsd nfsd=
: service mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 =
nfsv4_lock+0x9b nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+=
0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 10=
0616 nfsd nfsd: service mi_switch+0xe1 sleepq_wait+0x3a _s=
leep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b nfsrvd_dorpc+0x316 nfssvc_progr=
am+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_=
trampoline+0xe =0A 918 100617 nfsd nfsd: service mi_switc=
h+0xe1 sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b nfsrv=
d_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_star=
t+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100618 nfsd =
nfsd: service mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 nfsmsleep+=
0x66 nfsv4_lock+0x9b nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_inte=
rnal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 9=
18 100619 nfsd nfsd: service mi_switch+0xe1 sleepq_wait+0x=
3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab=
_vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrv=
d_dorpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_star=
t+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100620 nfsd =
nfsd: service mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr=
_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+=
0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_progr=
am+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_=
trampoline+0xe =0A 918 100621 nfsd nfsd: service mi_switc=
h+0xe1 sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b nfsrv=
d_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_star=
t+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100622 nfsd =
nfsd: service mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr=
_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+=
0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_progr=
am+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_=
trampoline+0xe =0A 918 100623 nfsd nfsd: service mi_switc=
h+0xe1 sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b nfsrv=
d_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_star=
t+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100624 nfsd =
nfsd: service mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr=
_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+=
0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_progr=
am+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_=
trampoline+0xe =0A 918 100625 nfsd nfsd: service mi_switc=
h+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x=
3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c n=
fsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_internal+=
0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 10=
0626 nfsd nfsd: service mi_switch+0xe1 sleepq_wait+0x3a sl=
eeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_=
lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dor=
pc+0x917 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb=
fork_exit+0x9a fork_trampoline+0xe =0A 918 100627 nfsd nfsd=
: service mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args=
+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d=
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x=
554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_tramp=
oline+0xe =0A 918 100628 nfsd nfsd: service mi_switch+0xe=
1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VO=
P_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_f=
htovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc77=
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100629 =
nfsd nfsd: service mi_switch+0xe1 sleepq_wait+0x3a sleeplk=
+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+=
0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x=
917 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork=
_exit+0x9a fork_trampoline+0xe =0A 918 100630 nfsd nfsd: ser=
vice mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x90=
2 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsv=
no_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 s=
vc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline=
+0xe =0A 918 100631 nfsd nfsd: service mi_switch+0xe1 sle=
epq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOC=
K1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp=
+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc77 svc_=
thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100632 nfsd =
nfsd: service mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15=
d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 =
zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 n=
fssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exit=
+0x9a fork_trampoline+0xe =0A 918 100633 nfsd nfsd: service =
mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop=
_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fh=
tovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_ru=
n_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =
=0A 918 100634 nfsd nfsd: service mi_switch+0xe1 sleepq_w=
ait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_AP=
V+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8=
nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc77 svc_threa=
d_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100635 nfsd =
nfsd: service mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __l=
ockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_f=
htovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc=
_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a=
fork_trampoline+0xe =0A 918 100636 nfsd nfsd: service mi=
_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdl=
ock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+=
0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_int=
ernal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A =
918 100637 nfsd nfsd: service mi_switch+0xe1 sleepq_wait+0=
x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xa=
b _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsr=
vd_dorpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_sta=
rt+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100638 nfsd =
nfsd: service mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmg=
r_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp=
+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_prog=
ram+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork=
_trampoline+0xe =0A 918 100639 nfsd nfsd: service mi_swit=
ch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0=
x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c =
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_internal=
+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 1=
00640 nfsd nfsd: service mi_switch+0xe1 sleepq_wait+0x3a s=
leeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn=
_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_do=
rpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0x=
b fork_exit+0x9a fork_trampoline+0xe =0A 918 100641 nfsd nfs=
d: service mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_arg=
s+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38=
d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0=
x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_tram=
poline+0xe =0A 918 100642 nfsd nfsd: service mi_switch+0x=
e1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_=
fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc7=
7 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100643=
nfsd nfsd: service mi_switch+0xe1 sleepq_wait+0x3a sleepl=
k+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock=
+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0=
x917 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb for=
k_exit+0x9a fork_trampoline+0xe =0A 918 100644 nfsd nfsd: se=
rvice mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x9=
02 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfs=
vno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 =
svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampolin=
e+0xe =0A 918 100645 nfsd nfsd: service mi_switch+0xe1 sl=
eepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LO=
CK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtov=
p+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc77 svc=
_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100646 nfsd=
nfsd: service mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x1=
5d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=
zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 =
nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exi=
t+0x9a fork_trampoline+0xe =0A 918 100647 nfsd nfsd: service=
mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vo=
p_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_f=
htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_r=
un_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=
=0A 918 100648 nfsd nfsd: service mi_switch+0xe1 sleepq_=
wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_A=
PV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc=
8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thre=
ad_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100649 nfsd =
nfsd: service mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __=
lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_=
fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssv=
c_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9=
a fork_trampoline+0xe =0A 918 100650 nfsd nfsd: service m=
i_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_std=
lock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp=
+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_in=
ternal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A =
918 100651 nfsd nfsd: service mi_switch+0xe1 sleepq_wait+=
0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0x=
ab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfs=
rvd_dorpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_st=
art+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100652 nfsd =
nfsd: service mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockm=
gr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtov=
p+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_pro=
gram+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a for=
k_trampoline+0xe =0A 918 100653 nfsd nfsd: service mi_swi=
tch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+=
0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c=
nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_interna=
l+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 =
100654 nfsd nfsd: service mi_switch+0xe1 sleepq_wait+0x3a =
sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _v=
n_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d=
orpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0=
xb fork_exit+0x9a fork_trampoline+0xe =0A 918 100655 nfsd nf=
sd: service mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_ar=
gs+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x3=
8d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+=
0x554 svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_tra=
mpoline+0xe =0A 918 100656 nfsd nfsd: service mi_switch+0=
xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c =
VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd=
_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554 svc_run_internal+0xc=
77 svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A 918 10065=
7 nfsd nfsd: service mi_switch+0xe1 sleepq_wait+0x3a sleep=
lk+0x15d __lockmgr_args+0x902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_loc=
k+0x43 zfs_fhtovp+0x38d nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+=
0x917 nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb fo=
rk_exit+0x9a fork_trampoline+0xe =0A 918 100658 nfsd nfsd: s=
ervice mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x=
902 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d nf=
svno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917 nfssvc_program+0x554=
svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a fork_trampoli=
ne+0xe=0A=0ALo=C3=AFc Blot,=0AUNIX Systems, Network and Security Engineer=
=0Ahttp://www.unix-experience.fr=0A=0A15 d=C3=A9cembre 2014 13:29 "Lo=C3=
=AFc Blot" <loic.blot@unix-experience.fr> a =C3=A9crit: =0A> Hmmm...=0A> =
now i'm experiencing a deadlock.=0A> =0A> 0 918 915 0 21 0 12352 3372 zfs=
D - 1:48.64 nfsd: server (nfsd)=0A> =0A> the only issue was to reboot th=
e server, but after rebooting deadlock arrives a second time when i=0A> s=
tart my jails over NFS.=0A> =0A> Regards,=0A> =0A> Lo=C3=AFc Blot,=0A> UN=
IX Systems, Network and Security Engineer=0A> http://www.unix-experience.=
fr=0A> =0A> 15 d=C3=A9cembre 2014 10:07 "Lo=C3=AFc Blot" <loic.blot@unix-=
experience.fr> a =C3=A9crit:=0A> =0A> Hi Rick,=0A> after talking with my =
N+1, NFSv4 is required on our infrastructure. I tried to upgrade NFSv4+ZF=
S=0A> server from 9.3 to 10.1, i hope this will resolve some issues...=0A=
> =0A> Regards,=0A> =0A> Lo=C3=AFc Blot,=0A> UNIX Systems, Network and Se=
curity Engineer=0A> http://www.unix-experience.fr=0A>; =0A> 10 d=C3=A9cemb=
re 2014 15:36 "Lo=C3=AFc Blot" <loic.blot@unix-experience.fr> a =C3=A9cri=
t:=0A> =0A> Hi Rick,=0A> thanks for your suggestion.=0A> For my locking b=
ug, rpc.lockd is stucked in rpcrecv state on the server. kill -9 doesn't =
affect the=0A> process, it's blocked.... (State: Ds)=0A> =0A> for the per=
formances=0A> =0A> NFSv3: 60Mbps=0A> NFSv4: 45Mbps=0A> Regards,=0A> =0A> =
Lo=C3=AFc Blot,=0A> UNIX Systems, Network and Security Engineer=0A> http:=
//www.unix-experience.fr=0A> =0A> 10 d=C3=A9cembre 2014 13:56 "Rick Mackl=
em" <rmacklem@uoguelph.ca> a =C3=A9crit:=0A> =0A>> Loic Blot wrote:=0A>> =
=0A>>> Hi Rick,=0A>>> I'm trying NFSv3.=0A>>> Some jails are starting ver=
y well but now i have an issue with lockd=0A>>> after some minutes:=0A>>>=
=0A>>> nfs server 10.10.X.8:/jails: lockd not responding=0A>>> nfs serve=
r 10.10.X.8:/jails lockd is alive again=0A>>> =0A>>> I look at mbuf, but =
i seems there is no problem.=0A>> =0A>> Well, if you need locks to be vis=
ible across multiple clients, then=0A>> I'm afraid you are stuck with usi=
ng NFSv4 and the performance you get=0A>> from it. (There is no way to do=
file handle affinity for NFSv4 because=0A>> the read and write ops are b=
uried in the compound RPC and not easily=0A>> recognized.)=0A>> =0A>> If =
the locks don't need to be visible across multiple clients, I'd=0A>> sugg=
est trying the "nolockd" option with nfsv3.=0A>> =0A>>> Here is my rc.con=
f on server:=0A>>> =0A>>> nfs_server_enable=3D"YES"=0A>>> nfsv4_server_en=
able=3D"YES"=0A>>> nfsuserd_enable=3D"YES"=0A>>> nfsd_server_flags=3D"-u =
-t -n 256"=0A>>> mountd_enable=3D"YES"=0A>>> mountd_flags=3D"-r"=0A>>> nf=
suserd_flags=3D"-usertimeout 0 -force 20"=0A>>> rpcbind_enable=3D"YES"=0A=
>>> rpc_lockd_enable=3D"YES"=0A>>> rpc_statd_enable=3D"YES"=0A>>> =0A>>> =
Here is the client:=0A>>> =0A>>> nfsuserd_enable=3D"YES"=0A>>> nfsuserd_f=
lags=3D"-usertimeout 0 -force 20"=0A>>> nfscbd_enable=3D"YES"=0A>>> rpc_l=
ockd_enable=3D"YES"=0A>>> rpc_statd_enable=3D"YES"=0A>>> =0A>>> Have you =
got an idea ?=0A>>> =0A>>> Regards,=0A>>> =0A>>> Lo=C3=AFc Blot,=0A>>> UN=
IX Systems, Network and Security Engineer=0A>>> http://www.unix-experienc=
e.fr=0A>>> =0A>>> 9 d=C3=A9cembre 2014 04:31 "Rick Macklem" <rmacklem@uog=
uelph.ca> a =C3=A9crit: =0A>>>> Loic Blot wrote:=0A>>>> =0A>>>>> Hi rick,=
=0A>>>>> =0A>>>>> I waited 3 hours (no lag at jail launch) and now I do: =
sysrc=0A>>>>> memcached_flags=3D"-v -m 512"=0A>>>>> Command was very very=
slow...=0A>>>>> =0A>>>>> Here is a dd over NFS:=0A>>>>> =0A>>>>> 6010629=
12 bytes transferred in 21.060679 secs (28539579 bytes/sec)=0A>>>> =0A>>>=
> Can you try the same read using an NFSv3 mount?=0A>>>> (If it runs much=
faster, you have probably been bitten by the ZFS=0A>>>> "sequential vs r=
andom" read heuristic which I've been told things=0A>>>> NFS is doing "ra=
ndom" reads without file handle affinity. File=0A>>>> handle affinity is =
very hard to do for NFSv4, so it isn't done.)=0A>> =0A>> I was actually s=
uggesting that you try the "dd" over nfsv3 to see how=0A>> the performanc=
e compared with nfsv4. If you do that, please post the=0A>> comparable re=
sults.=0A>> =0A>> Someday I would like to try and get ZFS's sequential vs=
random read=0A>> heuristic modified and any info on what difference in p=
erformance that=0A>> might make for NFS would be useful.=0A>> =0A>> rick=
=0A>> =0A>>>> rick=0A>>>> =0A>>>>> This is quite slow...=0A>>>>> =0A>>>>>=
You can found some nfsstat below (command isn't finished yet)=0A>>>>> =
=0A>>>>> nfsstat -c -w 1=0A>>>>> =0A>>>>> GtAttr Lookup Rdlink Read Write=
Rename Access Rddir=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 4 0 0 0 0 0 16 0=0A>=
>>>> 2 0 0 0 0 0 17 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>=
>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 4 0 0 0 0 4 0=0A>>=
>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>=
>> 0 0 0 0 0 0 0 0=0A>>>>> 4 0 0 0 0 0 3 0=0A>>>>> 0 0 0 0 0 0 3 0=0A>>>>=
> 37 10 0 8 0 0 14 1=0A>>>>> 18 16 0 4 1 2 4 0=0A>>>>> 78 91 0 82 6 12 30=
0=0A>>>>> 19 18 0 2 2 4 2 0=0A>>>>> 0 0 0 0 2 0 0 0=0A>>>>> 0 0 0 0 0 0 =
0 0=0A>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir=0A>>>>> 0=
0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 =
1 0 0 0 0 1 0=0A>>>>> 4 6 0 0 6 0 3 0=0A>>>>> 2 0 0 0 0 0 0 0=0A>>>>> 0 0=
0 0 0 0 0 0=0A>>>>> 1 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 1 0 0 0=0A>>>>> 0 0 =
0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0=
0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 =
0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 6 108 0 0 0 0 0 0=0A>>>>> 0 0 0=
0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> GtAttr Lookup Rdlink Read Writ=
e Rename Access Rddir=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>=
>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>=
>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 98 54 0 86 11 0 25 0=
=0A>>>>> 36 24 0 39 25 0 10 1=0A>>>>> 67 8 0 63 63 0 41 0=0A>>>>> 34 0 0 =
35 34 0 0 0=0A>>>>> 75 0 0 75 77 0 0 0=0A>>>>> 34 0 0 35 35 0 0 0=0A>>>>>=
75 0 0 74 76 0 0 0=0A>>>>> 33 0 0 34 33 0 0 0=0A>>>>> 0 0 0 0 5 0 0 0=0A=
>>>>> 0 0 0 0 0 0 6 0=0A>>>>> 11 0 0 0 0 0 11 0=0A>>>>> 0 0 0 0 0 0 0 0=
=0A>>>>> 0 17 0 0 0 0 1 0=0A>>>>> GtAttr Lookup Rdlink Read Write Rename =
Access Rddir=0A>>>>> 4 5 0 0 0 0 12 0=0A>>>>> 2 0 0 0 0 0 26 0=0A>>>>> 0 =
0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0=
0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 4 0 0 0 0 4 0=0A>>>>> 0 0 =
0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 4 0 0=
0 0 0 2 0=0A>>>>> 2 0 0 0 0 0 24 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0=
0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 =
0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> GtAttr =
Lookup Rdlink Read Write Rename Access Rddir=0A>>>>> 0 0 0 0 0 0 0 0=0A>>=
>>> 0 0 0 0 0 0 0 0=0A>>>>> 4 0 0 0 0 0 7 0=0A>>>>> 2 1 0 0 0 0 1 0=0A>>>=
>> 0 0 0 0 2 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 6 0 0 0=0A>>>>=
> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>>=
0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> =
4 6 0 0 0 0 3 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 2 0 0 0 0 0 0 0=0A>>>>> 0=
0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 =
0 0 0 0 0 0 0=0A>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir=
=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=
=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 4 71 0 0 0 0 0 0=
=0A>>>>> 0 1 0 0 0 0 0 0=0A>>>>> 2 36 0 0 0 0 1 0=0A>>>>> 0 0 0 0 0 0 0 0=
=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=
=0A>>>>> 1 0 0 0 0 0 1 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0 0 0 0 0 0 0=
=0A>>>>> 79 6 0 79 79 0 2 0=0A>>>>> 25 0 0 25 26 0 6 0=0A>>>>> 43 18 0 39=
46 0 23 0=0A>>>>> 36 0 0 36 36 0 31 0=0A>>>>> 68 1 0 66 68 0 0 0=0A>>>>>=
GtAttr Lookup Rdlink Read Write Rename Access Rddir=0A>>>>> 36 0 0 36 36=
0 0 0=0A>>>>> 48 0 0 48 49 0 0 0=0A>>>>> 20 0 0 20 20 0 0 0=0A>>>>> 0 0 =
0 0 0 0 0 0=0A>>>>> 3 14 0 1 0 0 11 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 0 0=
0 0 0 0 0 0=0A>>>>> 0 4 0 0 0 0 4 0=0A>>>>> 0 0 0 0 0 0 0 0=0A>>>>> 4 22=
0 0 0 0 16 0=0A>>>>> 2 0 0 0 0 0 23 0=0A>>>>> =0A>>>>> Regards,=0A>>>>> =
=0A>>>>> Lo=C3=AFc Blot,=0A>>>>> UNIX Systems, Network and Security Engin=
eer=0A>>>>> http://www.unix-experience.fr=0A>>>>>; =0A>>>>> 8 d=C3=A9cembr=
e 2014 09:36 "Lo=C3=AFc Blot" <loic.blot@unix-experience.fr> a=0A>>>>> =
=C3=A9crit: =0A>>>>>> Hi Rick,=0A>>>>>> I stopped the jails this week-end=
and started it this morning,=0A>>>>>> i'll=0A>>>>>> give you some stats =
this week.=0A>>>>>> =0A>>>>>> Here is my nfsstat -m output (with your rsi=
ze/wsize tweaks)=0A> =0A> nfsv4,tcp,resvport,hard,cto,sec=3Dsys,acdirmin=
=3D3,acdirmax=3D60,acregmin=3D5,acregmax=3D60,nametimeo=3D60,negna=0A> =
=0A>>>>>> =0A> =0A> etimeo=3D60,rsize=3D32768,wsize=3D32768,readdirsize=
=3D32768,readahead=3D1,wcommitsize=3D773136,timeout=3D120,retra=0A> =0A> =
s=3D2147483647=0A> =0A> On server side my disks are on a raid controller =
which show a=0A> 512b=0A> volume and write performances=0A> are very hone=
st (dd if=3D/dev/zero of=3D/jails/test.dd bs=3D4096=0A> count=3D100000000=
=3D> 450MBps)=0A> =0A> Regards,=0A> =0A> Lo=C3=AFc Blot,=0A> UNIX System=
s, Network and Security Engineer=0A> http://www.unix-experience.fr=0A>; =
=0A> 5 d=C3=A9cembre 2014 15:14 "Rick Macklem" <rmacklem@uoguelph.ca> a=
=0A> =C3=A9crit:=0A> =0A>> Loic Blot wrote:=0A>> =0A>>> Hi,=0A>>> i'm try=
ing to create a virtualisation environment based on=0A>>> jails.=0A>>> Th=
ose jails are stored under a big ZFS pool on a FreeBSD 9.3=0A>>> which=0A=
>>> export a NFSv4 volume. This NFSv4 volume was mounted on a big=0A>>> h=
ypervisor (2 Xeon E5v3 + 128GB memory and 8 ports (but only 1=0A>>> was=
=0A>>> used at this time).=0A>>> =0A>>> The problem is simple, my hypervi=
sors runs 6 jails (used 1% cpu=0A>>> and=0A>>> 10GB RAM approximatively a=
nd less than 1MB bandwidth) and works=0A>>> fine at start but the system =
slows down and after 2-3 days=0A>>> become=0A>>> unusable. When i look at=
top command i see 80-100% on system=0A>>> and=0A>>> commands are very ve=
ry slow. Many process are tagged with=0A>>> nfs_cl*.=0A>> =0A>> To be hon=
est, I would expect the slowness to be because of slow=0A>> response=0A>>=
from the NFSv4 server, but if you do:=0A>> # ps axHl=0A>> on a client wh=
en it is slow and post that, it would give us some=0A>> more=0A>> informa=
tion on where the client side processes are sitting.=0A>> If you also do =
something like:=0A>> # nfsstat -c -w 1=0A>> and let it run for a while, t=
hat should show you how many RPCs=0A>> are=0A>> being done and which ones=
.=0A>> =0A>> # nfsstat -m=0A>> will show you what your mount is actually =
using.=0A>> The only mount option I can suggest trying is=0A>> "rsize=3D3=
2768,wsize=3D32768",=0A>> since some network environments have difficulti=
es with 64K.=0A>> =0A>> There are a few things you can try on the NFSv4 s=
erver side, if=0A>> it=0A>> appears=0A>> that the clients are generating =
a large RPC load.=0A>> - disabling the DRC cache for TCP by setting vfs.n=
fsd.cachetcp=3D0=0A>> - If the server is seeing a large write RPC load, t=
hen=0A>> "sync=3Ddisabled"=0A>> might help, although it does run a risk o=
f data loss when the=0A>> server=0A>> crashes.=0A>> Then there are a coup=
le of other ZFS related things (I'm not a=0A>> ZFS=0A>> guy,=0A>> but the=
se have shown up on the mailing lists).=0A>> - make sure your volumes are=
4K aligned and ashift=3D12 (in case a=0A>> drive=0A>> that uses 4K secto=
rs is pretending to be 512byte sectored)=0A>> - never run over 70-80% ful=
l if write performance is an issue=0A>> - use a zil on an SSD with good w=
rite performance=0A>> =0A>> The only NFSv4 thing I can tell you is that i=
t is known that=0A>> ZFS's=0A>> algorithm for determining sequential vs r=
andom I/O fails for=0A>> NFSv4=0A>> during writing and this can be a perf=
ormance hit. The only=0A>> workaround=0A>> is to use NFSv3 mounts, since =
file handle affinity apparently=0A>> fixes=0A>> the problem and this is o=
nly done for NFSv3.=0A>> =0A>> rick=0A>> =0A>>> I saw that there are TSO =
issues with igb then i'm trying to=0A>>> disable=0A>>> it with sysctl but=
the situation wasn't solved.=0A>>> =0A>>> Someone has got ideas ? I can =
give you more informations if you=0A>>> need.=0A>>> =0A>>> Thanks in adva=
nce.=0A>>> Regards,=0A>>> =0A>>> Lo=C3=AFc Blot,=0A>>> UNIX Systems, Netw=
ork and Security Engineer=0A>>> http://www.unix-experience.fr=0A>>>; _____=
__________________________________________=0A>>> freebsd-fs@freebsd.org m=
ailing list=0A>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A=
>>> To unsubscribe, send any mail to=0A>>> "freebsd-fs-unsubscribe@freebs=
d.org"=0A> =0A> _______________________________________________=0A> freeb=
sd-fs@freebsd.org mailing list=0A> http://lists.freebsd.org/mailman/listi=
nfo/freebsd-fs=0A> To unsubscribe, send any mail to=0A> "freebsd-fs-unsub=
scribe@freebsd.org"=0A> =0A> ____________________________________________=
___=0A> freebsd-fs@freebsd.org mailing list=0A> http://lists.freebsd.org/=
mailman/listinfo/freebsd-fs=0A> To unsubscribe, send any mail to "freebsd=
-fs-unsubscribe@freebsd.org"=0A> =0A> ___________________________________=
____________=0A> freebsd-fs@freebsd.org mailing list=0A> http://lists.fre=
ebsd.org/mailman/listinfo/freebsd-fs=0A> To unsubscribe, send any mail to=
"freebsd-fs-unsubscribe@freebsd.org"=0A> _______________________________=
________________=0A> freebsd-fs@freebsd.org mailing list=0A> http://lists=
.freebsd.org/mailman/listinfo/freebsd-fs=0A> To unsubscribe, send any mai=
l to "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?e731671ef5dd2998ca4bf4ee1ffa429a>