Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 14 Jan 2015 09:08:23 +0000
From:      "=?utf-8?B?TG/Dr2MgQmxvdA==?=" <loic.blot@unix-experience.fr>
To:        "Rick Macklem" <rmacklem@uoguelph.ca>
Cc:        freebsd-fs@freebsd.org
Subject:   Re: High Kernel Load with nfsv4
Message-ID:  <4426a915c891b0faf962f1975c4dee1e@mail.unix-experience.fr>
In-Reply-To: <1566336890.7368425.1420586936307.JavaMail.root@uoguelph.ca>
References:  <1566336890.7368425.1420586936307.JavaMail.root@uoguelph.ca>

next in thread | previous in thread | raw e-mail | index | archive | help
Hi Rick,=0Ayou fix doesn't help, i'm sorry. It improves the NFS server sp=
eed, but don't remove the problem.=0A=0AI think have loader.conf variable=
s for there options could be great.=0A=0AI have wrote a cpp bench for tes=
ting my NFS server configuration but i can't reproduce the problem wherea=
s the bench let many file descriptors opened (randomly).=0A=0ARegards,=0A=
=0ALo=C3=AFc Blot,=0AUNIX Systems, Network and Security Engineer=0Ahttp:/=
/www.unix-experience.fr=0A=0A7 janvier 2015 00:29 "Rick Macklem" <rmackle=
m@uoguelph.ca> a =C3=A9crit: =0A> Loic Blot wrote:=0A> =0A>> Hi Rick,=0A>=
> =0A>> i saw that some people has issues with igb cards with NFS=0A>> Fo=
r example:=0A>> http://freebsd.1045724.n5.nabble.com/NFS-over-LAGG-lacp-p=
oor-performance-td5906349.html=0A>> =0A>> can my problem be related ? I u=
se igb with default queue number. Here=0A>> are my vmstat -i outputs=0A> =
=0A> I have no idea. Maybe someone familiar with this will respond?=0A> =
=0A> I do think that the large # of NFSv4 Opens (which are actually a for=
m of lock)=0A> could be a factor. The client and server has to search tho=
se lists for a match=0A> for many NFSv4 operations, including all reads/w=
rites.=0A> =0A> On the server side, the default hash table sizes are very=
 small. This is in=0A> part that I did testing on 256Mbyte i386 systems, =
so that the values were=0A> safe for such a machine.=0A> I'd suggest you =
increase the following on the server's kernel.=0A> In sys/fs/nfs/nfs.h:=
=0A> NFSSTATEHASHSIZE - This one is in every client header, so if you hav=
e a large#=0A> of clients, you don't want to increase it too much. Howeve=
r=0A> for a fairly large server handling not too many clients, I'd=0A> tr=
y something like 1000 instead of 10.=0A> (I just tried 100 on the small i=
386 laptop I have handy and it=0A> seemed ok for a small test.)=0A> NFSLO=
CKHASHSIZE - This one is a single global table, so I'd bump it way up,=0A=
> 20000 maybe?=0A> In sys/fs/nfsport.h:=0A> NFSRV_V4STATELIMIT - The comm=
ent notes that the default of 500000 seems safe for=0A> a 256Mbyte i386, =
so I'd bump it to something like 2000000=0A> for your case.=0A> You will =
have to rebuild a kernel from sources after editing these values and=0A> =
boot it on the server. Maybe these should become tunables so building a k=
ernel=0A> isn't necessary?=0A> =0A> I looked and there isn't much that ca=
n be done in the client. At this point,=0A> the open_owners and opens are=
 single lists for a client (a mount point on a=0A> client machine for Fre=
eBSD). If you post what you get for "nfsstat -e -c"=0A> on a typical clie=
nt in your setup, that would tell me if it is the open_owners=0A> (which =
I suspect) or opens that will be a long list. (I would have to code=0A> a=
 patch to make either of these a hash table instead of a single linked=0A=
> list. I should do this. It was on my to-do list, but got forgotten.;-)=
=0A> =0A> rick=0A> =0A>> Server side:=0A>> =0A>> interrupt total rate=0A>=
> irq1: atkbd0 18 0=0A>> irq20: ehci1 2790134 2=0A>> irq21: ehci0 2547642=
 2=0A>> cpu0:timer 36299188 35=0A>> irq264: ciss0 6352476 6=0A>> irq265: =
igb0:que 0 2716692 2=0A>> irq266: igb0:que 1 32205278 31=0A>> irq267: igb=
0:que 2 38395109 37=0A>> irq268: igb0:que 3 1413468 1=0A>> irq269: igb0:q=
ue 4 39207930 38=0A>> irq270: igb0:que 5 1622715 1=0A>> irq271: igb0:que =
6 1634676 1=0A>> irq272: igb0:que 7 1190123 1=0A>> irq273: igb0:link 2 0=
=0A>> cpu1:timer 14074423 13=0A>> cpu8:timer 12204739 11=0A>> cpu9:timer =
11384192 11=0A>> cpu3:timer 10461566 10=0A>> cpu4:timer 12785103 12=0A>> =
cpu6:timer 10739344 10=0A>> cpu5:timer 10978294 10=0A>> cpu7:timer 105997=
05 10=0A>> cpu2:timer 13998891 13=0A>> cpu10:timer 11602361 11=0A>> cpu11=
:timer 11568523 11=0A>> Total 296772592 290=0A>> =0A>> And client side:=
=0A>> interrupt total rate=0A>> irq9: acpi0 4 0=0A>> irq22: ehci1 950519 =
2=0A>> irq23: ehci0 1865060 4=0A>> cpu0:timer 248128035 546=0A>> irq268: =
mfi0 406896 0=0A>> irq269: igb0:que 0 2510556 5=0A>> irq270: igb0:que 1 2=
825336 6=0A>> irq271: igb0:que 2 2092958 4=0A>> irq272: igb0:que 3 196084=
9 4=0A>> irq273: igb0:que 4 2645369 5=0A>> irq274: igb0:que 5 2735187 6=
=0A>> irq275: igb0:que 6 2290531 5=0A>> irq276: igb0:que 7 2384370 5=0A>>=
 irq277: igb0:link 2 0=0A>> irq287: igb2:que 0 1465051 3=0A>> irq288: igb=
2:que 1 856381 1=0A>> irq289: igb2:que 2 809318 1=0A>> irq290: igb2:que 3=
 897154 1=0A>> irq291: igb2:que 4 875755 1=0A>> irq292: igb2:que 5 358661=
17 78=0A>> irq293: igb2:que 6 846517 1=0A>> irq294: igb2:que 7 857979 1=
=0A>> irq295: igb2:link 2 0=0A>> irq296: igb3:que 0 535212 1=0A>> irq297:=
 igb3:que 1 454359 1=0A>> irq298: igb3:que 2 454142 1=0A>> irq299: igb3:q=
ue 3 454623 1=0A>> irq300: igb3:que 4 456297 1=0A>> irq301: igb3:que 5 45=
5482 1=0A>> irq302: igb3:que 6 456128 1=0A>> irq303: igb3:que 7 454680 1=
=0A>> irq304: igb3:link 3 0=0A>> irq305: ahci0 75 0=0A>> cpu1:timer 25723=
3702 566=0A>> cpu13:timer 255603184 562=0A>> cpu7:timer 258492826 569=0A>=
> cpu12:timer 255819351 563=0A>> cpu6:timer 258493465 569=0A>> cpu15:time=
r 254694003 560=0A>> cpu3:timer 258171320 568=0A>> cpu22:timer 256506877 =
564=0A>> cpu5:timer 253401435 558=0A>> cpu16:timer 255412360 562=0A>> cpu=
11:timer 257318013 566=0A>> cpu20:timer 253648060 558=0A>> cpu2:timer 257=
864543 567=0A>> cpu17:timer 261828899 576=0A>> cpu9:timer 257497326 567=
=0A>> cpu18:timer 258451190 569=0A>> cpu8:timer 257784504 567=0A>> cpu14:=
timer 254923723 561=0A>> cpu10:timer 257265498 566=0A>> cpu19:timer 25877=
5946 569=0A>> cpu4:timer 256368658 564=0A>> cpu23:timer 255050534 561=0A>=
> cpu21:timer 257663842 567=0A>> Total 6225260206 13710=0A>> =0A>> Please=
 note igb2 on client side is the dedicated link for NFSv4=0A>> =0A>> Rega=
rds,=0A>> =0A>> Lo=C3=AFc Blot,=0A>> UNIX Systems, Network and Security E=
ngineer=0A>> http://www.unix-experience.fr=0A>>; =0A>> 6 janvier 2015 04:1=
7 "Rick Macklem" <rmacklem@uoguelph.ca> a =C3=A9crit:=0A>>> Loic Blot wro=
te:=0A>>> =0A>>>> Hi Rick,=0A>>>> nfsstat -e -s don't show usefull datas =
on server.=0A>>> =0A>>> Well, as far as I know, it returns valid informat=
ion.=0A>>> (See below.)=0A>>> =0A>>>> Server Info:=0A>>>> Getattr Setattr=
 Lookup Readlink Read Write Create=0A>>>> Remove=0A>>>> 26935254 16911 57=
55728 302 2334920 3673866 0=0A>>>> 328332=0A>>>> Rename Link Symlink Mkdi=
r Rmdir Readdir RdirPlus=0A>>>> Access=0A>>>> 77980 28 0 0 3 8900 3=0A>>>=
> 1806052=0A>>>> Mknod Fsstat Fsinfo PathConf Commit LookupP SetClId=0A>>=
>> SetClIdCf=0A>>>> 1 1095 0 0 614377 8172 8=0A>>>> 8=0A>>>> Open OpenAtt=
r OpenDwnGr OpenCfrm DelePurge DeleRet GetFH=0A>>>> Lock=0A>>>> 1595299 0=
 44145 1495 0 0 5197490=0A>>>> 635015=0A>>>> LockT LockU Close Verify NVe=
rify PutFH PutPubFH=0A>>>> PutRootFH=0A>>>> 0 614919 1270938 0 0 22688676=
 0=0A>>>> 5=0A>>>> Renew RestoreFH SaveFH Secinfo RelLckOwn V4Create=0A>>=
>> 42104 197606 275820 0 143 4578=0A>>>> Server:=0A>>>> Retfailed Faults =
Clients=0A>>>> 0 0 6=0A>>>> OpenOwner Opens LockOwner Locks Delegs=0A>>>>=
 32335 145448 204 181 0=0A>>> =0A>>> Well, 145448 Opens are a lot of Open=
 files. Each of these uses=0A>>> a kernel malloc'd data structure that is=
 linked into multiple=0A>>> linked lists.=0A>>> =0A>>> The question is..w=
hy aren't these Opens being closed?=0A>>> Since FreeBSD does I/O on an mm=
ap'd file after closing it,=0A>>> the FreeBSD NFSv4 client is forced to d=
elay doing Close RPCs=0A>>> until the vnode is VOP_INACTIVE()/VOP_RECLAIM=
()'d. (The=0A>>> VOP_RECLAIM() case is needed, since VOP_INACTIVE() isn't=
=0A>>> guaranteed to be called.)=0A>>> =0A>>> Since there were about 1.5 =
million Opens and 1.27 million=0A>>> Closes, it does appear that Opens ar=
e being Closed.=0A>>> Now, I'm not sure I would have imagined 1.5million =
file Opens=0A>>> in a few days. My guess is this is the bottleneck.=0A>>>=
 =0A>>> I'd suggest that you do:=0A>>> # nfsstat -e -c=0A>>> on each of t=
he NFSv4 clients and see how many Opens/client=0A>>> there are. I vaguely=
 remember an upper limit in the client,=0A>>> but can't remember what it =
is set to.=0A>>> --> I suspect the client Open/Lock limit needs to be inc=
reased.=0A>>> (I can't remember if the server also has a limit, but I=0A>=
>> think it does.)=0A>>> Then the size of the hash tables used to search =
the Opens=0A>>> may also need to be increased a lot.=0A>>> =0A>>> Also, I=
'd suggest you take a look at whatever apps. are=0A>>> running on the cli=
ent(s) and try to figure out why they=0A>>> are Opening so many files?=0A=
>>> =0A>>> My guess is that the client(s) are gettig bogged down by all=
=0A>>> these Opens.=0A>>> =0A>>>> Server Cache Stats:=0A>>>> Inprog Idem =
Non-idem Misses CacheSize TCPPeak=0A>>>> 0 0 1 15082947 60 16522=0A>>>> =
=0A>>>> Only GetAttr and Lookup increase and it's only every 4-5 seconds=
=0A>>>> and=0A>>>> only +2 to +5 into theses values.=0A>>>> =0A>>>> Now o=
n client, if i take four processes stack i got=0A>>>> =0A>>>> PID TID COM=
M TDNAME KSTACK=0A>>>> 63170 102547 mv - mi_switch+0xe1=0A>>>> turnstile_=
wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65=0A>>>> nfs_lookup+0x=
3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4=0A>>>> vn_open_cred+0x21=
d kern_openat+0x26f amd64_syscall+0x351=0A>>>> Xfast_syscall+0xfb=0A>>>> =
=0A>>>> Another mv:=0A>>>> 63140 101738 mv - mi_switch+0xe1=0A>>>> turnst=
ile_wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65=0A>>>> nfs_looku=
p+0x3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4=0A>>>> kern_statat_v=
nhook+0xae sys_lstat+0x30 amd64_syscall+0x351=0A>>>> Xfast_syscall+0xfb=
=0A>>>> =0A>>>> 62070 102170 sendmail - mi_switch+0xe1=0A>>>> sleepq_time=
dwait+0x3a _sleep+0x26e clnt_vc_call+0x666=0A>>>> clnt_reconnect_call+0x4=
fa newnfs_request+0xa8c nfscl_request+0x72=0A>>>> nfsrpc_lookup+0x1fb nfs=
_lookup+0x508 VOP_LOOKUP_APV+0xa1=0A>>>> lookup+0x59c namei+0x4d4 kern_st=
atat_vnhook+0xae sys_lstat+0x30=0A>>>> amd64_syscall+0x351 Xfast_syscall+=
0xfb=0A>>>> =0A>>>> 63200 100930 mv - mi_switch+0xe1=0A>>>> turnstile_wai=
t+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65=0A>>>> nfs_lookup+0x3d0=
 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4=0A>>>> kern_statat_vnhook+0=
xae sys_lstat+0x30 amd64_syscall+0x351=0A>>>> Xfast_syscall+0xfb=0A>>> =
=0A>>> The above simply says that thread 102710 is waiting for a Lookup=
=0A>>> reply from the server and the other 3 are waiting for the mutex=0A=
>>> that protects the state structures in the client. (I suspect=0A>>> so=
me other thread in the client is wading through the Open list,=0A>>> if a=
 single client has a lot of these 145K Opens.)=0A>>> =0A>>>> When client =
is in this state, server was doing nothing special=0A>>>> (procstat -kk)=
=0A>>>> =0A>>>> PID TID COMM TDNAME KSTACK=0A>>>> 895 100538 nfsd nfsd: m=
aster mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_timedwait_si=
g+0x10=0A>>>> _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1 svc_run+=
0x1de=0A>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c=0A>>>> a=
md64_syscall+0x351 Xfast_syscall+0xfb=0A>>>> 895 100568 nfsd nfsd: servic=
e mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv=
_wait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e=
xit+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 895 100569 nfsd nfsd: service =
mi_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_w=
ait_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exi=
t+0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 895 100570 nfsd nfsd: service mi=
_switch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wai=
t_sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a=0A>>>> fork_trampoline+0xe=0A>>>> 895 100571 nfsd nfsd: service mi_s=
witch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_=
sig+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x=
9a=0A>>>> fork_trampoline+0xe=0A>>>> 895 100572 nfsd nfsd: service mi_swi=
tch+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si=
g+0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100573 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100575 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100576 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100577 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100578 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100579 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100580 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100581 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100582 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100583 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100584 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100585 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100586 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100587 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100588 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100589 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100590 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100592 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100593 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100594 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100595 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100596 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100597 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100598 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100599 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100600 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100602 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100603 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100604 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100605 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100606 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100607 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100608 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100609 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100610 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100611 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100612 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100613 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100614 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100615 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100617 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100618 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100619 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100621 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100622 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100623 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100624 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100625 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100626 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100627 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100628 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100629 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100630 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100631 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100632 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100633 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100634 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100635 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100636 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100638 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100639 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100640 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100641 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100642 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100643 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100644 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100645 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100646 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100647 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100648 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100649 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100651 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100652 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100653 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100654 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100655 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100656 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100657 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100658 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100659 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100661 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100662 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100684 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100685 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100686 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100797 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100798 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100799 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100800 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> 895 100801 nfsd nfsd: service mi_switc=
h+0xe1=0A>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+=
0x16a=0A>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>> fork_trampoline+0xe=0A>>>> =0A>>>> I really think it's a client s=
ide problem, maybe a lookup problem.=0A>>>> =0A>>>> Regards,=0A>>>> =0A>>=
>> Lo=C3=AFc Blot,=0A>>>> UNIX Systems, Network and Security Engineer=0A>=
>>> http://www.unix-experience.fr=0A>>>>; =0A>>>> 5 janvier 2015 14:35 "Ri=
ck Macklem" <rmacklem@uoguelph.ca> a=0A>>>> =C3=A9crit:=0A>>>>> Loic Blot=
 wrote:=0A>>>>> =0A>>>>>> Hi,=0A>>>>>> happy new year Rick and @freebsd-f=
s.=0A>>>>>> =0A>>>>>> After some days, i looked my NFSv4.1 mount. At serv=
er start it=0A>>>>>> was=0A>>>>>> calm, but after 4 days, here is the top=
 stat...=0A>>>>>> =0A>>>>>> CPU: 0.0% user, 0.0% nice, 100% system, 0.0% =
interrupt, 0.0%=0A>>>>>> idle=0A>>>>>> =0A>>>>>> Definitively i think it'=
s a problem on client side. What can i=0A>>>>>> look=0A>>>>>> into runnin=
g kernel to resolve this issue ?=0A>>>>> =0A>>>>> Well, I'd start with:=
=0A>>>>> # nfsstat -e -s=0A>>>>> - run repeatedly on the server (once eve=
ry N seconds in a loop).=0A>>>>> Then look at the output, comparing the c=
ounts and see which RPCs=0A>>>>> are being performed by the client(s). Yo=
u are looking for which=0A>>>>> RPCs are being done a lot. (If one RPC is=
 almost 100% of the=0A>>>>> load,=0A>>>>> then it might be a client/cachi=
ng issue for whatever that RPC is=0A>>>>> doing.)=0A>>>>> =0A>>>>> Also l=
ook at the Open/Lock counts near the end of the output.=0A>>>>> If the # =
of Opens/Locks is large, it may be possible to reduce=0A>>>>> the=0A>>>>>=
 CPU overheads by using larger hash tables.=0A>>>>> =0A>>>>> Then you nee=
d to profile the server kernel to see where the CPU=0A>>>>> is being used=
.=0A>>>>> Hopefully someone else can fill you in on how to do that, becau=
se=0A>>>>> I'll admit I don't know how to.=0A>>>>> Basically you are look=
ing to see if the CPU is being used in=0A>>>>> the NFS server code or ZFS=
.=0A>>>>> =0A>>>>> Good luck with it, rick=0A>>>>> =0A>>>>>> Regards,=0A>=
>>>>> =0A>>>>>> Lo=C3=AFc Blot,=0A>>>>>> UNIX Systems, Network and Securi=
ty Engineer=0A>>>>>> http://www.unix-experience.fr=0A>>>>>>; =0A>>>>>> 30 =
d=C3=A9cembre 2014 16:16 "Lo=C3=AFc Blot"=0A>>>>>> <loic.blot@unix-experi=
ence.fr>=0A>>>>>> a=0A>>>>>> =C3=A9crit:=0A>>>>>>> Hi Rick,=0A>>>>>>> i u=
pgraded my jail host from FreeBSD 9.3 to 10.1 to use NFS=0A>>>>>>> v4.1=
=0A>>>>>>> (mountoptions:=0A>>>>>>> rw,rsize=3D32768,wsize=3D32768,tcp,nf=
sv4,minorversion=3D1)=0A>>>>>>> =0A>>>>>>> Performance is quite stable bu=
t it's slow. Not as slow as=0A>>>>>>> before=0A>>>>>>> but slow... servic=
es was launched=0A>>>>>>> but no client are using them and system CPU % w=
as 10-50%.=0A>>>>>>> =0A>>>>>>> I don't see anything on NFSv4.1 server, i=
t's perfectly stable=0A>>>>>>> and=0A>>>>>>> functionnal.=0A>>>>>>> =0A>>=
>>>>> Regards,=0A>>>>>>> =0A>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>> UNIX System=
s, Network and Security Engineer=0A>>>>>>> http://www.unix-experience.fr=
=0A>>>>>>> =0A>>>>>>> 23 d=C3=A9cembre 2014 00:20 "Rick Macklem" <rmackle=
m@uoguelph.ca> a=0A>>>>>>> =C3=A9crit:=0A>>>>>>> =0A>>>>>>>> Loic Blot wr=
ote:=0A>>>>>>>> =0A>>>>>>>>> Hi,=0A>>>>>>>>> =0A>>>>>>>>> To clarify beca=
use of our exchanges. Here are the current=0A>>>>>>>>> sysctl=0A>>>>>>>>>=
 options for server:=0A>>>>>>>>> =0A>>>>>>>>> vfs.nfsd.enable_nobodycheck=
=3D0=0A>>>>>>>>> vfs.nfsd.enable_nogroupcheck=3D0=0A>>>>>>>>> =0A>>>>>>>>=
> vfs.nfsd.maxthreads=3D200=0A>>>>>>>>> vfs.nfsd.tcphighwater=3D10000=0A>=
>>>>>>>> vfs.nfsd.tcpcachetimeo=3D300=0A>>>>>>>>> vfs.nfsd.server_min_nfs=
vers=3D4=0A>>>>>>>>> =0A>>>>>>>>> kern.maxvnodes=3D10000000=0A>>>>>>>>> k=
ern.ipc.maxsockbuf=3D4194304=0A>>>>>>>>> net.inet.tcp.sendbuf_max=3D41943=
04=0A>>>>>>>>> net.inet.tcp.recvbuf_max=3D4194304=0A>>>>>>>>> =0A>>>>>>>>=
> vfs.lookup_shared=3D0=0A>>>>>>>>> =0A>>>>>>>>> Regards,=0A>>>>>>>>> =0A=
>>>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>>>> UNIX Systems, Network and Security =
Engineer=0A>>>>>>>>> http://www.unix-experience.fr=0A>>>>>>>>>; =0A>>>>>>>=
>> 22 d=C3=A9cembre 2014 09:42 "Lo=C3=AFc Blot"=0A>>>>>>>>> <loic.blot@un=
ix-experience.fr>=0A>>>>>>>>> a=0A>>>>>>>>> =C3=A9crit:=0A>>>>>>>>> =0A>>=
>>>>>>> Hi Rick,=0A>>>>>>>>> my 5 jails runs this weekend and now i have =
some stats on=0A>>>>>>>>> this=0A>>>>>>>>> monday.=0A>>>>>>>>> =0A>>>>>>>=
>> Hopefully deadlock was fixed, yeah, but everything isn't good=0A>>>>>>=
>>> :(=0A>>>>>>>>> =0A>>>>>>>>> On NFSv4 server (FreeBSD 10.1) system use=
s 35% CPU=0A>>>>>>>>> =0A>>>>>>>>> As i can see this is because of nfsd:=
=0A>>>>>>>>> =0A>>>>>>>>> 918 root 96 20 0 12352K 3372K rpcsvc 6 51.4H=0A=
>>>>>>>>> 273.68% nfsd: server (nfsd)=0A>>>>>>>>> =0A>>>>>>>>> If i look =
at dmesg i see:=0A>>>>>>>>> nfsd server cache flooded, try increasing=0A>=
>>>>>>>> vfs.nfsd.tcphighwater=0A>>>>>>>> =0A>>>>>>>> Well, you have a co=
uple of choices:=0A>>>>>>>> 1 - Use NFSv4.1 (add "minorversion=3D1" to yo=
ur mount options).=0A>>>>>>>> (NFSv4.1 avoids use of the DRC and instead =
uses something=0A>>>>>>>> called sessions. See below.)=0A>>>>>>>> OR=0A>>=
>>>>>> =0A>>>>>>>>> vfs.nfsd.tcphighwater was set to 10000, i increase it=
 to=0A>>>>>>>>> 15000=0A>>>>>>>> =0A>>>>>>>> 2 - Bump vfs.nfsd.tcphighwat=
er way up, until you no longer see=0A>>>>>>>> "nfs server cache flooded" =
messages. (I think Garrett Wollman=0A>>>>>>>> uses=0A>>>>>>>> 100000. (Yo=
u may still see quite a bit of CPU overheads.)=0A>>>>>>>> =0A>>>>>>>> OR=
=0A>>>>>>>> =0A>>>>>>>> 3 - Set vfs.nfsd.cachetcp=3D0 (which disables the=
 DRC and gets=0A>>>>>>>> rid=0A>>>>>>>> of the CPU overheads). However, t=
here is a risk of data=0A>>>>>>>> corruption=0A>>>>>>>> if you have a cli=
ent->server network partitioning of a=0A>>>>>>>> moderate=0A>>>>>>>> dura=
tion, because a non-idempotent RPC may get redone, becasue=0A>>>>>>>> the=
 client times out waiting for a reply. If a non-idempotent=0A>>>>>>>> RPC=
 gets done twice on the server, data corruption can happen.=0A>>>>>>>> (T=
he DRC provides improved correctness, but does add=0A>>>>>>>> overhead.)=
=0A>>>>>>>> =0A>>>>>>>> If #1 works for you, it is the preferred solution=
, since=0A>>>>>>>> Sessions=0A>>>>>>>> in NFSv4.1 solves the correctness =
problem in a good, space=0A>>>>>>>> bound=0A>>>>>>>> way. A session basic=
ally has N (usually 32 or 64) slots and=0A>>>>>>>> only=0A>>>>>>>> allows=
 one outstanding RPC/slot. As such, it can cache the=0A>>>>>>>> previous=
=0A>>>>>>>> reply for each slot (32 or 64 of them) and guarantee "exactly=
=0A>>>>>>>> once"=0A>>>>>>>> RPC semantics.=0A>>>>>>>> =0A>>>>>>>> rick=
=0A>>>>>>>> =0A>>>>>>>>> Here is 'nfsstat -s' output:=0A>>>>>>>>> =0A>>>>=
>>>>> Server Info:=0A>>>>>>>>> Getattr Setattr Lookup Readlink Read Write=
 Create=0A>>>>>>>>> Remove=0A>>>>>>>>> 12600652 1812 2501097 156 1386423 =
1983729 123=0A>>>>>>>>> 162067=0A>>>>>>>>> Rename Link Symlink Mkdir Rmdi=
r Readdir RdirPlus=0A>>>>>>>>> Access=0A>>>>>>>>> 36762 9 0 0 0 3147 0=0A=
>>>>>>>>> 623524=0A>>>>>>>>> Mknod Fsstat Fsinfo PathConf Commit=0A>>>>>>=
>>> 0 0 0 0 328117=0A>>>>>>>>> Server Ret-Failed=0A>>>>>>>>> 0=0A>>>>>>>>=
> Server Faults=0A>>>>>>>>> 0=0A>>>>>>>>> Server Cache Stats:=0A>>>>>>>>>=
 Inprog Idem Non-idem Misses=0A>>>>>>>>> 0 0 0 12635512=0A>>>>>>>>> Serve=
r Write Gathering:=0A>>>>>>>>> WriteOps WriteRPC Opsaved=0A>>>>>>>>> 1983=
729 1983729 0=0A>>>>>>>>> =0A>>>>>>>>> And here is 'procstat -kk' for nfs=
d (server)=0A>>>>>>>>> =0A>>>>>>>>> 918 100528 nfsd nfsd: master mi_switc=
h+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10=0A=
>>>>>>>>> _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1=0A>>>>>>>>> =
svc_run+0x1de=0A>>>>>>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+=
0x9c=0A>>>>>>>>> amd64_syscall+0x351 Xfast_syscall+0xfb=0A>>>>>>>>> 918 1=
00568 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+=
0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_r=
un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tr=
ampoline+0xe=0A>>>>>>>>> 918 100569 nfsd nfsd: service mi_switch+0xe1=0A>=
>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wa=
it_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork=
_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100570 nfsd nf=
sd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_w=
ait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0=
x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=
=0A>>>>>>>>> 918 100571 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sle=
epq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=
=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100572 nfsd nfsd: servic=
e mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0x=
f=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_t=
hread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>=
> 918 100573 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_s=
ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>=
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> =
fork_trampoline+0xe=0A>>>>>>>>> 918 100574 nfsd nfsd: service mi_switch+0=
xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>>=
 _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0=
xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100575 =
nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab s=
leepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_int=
ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoli=
ne+0xe=0A>>>>>>>>> 918 100576 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>=
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig=
+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100577 nfsd nfsd: se=
rvice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_si=
g+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>=
>>>>> 918 100578 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_cat=
ch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>=
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>=
>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100579 nfsd nfsd: service mi_swit=
ch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>=
>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_sta=
rt+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100=
580 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0x=
ab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run=
_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tram=
poline+0xe=0A>>>>>>>>> 918 100581 nfsd nfsd: service mi_switch+0xe1=0A>>>=
>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait=
_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e=
xit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100582 nfsd nfsd=
: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wai=
t_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x8=
7e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A=
>>>>>>>>> 918 100583 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq=
_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A=
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>=
>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100584 nfsd nfsd: service mi_=
switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>=
>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread=
_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918=
 100585 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signal=
s+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_=
trampoline+0xe=0A>>>>>>>>> 918 100586 nfsd nfsd: service mi_switch+0xe1=
=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _c=
v_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100587 nfs=
d nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab slee=
pq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_intern=
al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+=
0xe=0A>>>>>>>>> 918 100588 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> =
sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x=
16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9=
a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100589 nfsd nfsd: servi=
ce mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0=
xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_=
thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>=
>> 918 100590 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_=
signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>=
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>>=
 fork_trampoline+0xe=0A>>>>>>>>> 918 100591 nfsd nfsd: service mi_switch+=
0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>=
> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+=
0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100592=
 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab =
sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_in=
ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampol=
ine+0xe=0A>>>>>>>>> 918 100593 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>=
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_si=
g+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit=
+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100594 nfsd nfsd: s=
ervice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_s=
ig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e =
svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>=
>>>>>> 918 100595 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_ca=
tch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>=
>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>=
>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100596 nfsd nfsd: service mi_swi=
tch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>=
>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_st=
art+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 10=
0597 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_ru=
n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tra=
mpoline+0xe=0A>>>>>>>>> 918 100598 nfsd nfsd: service mi_switch+0xe1=0A>>=
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wai=
t_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_=
exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100599 nfsd nfs=
d: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wa=
it_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x=
87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=
=0A>>>>>>>>> 918 100600 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sle=
epq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=
=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100601 nfsd nfsd: servic=
e mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0x=
f=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_t=
hread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>=
> 918 100602 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_s=
ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>=
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> =
fork_trampoline+0xe=0A>>>>>>>>> 918 100603 nfsd nfsd: service mi_switch+0=
xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>>=
 _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0=
xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100604 =
nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab s=
leepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_int=
ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoli=
ne+0xe=0A>>>>>>>>> 918 100605 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>=
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig=
+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100606 nfsd nfsd: se=
rvice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_si=
g+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>=
>>>>> 918 100607 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_cat=
ch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>=
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>=
>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100608 nfsd nfsd: service mi_swit=
ch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>=
>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_sta=
rt+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100=
609 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0x=
ab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run=
_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tram=
poline+0xe=0A>>>>>>>>> 918 100610 nfsd nfsd: service mi_switch+0xe1=0A>>>=
>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait=
_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e=
xit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100611 nfsd nfsd=
: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wai=
t_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x8=
7e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A=
>>>>>>>>> 918 100612 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq=
_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A=
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>=
>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100613 nfsd nfsd: service mi_=
switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>=
>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread=
_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918=
 100614 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signal=
s+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_=
trampoline+0xe=0A>>>>>>>>> 918 100615 nfsd nfsd: service mi_switch+0xe1=
=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _c=
v_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100616 nfs=
d nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab slee=
pq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_intern=
al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+=
0xe=0A>>>>>>>>> 918 100617 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> =
sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x=
16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9=
a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100618 nfsd nfsd: servi=
ce mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0=
xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_=
thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>=
>> 918 100619 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_=
signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>=
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>>=
 fork_trampoline+0xe=0A>>>>>>>>> 918 100620 nfsd nfsd: service mi_switch+=
0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>=
> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+=
0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100621=
 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab =
sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_in=
ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampol=
ine+0xe=0A>>>>>>>>> 918 100622 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>=
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_si=
g+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit=
+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100623 nfsd nfsd: s=
ervice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_s=
ig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e =
svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>=
>>>>>> 918 100624 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_ca=
tch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>=
>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>=
>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100625 nfsd nfsd: service mi_swi=
tch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>=
>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_st=
art+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 10=
0626 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_ru=
n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tra=
mpoline+0xe=0A>>>>>>>>> 918 100627 nfsd nfsd: service mi_switch+0xe1=0A>>=
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wai=
t_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_=
exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100628 nfsd nfs=
d: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wa=
it_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x=
87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=
=0A>>>>>>>>> 918 100629 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sle=
epq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=
=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100630 nfsd nfsd: servic=
e mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0x=
f=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_t=
hread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>=
> 918 100631 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_s=
ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>=
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> =
fork_trampoline+0xe=0A>>>>>>>>> 918 100632 nfsd nfsd: service mi_switch+0=
xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>>=
 _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0=
xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100633 =
nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab s=
leepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_int=
ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoli=
ne+0xe=0A>>>>>>>>> 918 100634 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>=
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig=
+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100635 nfsd nfsd: se=
rvice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_si=
g+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>=
>>>>> 918 100636 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_cat=
ch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>=
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>=
>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100637 nfsd nfsd: service mi_swit=
ch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>=
>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_sta=
rt+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100=
638 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0x=
ab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run=
_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tram=
poline+0xe=0A>>>>>>>>> 918 100639 nfsd nfsd: service mi_switch+0xe1=0A>>>=
>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait=
_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e=
xit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100640 nfsd nfsd=
: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wai=
t_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x8=
7e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A=
>>>>>>>>> 918 100641 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq=
_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A=
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>=
>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100642 nfsd nfsd: service mi_=
switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>=
>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread=
_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918=
 100643 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signal=
s+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_=
trampoline+0xe=0A>>>>>>>>> 918 100644 nfsd nfsd: service mi_switch+0xe1=
=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _c=
v_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100645 nfs=
d nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab slee=
pq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_intern=
al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+=
0xe=0A>>>>>>>>> 918 100646 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> =
sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x=
16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9=
a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100647 nfsd nfsd: servi=
ce mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0=
xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_=
thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>=
>> 918 100648 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_=
signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>=
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>>=
 fork_trampoline+0xe=0A>>>>>>>>> 918 100649 nfsd nfsd: service mi_switch+=
0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>=
> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+=
0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100650=
 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab =
sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_in=
ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampol=
ine+0xe=0A>>>>>>>>> 918 100651 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>=
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_si=
g+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit=
+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100652 nfsd nfsd: s=
ervice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_s=
ig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e =
svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>=
>>>>>> 918 100653 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_ca=
tch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>=
>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>=
>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100654 nfsd nfsd: service mi_swi=
tch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>=
>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_st=
art+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 10=
0655 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_ru=
n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tra=
mpoline+0xe=0A>>>>>>>>> 918 100656 nfsd nfsd: service mi_switch+0xe1=0A>>=
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wai=
t_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_=
exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100657 nfsd nfs=
d: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wa=
it_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x=
87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=
=0A>>>>>>>>> 918 100658 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sle=
epq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=
=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100659 nfsd nfsd: servic=
e mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0x=
f=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_t=
hread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>=
> 918 100660 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_s=
ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>=
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> =
fork_trampoline+0xe=0A>>>>>>>>> 918 100661 nfsd nfsd: service mi_switch+0=
xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>>=
 _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0=
xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 918 100662 =
nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab s=
leepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_int=
ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoli=
ne+0xe=0A>>>>>>>>> ---=0A>>>>>>>>> =0A>>>>>>>>> Now if we look at client =
(FreeBSD 9.3)=0A>>>>>>>>> =0A>>>>>>>>> We see system was very busy and do=
 many and many interrupts=0A>>>>>>>>> =0A>>>>>>>>> CPU: 0.0% user, 0.0% n=
ice, 37.8% system, 51.2% interrupt,=0A>>>>>>>>> 11.0%=0A>>>>>>>>> idle=0A=
>>>>>>>>> =0A>>>>>>>>> A look at process list shows that there are many s=
endmail=0A>>>>>>>>> process=0A>>>>>>>>> in=0A>>>>>>>>> state nfstry=0A>>>=
>>>>>> =0A>>>>>>>>> nfstry 18 32:27 0.88% sendmail: Queue runner@00:30:00=
 for=0A>>>>>>>>> /var/spool/clientm=0A>>>>>>>>> =0A>>>>>>>>> Here is 'nfs=
stat -c' output:=0A>>>>>>>>> =0A>>>>>>>>> Client Info:=0A>>>>>>>>> Rpc Co=
unts:=0A>>>>>>>>> Getattr Setattr Lookup Readlink Read Write Create=0A>>>=
>>>>>> Remove=0A>>>>>>>>> 1051347 1724 2494481 118 903902 1901285 162676=
=0A>>>>>>>>> 161899=0A>>>>>>>>> Rename Link Symlink Mkdir Rmdir Readdir R=
dirPlus=0A>>>>>>>>> Access=0A>>>>>>>>> 36744 2 0 114 40 3131 0=0A>>>>>>>>=
> 544136=0A>>>>>>>>> Mknod Fsstat Fsinfo PathConf Commit=0A>>>>>>>>> 9 0 =
0 0 245821=0A>>>>>>>>> Rpc Info:=0A>>>>>>>>> TimedOut Invalid X Replies R=
etries Requests=0A>>>>>>>>> 0 0 0 0 8356557=0A>>>>>>>>> Cache Info:=0A>>>=
>>>>>> Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits=0A>>>=
>>>>>> Misses=0A>>>>>>>>> 108754455 491475 54229224 2437229 46814561 8217=
23 5132123=0A>>>>>>>>> 1871871=0A>>>>>>>>> BioRLHits Misses BioD Hits Mis=
ses DirE Hits Misses Accs Hits=0A>>>>>>>>> Misses=0A>>>>>>>>> 144035 118 =
53736 2753 27813 1 57238839=0A>>>>>>>>> 544205=0A>>>>>>>>> =0A>>>>>>>>> I=
f you need more things, tell me, i let the PoC in this=0A>>>>>>>>> state.=
=0A>>>>>>>>> =0A>>>>>>>>> Thanks=0A>>>>>>>>> =0A>>>>>>>>> Regards,=0A>>>>=
>>>>> =0A>>>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>>>> UNIX Systems, Network and =
Security Engineer=0A>>>>>>>>> http://www.unix-experience.fr=0A>>>>>>>>>; =
=0A>>>>>>>>> 21 d=C3=A9cembre 2014 01:33 "Rick Macklem" <rmacklem@uoguelp=
h.ca>=0A>>>>>>>>> a=0A>>>>>>>>> =C3=A9crit:=0A>>>>>>>>> =0A>>>>>>>>> Loic=
 Blot wrote:=0A>>>>>>>>> =0A>>>>>>>>>> Hi Rick,=0A>>>>>>>>>> ok, i don't =
need locallocks, i haven't understand option was=0A>>>>>>>>>> for=0A>>>>>=
>>>>> that=0A>>>>>>>>>> usage, i removed it.=0A>>>>>>>>>> I do more tests=
 on monday.=0A>>>>>>>>>> Thanks for the deadlock fix, for other people :)=
=0A>>>>>>>>> =0A>>>>>>>>> Good. Please let us know if running with=0A>>>>=
>>>>> vfs.nfsd.enable_locallocks=3D0=0A>>>>>>>>> gets rid of the deadlock=
s? (I think it fixes the one you=0A>>>>>>>>> saw.)=0A>>>>>>>>> =0A>>>>>>>=
>> On the performance side, you might also want to try different=0A>>>>>>=
>>> values=0A>>>>>>>>> of=0A>>>>>>>>> readahead, if the Linux client has =
such a mount option. (With=0A>>>>>>>>> the=0A>>>>>>>>> NFSv4-ZFS sequenti=
al vs random I/O heuristic, I have no idea=0A>>>>>>>>> what=0A>>>>>>>>> t=
he=0A>>>>>>>>> optimal readahead value would be.)=0A>>>>>>>>> =0A>>>>>>>>=
> Good luck with it and please let us know how it goes, rick=0A>>>>>>>>> =
ps: I now have a patch to fix the deadlock when=0A>>>>>>>>> vfs.nfsd.enab=
le_locallocks=3D1=0A>>>>>>>>> is set. I'll post it for anyone who is inte=
rested after I put=0A>>>>>>>>> it=0A>>>>>>>>> through some testing.=0A>>>=
>>>>>> =0A>>>>>>>>> --=0A>>>>>>>>> Best regards,=0A>>>>>>>>> Lo=C3=AFc BL=
OT,=0A>>>>>>>>> UNIX systems, security and network engineer=0A>>>>>>>>> h=
ttp://www.unix-experience.fr=0A>>>>>>>>> =0A>>>>>>>>> Le jeudi 18 d=C3=A9=
cembre 2014 =C3=A0 19:46 -0500, Rick Macklem a =C3=A9crit=0A>>>>>>>>> :=
=0A>>>>>>>>> =0A>>>>>>>>> Loic Blot wrote:=0A>>>>>>>>>> Hi rick,=0A>>>>>>=
>>>> i tried to start a LXC container on Debian Squeeze from my=0A>>>>>>>=
>>> freebsd=0A>>>>>>>>>> ZFS+NFSv4 server and i also have a deadlock on n=
fsd=0A>>>>>>>>>> (vfs.lookup_shared=3D0). Deadlock procs each time i laun=
ch a=0A>>>>>>>>>> squeeze=0A>>>>>>>>>> container, it seems (3 tries, 3 fa=
ils).=0A>>>>>>>>> =0A>>>>>>>>> Well, I`ll take a look at this `procstat -=
kk`, but the only=0A>>>>>>>>> thing=0A>>>>>>>>> I`ve seen posted w.r.t. a=
voiding deadlocks in ZFS is to not=0A>>>>>>>>> use=0A>>>>>>>>> nullfs. (I=
 have no idea if you are using any nullfs mounts,=0A>>>>>>>>> but=0A>>>>>=
>>>> if so, try getting rid of them.)=0A>>>>>>>>> =0A>>>>>>>>> Here`s a h=
igh level post about the ZFS and vnode locking=0A>>>>>>>>> problem,=0A>>>=
>>>>>> but there is no patch available, as far as I know.=0A>>>>>>>>> =0A=
>>>>>>>>> http://docs.FreeBSD.org/cgi/mid.cgi?54739F41.8030407=0A>>>>>>>>=
> =0A>>>>>>>>> rick=0A>>>>>>>>> =0A>>>>>>>>> 921 - D 0:00.02 nfsd: server=
 (nfsd)=0A>>>>>>>>> =0A>>>>>>>>> Here is the procstat -kk=0A>>>>>>>>> =0A=
>>>>>>>>> PID TID COMM TDNAME KSTACK=0A>>>>>>>>> 921 100538 nfsd nfsd: ma=
ster mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_=
args+0xc9e=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=
=0A>>>>>>>>> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14a=
d=0A>>>>>>>>> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554=
=0A>>>>>>>>> svc_run_internal+0xc77 svc_run+0x1de nfsrvd_nfsd+0x1ca=0A>>>=
>>>>>> nfssvc_nfsd+0x107 sys_nfssvc+0x9c=0A>>>>>>>>> 921 100572 nfsd nfsd=
: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wai=
t_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x8=
7e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A=
>>>>>>>>> 921 100573 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq=
_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A=
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>=
>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100574 nfsd nfsd: service mi_=
switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>=
>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread=
_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921=
 100575 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signal=
s+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_=
trampoline+0xe=0A>>>>>>>>> 921 100576 nfsd nfsd: service mi_switch+0xe1=
=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _c=
v_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100577 nfs=
d nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab slee=
pq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_intern=
al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+=
0xe=0A>>>>>>>>> 921 100578 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> =
sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x=
16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9=
a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100579 nfsd nfsd: servi=
ce mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0=
xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_=
thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>=
>> 921 100580 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_=
signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>=
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>>=
 fork_trampoline+0xe=0A>>>>>>>>> 921 100581 nfsd nfsd: service mi_switch+=
0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>=
> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+=
0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100582=
 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab =
sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_in=
ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampol=
ine+0xe=0A>>>>>>>>> 921 100583 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>=
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_si=
g+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit=
+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100584 nfsd nfsd: s=
ervice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_s=
ig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e =
svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>=
>>>>>> 921 100585 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_ca=
tch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>=
>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>=
>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100586 nfsd nfsd: service mi_swi=
tch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>=
>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_st=
art+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 10=
0587 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_ru=
n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tra=
mpoline+0xe=0A>>>>>>>>> 921 100588 nfsd nfsd: service mi_switch+0xe1=0A>>=
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wai=
t_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_=
exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100589 nfsd nfs=
d: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wa=
it_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x=
87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=
=0A>>>>>>>>> 921 100590 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sle=
epq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=
=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100591 nfsd nfsd: servic=
e mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0x=
f=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_t=
hread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>=
> 921 100592 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_s=
ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>=
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> =
fork_trampoline+0xe=0A>>>>>>>>> 921 100593 nfsd nfsd: service mi_switch+0=
xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>>=
 _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0=
xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100594 =
nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab s=
leepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_int=
ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoli=
ne+0xe=0A>>>>>>>>> 921 100595 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>=
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig=
+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100596 nfsd nfsd: se=
rvice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_si=
g+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>=
>>>>> 921 100597 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_cat=
ch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>=
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>=
>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100598 nfsd nfsd: service mi_swit=
ch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>=
>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_sta=
rt+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100=
599 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0x=
ab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run=
_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tram=
poline+0xe=0A>>>>>>>>> 921 100600 nfsd nfsd: service mi_switch+0xe1=0A>>>=
>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait=
_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e=
xit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100601 nfsd nfsd=
: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wai=
t_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x8=
7e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A=
>>>>>>>>> 921 100602 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq=
_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A=
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>=
>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100603 nfsd nfsd: service mi_=
switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>=
>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread=
_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921=
 100604 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signal=
s+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_=
trampoline+0xe=0A>>>>>>>>> 921 100605 nfsd nfsd: service mi_switch+0xe1=
=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _c=
v_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100606 nfs=
d nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab slee=
pq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_intern=
al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+=
0xe=0A>>>>>>>>> 921 100607 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> =
sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x=
16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9=
a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100608 nfsd nfsd: servi=
ce mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0=
xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_=
thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>=
>> 921 100609 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_=
signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>=
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>>=
 fork_trampoline+0xe=0A>>>>>>>>> 921 100610 nfsd nfsd: service mi_switch+=
0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>=
> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+=
0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100611=
 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab =
sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_in=
ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampol=
ine+0xe=0A>>>>>>>>> 921 100612 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>=
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_si=
g+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit=
+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100613 nfsd nfsd: s=
ervice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_s=
ig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e =
svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>=
>>>>>> 921 100614 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_ca=
tch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>=
>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>=
>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100615 nfsd nfsd: service mi_swi=
tch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>=
>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_st=
art+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 10=
0616 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a _slee=
p+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>>>>> nfsrv_getlockfile+0x17=
9 nfsrv_lockctrl+0x21f=0A>>>>>>>>> nfsrvd_lock+0x5b1=0A>>>>>>>>> nfsrvd_d=
orpc+0xec6 nfssvc_program+0x554=0A>>>>>>>>> svc_run_internal+0xc77=0A>>>>=
>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>=
 921 100617 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_si=
gnals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>>=
 svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> f=
ork_trampoline+0xe=0A>>>>>>>>> 921 100618 nfsd nfsd: service mi_switch+0x=
e1=0A>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x=
9b=0A>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554=0A>>>>>>>>> svc_ru=
n_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_tra=
mpoline+0xe=0A>>>>>>>>> 921 100619 nfsd nfsd: service mi_switch+0xe1=0A>>=
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wai=
t_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_=
exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100620 nfsd nfs=
d: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wa=
it_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x=
87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=
=0A>>>>>>>>> 921 100621 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sle=
epq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=
=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100622 nfsd nfsd: servic=
e mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0x=
f=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_t=
hread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>=
> 921 100623 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_s=
ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>=
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> =
fork_trampoline+0xe=0A>>>>>>>>> 921 100624 nfsd nfsd: service mi_switch+0=
xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>>=
 _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0=
xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100625 =
nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab s=
leepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_int=
ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoli=
ne+0xe=0A>>>>>>>>> 921 100626 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>=
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig=
+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100627 nfsd nfsd: se=
rvice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_si=
g+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>=
>>>>> 921 100628 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_cat=
ch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>=
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>=
>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100629 nfsd nfsd: service mi_swit=
ch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>=
>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_sta=
rt+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100=
630 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0x=
ab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run=
_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tram=
poline+0xe=0A>>>>>>>>> 921 100631 nfsd nfsd: service mi_switch+0xe1=0A>>>=
>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait=
_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e=
xit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100632 nfsd nfsd=
: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wai=
t_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x8=
7e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A=
>>>>>>>>> 921 100633 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq=
_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A=
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>=
>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100634 nfsd nfsd: service mi_=
switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>=
>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread=
_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921=
 100635 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signal=
s+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_=
trampoline+0xe=0A>>>>>>>>> 921 100636 nfsd nfsd: service mi_switch+0xe1=
=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _c=
v_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100637 nfs=
d nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab slee=
pq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_intern=
al+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+=
0xe=0A>>>>>>>>> 921 100638 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> =
sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x=
16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9=
a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100639 nfsd nfsd: servi=
ce mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0=
xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_=
thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>=
>> 921 100640 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_=
signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>=
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>>=
 fork_trampoline+0xe=0A>>>>>>>>> 921 100641 nfsd nfsd: service mi_switch+=
0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>=
> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+=
0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100642=
 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab =
sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_in=
ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampol=
ine+0xe=0A>>>>>>>>> 921 100643 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>=
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_si=
g+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit=
+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100644 nfsd nfsd: s=
ervice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_s=
ig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e =
svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>=
>>>>>> 921 100645 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_ca=
tch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>=
>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>=
>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100646 nfsd nfsd: service mi_swi=
tch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>=
>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_st=
art+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 10=
0647 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0=
xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_ru=
n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tra=
mpoline+0xe=0A>>>>>>>>> 921 100648 nfsd nfsd: service mi_switch+0xe1=0A>>=
>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wai=
t_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_=
exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100649 nfsd nfs=
d: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wa=
it_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x=
87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=
=0A>>>>>>>>> 921 100650 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sle=
epq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=
=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=
=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100651 nfsd nfsd: servic=
e mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0x=
f=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_t=
hread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>=
> 921 100652 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_s=
ignals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>=
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> =
fork_trampoline+0xe=0A>>>>>>>>> 921 100653 nfsd nfsd: service mi_switch+0=
xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>>=
 _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0=
xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100654 =
nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab s=
leepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_int=
ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoli=
ne+0xe=0A>>>>>>>>> 921 100655 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>=
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig=
+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100656 nfsd nfsd: se=
rvice mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_si=
g+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e s=
vc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>=
>>>>> 921 100657 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_cat=
ch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>=
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>=
>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100658 nfsd nfsd: service mi_swit=
ch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>=
>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_sta=
rt+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100=
659 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0x=
ab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run=
_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_tram=
poline+0xe=0A>>>>>>>>> 921 100660 nfsd nfsd: service mi_switch+0xe1=0A>>>=
>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait=
_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_e=
xit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100661 nfsd nfsd=
: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wai=
t_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x8=
7e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A=
>>>>>>>>> 921 100662 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq=
_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A=
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>=
>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100663 nfsd nfsd: service mi_=
switch+0xe1=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>=
>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread=
_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921=
 100664 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_catch_signal=
s+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>> svc=
_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_=
trampoline+0xe=0A>>>>>>>>> 921 100665 nfsd nfsd: service mi_switch+0xe1=
=0A>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>> _c=
v_wait_sig+0x16a=0A>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb =
fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>> 921 100666 nfs=
d nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 =
nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>>>>> nfsrv_setclient+0xbd nfsrvd_se=
tclientid+0x3c8=0A>>>>>>>>> nfsrvd_dorpc+0xc76=0A>>>>>>>>> nfssvc_program=
+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>=
> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> =0A>>>>>>>>> Regards,=0A=
>>>>>>>>> =0A>>>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>>>> UNIX Systems, Network =
and Security Engineer=0A>>>>>>>>> http://www.unix-experience.fr=0A>>>>>>>=
>> =0A>>>>>>>>> 15 d=C3=A9cembre 2014 15:18 "Rick Macklem" <rmacklem@uogu=
elph.ca>=0A>>>>>>>>> a=0A>>>>>>>>> =C3=A9crit:=0A>>>>>>>>> =0A>>>>>>>>> L=
oic Blot wrote:=0A>>>>>>>>> =0A>>>>>>>>>> For more informations, here is =
procstat -kk on nfsd, if you=0A>>>>>>>>>> need=0A>>>>>>>>>> more=0A>>>>>>=
>>>> hot datas, tell me.=0A>>>>>>>>>> =0A>>>>>>>>>> Regards, PID TID COMM=
 TDNAME KSTACK=0A>>>>>>>>>> 918 100529 nfsd nfsd: master mi_switch+0xe1=
=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>=
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_=
fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorp=
c+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_run+=
0x1de=0A>>>>>>>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c=0A=
>>>>>>>>>> amd64_syscall+0x351=0A>>>>>>>>> =0A>>>>>>>>> Well, most of the=
 threads are stuck like this one, waiting=0A>>>>>>>>> for=0A>>>>>>>>> a=
=0A>>>>>>>>> vnode=0A>>>>>>>>> lock in ZFS. All of them appear to be in z=
fs_fhtovp().=0A>>>>>>>>> I`m not a ZFS guy, so I can`t help much. I`ll tr=
y changing=0A>>>>>>>>> the=0A>>>>>>>>> subject line=0A>>>>>>>>> to includ=
e ZFS vnode lock, so maybe the ZFS guys will take a=0A>>>>>>>>> look.=0A>=
>>>>>>>> =0A>>>>>>>>> The only thing I`ve seen suggested is trying:=0A>>>=
>>>>>> sysctl vfs.lookup_shared=3D0=0A>>>>>>>>> to disable shared vop_loo=
kup()s. Apparently zfs_lookup()=0A>>>>>>>>> doesn`t=0A>>>>>>>>> obey the =
vnode locking rules for lookup and rename, according=0A>>>>>>>>> to=0A>>>=
>>>>>> the posting I saw.=0A>>>>>>>>> =0A>>>>>>>>> I`ve added a couple of=
 comments about the other threads=0A>>>>>>>>> below,=0A>>>>>>>>> but=0A>>=
>>>>>>> they are all either waiting for an RPC request or waiting for=0A>=
>>>>>>>> the=0A>>>>>>>>> threads stuck on the ZFS vnode lock to complete.=
=0A>>>>>>>>> =0A>>>>>>>>> rick=0A>>>>>>>>> =0A>>>>>>>>>> 918 100564 nfsd =
nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_catch_signals+0xab sleep=
q_wait_sig+0xf=0A>>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>>> svc_run_inter=
nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>>> fork_trampolin=
e+0xe=0A>>>>>>>>> =0A>>>>>>>>> Fyi, this thread is just waiting for an RP=
C to arrive.=0A>>>>>>>>> (Normal)=0A>>>>>>>>> =0A>>>>>>>>>> 918 100565 nf=
sd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_catch_signals+0xab sl=
eepq_wait_sig+0xf=0A>>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>>> svc_run_in=
ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>>> fork_trampo=
line+0xe=0A>>>>>>>>>> 918 100566 nfsd nfsd: service mi_switch+0xe1=0A>>>>=
>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>>> _cv_wai=
t_sig+0x16a=0A>>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork=
_exit+0x9a=0A>>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>>> 918 100567 nfsd =
nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_catch_signals+0xab sleep=
q_wait_sig+0xf=0A>>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>>> svc_run_inter=
nal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>>> fork_trampolin=
e+0xe=0A>>>>>>>>>> 918 100568 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>=
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>>> _cv_wait_s=
ig+0x16a=0A>>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_ex=
it+0x9a=0A>>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>>> 918 100569 nfsd nfs=
d: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_catch_signals+0xab sleepq_w=
ait_sig+0xf=0A>>>>>>>>>> _cv_wait_sig+0x16a=0A>>>>>>>>>> svc_run_internal=
+0x87e svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>>> fork_trampoline+0=
xe=0A>>>>>>>>>> 918 100570 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>>=
 sleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A>>>>>>>>>> _cv_wait_sig+=
0x16a=0A>>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+=
0x9a=0A>>>>>>>>>> fork_trampoline+0xe=0A>>>>>>>>>> 918 100571 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsle=
ep+0x66 nfsv4_lock+0x9b=0A>>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x=
554=0A>>>>>>>>>> svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=
 fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100572 nfsd nfsd: se=
rvice mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep=
+0x66 nfsv4_lock+0x9b=0A>>>>>>>>>> nfsrv_setclient+0xbd nfsrvd_setclienti=
d+0x3c8=0A>>>>>>>>>> nfsrvd_dorpc+0xc76=0A>>>>>>>>>> nfssvc_program+0x554=
 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fo=
rk_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> =0A>>>>>>>>> This one (and a=
 few others) are waiting for the nfsv4_lock.=0A>>>>>>>>> This=0A>>>>>>>>>=
 happens=0A>>>>>>>>> because other threads are stuck with RPCs in progres=
s. (ie.=0A>>>>>>>>> The=0A>>>>>>>>> ones=0A>>>>>>>>> waiting on the vnode=
 lock in zfs_fhtovp().)=0A>>>>>>>>> For these, the RPC needs to lock out =
other threads to do the=0A>>>>>>>>> operation,=0A>>>>>>>>> so it waits fo=
r the nfsv4_lock() which can exclusively lock=0A>>>>>>>>> the=0A>>>>>>>>>=
 NFSv4=0A>>>>>>>>> data structures once all other nfsd threads complete t=
heir=0A>>>>>>>>> RPCs=0A>>>>>>>>> in=0A>>>>>>>>> progress.=0A>>>>>>>>> =
=0A>>>>>>>>>> 918 100573 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> s=
leepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>>>>>> =
nfsrvd_dorpc+0x316 nfssvc_program+0x554=0A>>>>>>>>>> svc_run_internal+0xc=
77=0A>>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=
=0A>>>>>>>>> =0A>>>>>>>>> Same as above.=0A>>>>>>>>> =0A>>>>>>>>>> 918 10=
0574 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a slee=
plk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_AP=
V+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fh=
tovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program=
+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>=
>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100575 nfsd nfsd:=
 service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __loc=
kmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock=
+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_=
fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_=
internal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0=
x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100576 nfsd nfsd: service mi_swi=
tch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=
=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>=
>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfs=
rvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=
=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_tramp=
oline+0xe=0A>>>>>>>>>> 918 100577 nfsd nfsd: service mi_switch+0xe1=0A>>>=
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> =
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp=
+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x91=
7=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> s=
vc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>=
>>>>>>> 918 100578 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_=
wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3=
c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>=
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> =
nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start=
+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100=
579 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleep=
lk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV=
+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fht=
ovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+=
0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>=
>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100580 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lock=
mgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+=
0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_f=
htovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_i=
nternal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x=
9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100581 nfsd nfsd: service mi_swit=
ch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=
=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>=
>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfs=
rvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=
=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_tramp=
oline+0xe=0A>>>>>>>>>> 918 100582 nfsd nfsd: service mi_switch+0xe1=0A>>>=
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> =
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp=
+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x91=
7=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> s=
vc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>=
>>>>>>> 918 100583 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_=
wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3=
c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>=
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> =
nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start=
+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100=
584 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleep=
lk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV=
+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fht=
ovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+=
0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>=
>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100585 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lock=
mgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+=
0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_f=
htovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_i=
nternal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x=
9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100586 nfsd nfsd: service mi_swit=
ch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=
=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>=
>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfs=
rvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=
=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_tramp=
oline+0xe=0A>>>>>>>>>> 918 100587 nfsd nfsd: service mi_switch+0xe1=0A>>>=
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> =
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp=
+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x91=
7=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> s=
vc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>=
>>>>>>> 918 100588 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_=
wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3=
c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>=
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> =
nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start=
+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100=
589 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleep=
lk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV=
+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fht=
ovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+=
0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>=
>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100590 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lock=
mgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+=
0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_f=
htovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_i=
nternal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x=
9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100591 nfsd nfsd: service mi_swit=
ch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=
=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>=
>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfs=
rvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=
=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_tramp=
oline+0xe=0A>>>>>>>>>> 918 100592 nfsd nfsd: service mi_switch+0xe1=0A>>>=
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> =
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp=
+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x91=
7=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> s=
vc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>=
>>>>>>> 918 100593 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_=
wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3=
c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>=
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> =
nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start=
+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100=
594 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleep=
lk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV=
+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fht=
ovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+=
0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>=
>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100595 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lock=
mgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+=
0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_f=
htovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_i=
nternal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x=
9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100596 nfsd nfsd: service mi_swit=
ch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=
=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>=
>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfs=
rvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=
=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_tramp=
oline+0xe=0A>>>>>>>>>> 918 100597 nfsd nfsd: service mi_switch+0xe1=0A>>>=
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> =
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp=
+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x91=
7=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> s=
vc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>=
>>>>>>> 918 100598 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_=
wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3=
c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>=
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> =
nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start=
+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100=
599 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleep=
lk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV=
+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fht=
ovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+=
0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>=
>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100600 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lock=
mgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+=
0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_f=
htovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_i=
nternal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x=
9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100601 nfsd nfsd: service mi_swit=
ch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=
=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>=
>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfs=
rvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=
=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_tramp=
oline+0xe=0A>>>>>>>>>> 918 100602 nfsd nfsd: service mi_switch+0xe1=0A>>>=
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> =
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp=
+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x91=
7=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> s=
vc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>=
>>>>>>> 918 100603 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_=
wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3=
c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>=
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> =
nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start=
+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100=
604 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleep=
lk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV=
+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fht=
ovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+=
0x554 svc_run_internal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>=
>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100605 nfsd nfsd: =
service mi_switch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lock=
mgr_args+0x902=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+=
0x43=0A>>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_f=
htovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_i=
nternal+0xc77=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x=
9a fork_trampoline+0xe=0A>>>>>>>>>> 918 100606 nfsd nfsd: service mi_swit=
ch+0xe1=0A>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=
=0A>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>=
>>> zfs_fhtovp+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfs=
rvd_dorpc+0x917=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=
=0A>>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_tramp=
oline+0xe=0A>>>>>>>>>> 918 100607 nfsd nfsd: service mi_switch+0xe1=0A>>>=
>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>> =
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>>> zfs_fhtovp=
+0x38d=0A>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x91=
7=0A>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>>> s=
vc_thread_start+0xb=0A>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>=
>>>>>> =0A>>>>>>>>> Lots more waiting for the ZFS vnode lock in zfs_fhtov=
p().=0A>>>>>>>>> =0A>>>>>>>>> 918 100608 nfsd nfsd: service mi_switch+0xe=
1=0A>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9=
b=0A>>>>>>>>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f=0A>>>>>>>>> nf=
srvd_lock+0x5b1=0A>>>>>>>>> nfsrvd_dorpc+0xec6 nfssvc_program+0x554=0A>>>=
>>>>>> svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb fork_exit+=
0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100609 nfsd nfsd: service mi_swi=
tch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=
=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>=
> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd=
_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>=
>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0=
xe=0A>>>>>>>>> 918 100610 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> s=
leepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e=0A>>>>>>>>> vop_stdloc=
k+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> nfsvno_advlock+0x119 =
nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad=0A>>>>>>>>> nfsrvd_locku+0x283 n=
fsrvd_dorpc+0xec6 nfssvc_program+0x554=0A>>>>>>>>> svc_run_internal+0xc77=
 svc_thread_start+0xb fork_exit+0x9a=0A>>>>>>>>> fork_trampoline+0xe=0A>>=
>>>>>>> 918 100611 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_w=
ait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>>>>> nfsrvd_d=
orpc+0x316 nfssvc_program+0x554=0A>>>>>>>>> svc_run_internal+0xc77=0A>>>>=
>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>>=
 918 100612 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3=
a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>>>>> nfsrvd_dorpc+0x=
316 nfssvc_program+0x554=0A>>>>>>>>> svc_run_internal+0xc77=0A>>>>>>>>> s=
vc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 10=
0613 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a _slee=
p+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>>>>> nfsrvd_dorpc+0x316 nfs=
svc_program+0x554=0A>>>>>>>>> svc_run_internal+0xc77=0A>>>>>>>>> svc_thre=
ad_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100614 nf=
sd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a _sleep+0x287=
 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_pro=
gram+0x554=0A>>>>>>>>> svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_star=
t+0xb fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100615 nfsd nfsd=
: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsl=
eep+0x66 nfsv4_lock+0x9b=0A>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x=
554=0A>>>>>>>>> svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb f=
ork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100616 nfsd nfsd: servi=
ce mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x6=
6 nfsv4_lock+0x9b=0A>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554=0A>=
>>>>>>>> svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb fork_exi=
t+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100617 nfsd nfsd: service mi_s=
witch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4=
_lock+0x9b=0A>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554=0A>>>>>>>>=
> svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a =
fork_trampoline+0xe=0A>>>>>>>>> 918 100618 nfsd nfsd: service mi_switch+0=
xe1=0A>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0=
x9b=0A>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554=0A>>>>>>>>> svc_r=
un_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_tr=
ampoline+0xe=0A>>>>>>>>> 918 100619 nfsd nfsd: service mi_switch+0xe1=0A>=
>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> =
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+=
0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=
=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_=
thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>=
>> 918 100620 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0=
x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_L=
OCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsv=
no_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_pro=
gram+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>=
>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100621 nfsd nfsd=
: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsl=
eep+0x66 nfsv4_lock+0x9b=0A>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x=
554=0A>>>>>>>>> svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb f=
ork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100622 nfsd nfsd: servi=
ce mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_ar=
gs+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A=
>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc=
8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0x=
c77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_tram=
poline+0xe=0A>>>>>>>>> 918 100623 nfsd nfsd: service mi_switch+0xe1=0A>>>=
>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A>>>=
>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554=0A>>>>>>>>> svc_run_intern=
al+0xc77=0A>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+=
0xe=0A>>>>>>>>> 918 100624 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> =
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlo=
ck+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>=
>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>=
>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_sta=
rt+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100=
625 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleepl=
k+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0=
xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+=
0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554=
 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork=
_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100626 nfsd nfsd: service =
mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+=
0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>=
>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 n=
fsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=
=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampol=
ine+0xe=0A>>>>>>>>> 918 100627 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>=
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_s=
tdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=
=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>=
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread=
_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918=
 100628 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sl=
eeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_A=
PV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fht=
ovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0=
x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> =
fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100629 nfsd nfsd: serv=
ice mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_a=
rgs+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=
=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+=
0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal=
+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_t=
rampoline+0xe=0A>>>>>>>>> 918 100630 nfsd nfsd: service mi_switch+0xe1=0A=
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>=
 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp=
+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=
=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_=
thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>=
>> 918 100631 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0=
x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_L=
OCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsv=
no_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_pro=
gram+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>=
>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100632 nfsd nfsd=
: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __loc=
kmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+=
0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fht=
ovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_inte=
rnal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fo=
rk_trampoline+0xe=0A>>>>>>>>> 918 100633 nfsd nfsd: service mi_switch+0xe=
1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>=
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fh=
tovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0=
x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> =
svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>=
>>>>>> 918 100634 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wa=
it+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> =
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc=
_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A=
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100635 nfsd =
nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d _=
_lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_l=
ock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd=
_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_=
internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9=
a fork_trampoline+0xe=0A>>>>>>>>> 918 100636 nfsd nfsd: service mi_switch=
+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>=
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zf=
s_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dor=
pc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>=
>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=
=0A>>>>>>>>> 918 100637 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sle=
epq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+=
0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>=
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> =
nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+=
0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100638=
 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0=
x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab=
 _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7=
c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 sv=
c_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_ex=
it+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100639 nfsd nfsd: service mi_=
switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x9=
02=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>=
>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsr=
vd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A=
>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline=
+0xe=0A>>>>>>>>> 918 100640 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>=
 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdl=
ock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A=
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>=
>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_st=
art+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 10=
0641 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleep=
lk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+=
0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp=
+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x55=
4 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> for=
k_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100642 nfsd nfsd: service=
 mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args=
+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>=
>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 =
nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc7=
7=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampo=
line+0xe=0A>>>>>>>>> 918 100643 nfsd nfsd: service mi_switch+0xe1=0A>>>>>=
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_=
stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38=
d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>=
>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_threa=
d_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 91=
8 100644 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a s=
leeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_=
APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fh=
tovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+=
0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>>=
 fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100645 nfsd nfsd: ser=
vice mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_=
args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=
=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+=
0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal=
+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_t=
rampoline+0xe=0A>>>>>>>>> 918 100646 nfsd nfsd: service mi_switch+0xe1=0A=
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>>=
 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp=
+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=
=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_=
thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>=
>> 918 100647 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0=
x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_L=
OCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsv=
no_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_pro=
gram+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>=
>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100648 nfsd nfsd=
: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __loc=
kmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+=
0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fht=
ovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_inte=
rnal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fo=
rk_trampoline+0xe=0A>>>>>>>>> 918 100649 nfsd nfsd: service mi_switch+0xe=
1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>=
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fh=
tovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0=
x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> =
svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>=
>>>>>> 918 100650 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wa=
it+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c V=
OP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> =
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc=
_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A=
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100651 nfsd =
nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d _=
_lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_l=
ock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd=
_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_=
internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9=
a fork_trampoline+0xe=0A>>>>>>>>> 918 100652 nfsd nfsd: service mi_switch=
+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>=
>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zf=
s_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dor=
pc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>=
>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=
=0A>>>>>>>>> 918 100653 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sle=
epq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+=
0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>=
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> =
nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+=
0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100654=
 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0=
x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab=
 _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7=
c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 sv=
c_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_ex=
it+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100655 nfsd nfsd: service mi_=
switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x9=
02=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>=
>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsr=
vd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A=
>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline=
+0xe=0A>>>>>>>>> 918 100656 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>>=
 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdl=
ock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A=
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>=
>>> nfssvc_program+0x554 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_st=
art+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 10=
0657 nfsd nfsd: service mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleep=
lk+0x15d __lockmgr_args+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+=
0xab _vn_lock+0x43=0A>>>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp=
+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x55=
4 svc_run_internal+0xc77=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> for=
k_exit+0x9a fork_trampoline+0xe=0A>>>>>>>>> 918 100658 nfsd nfsd: service=
 mi_switch+0xe1=0A>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args=
+0x902=0A>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>>=
>>>>>>> zfs_fhtovp+0x38d=0A>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 =
nfsrvd_dorpc+0x917=0A>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc7=
7=0A>>>>>>>>> svc_thread_start+0xb=0A>>>>>>>>> fork_exit+0x9a fork_trampo=
line+0xe=0A>>>>>>>>> =0A>>>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>>>> UNIX System=
s, Network and Security Engineer=0A>>>>>>>>> http://www.unix-experience.f=
r=0A>>>>>>>>> =0A>>>>>>>>> 15 d=C3=A9cembre 2014 13:29 "Lo=C3=AFc Blot"=
=0A>>>>>>>>> <loic.blot@unix-experience.fr>=0A>>>>>>>>> a=0A>>>>>>>>> =C3=
=A9crit:=0A>>>>>>>>> =0A>>>>>>>>> Hmmm...=0A>>>>>>>>> now i'm experiencin=
g a deadlock.=0A>>>>>>>>> =0A>>>>>>>>> 0 918 915 0 21 0 12352 3372 zfs D =
- 1:48.64 nfsd: server=0A>>>>>>>>> (nfsd)=0A>>>>>>>>> =0A>>>>>>>>> the on=
ly issue was to reboot the server, but after rebooting=0A>>>>>>>>> deadlo=
ck arrives a second time when i=0A>>>>>>>>> start my jails over NFS.=0A>>=
>>>>>>> =0A>>>>>>>>> Regards,=0A>>>>>>>>> =0A>>>>>>>>> Lo=C3=AFc Blot,=0A=
>>>>>>>>> UNIX Systems, Network and Security Engineer=0A>>>>>>>>> http://=
www.unix-experience.fr=0A>>>>>>>>> =0A>>>>>>>>> 15 d=C3=A9cembre 2014 10:=
07 "Lo=C3=AFc Blot"=0A>>>>>>>>> <loic.blot@unix-experience.fr>=0A>>>>>>>>=
> a=0A>>>>>>>>> =C3=A9crit:=0A>>>>>>>>> =0A>>>>>>>>> Hi Rick,=0A>>>>>>>>>=
 after talking with my N+1, NFSv4 is required on our=0A>>>>>>>>> infrastr=
ucture.=0A>>>>>>>>> I tried to upgrade NFSv4+ZFS=0A>>>>>>>>> server from =
9.3 to 10.1, i hope this will resolve some=0A>>>>>>>>> issues...=0A>>>>>>=
>>> =0A>>>>>>>>> Regards,=0A>>>>>>>>> =0A>>>>>>>>> Lo=C3=AFc Blot,=0A>>>>=
>>>>> UNIX Systems, Network and Security Engineer=0A>>>>>>>>> http://www.=
unix-experience.fr=0A>>>>>>>>> =0A>>>>>>>>> 10 d=C3=A9cembre 2014 15:36 "=
Lo=C3=AFc Blot"=0A>>>>>>>>> <loic.blot@unix-experience.fr>=0A>>>>>>>>> a=
=0A>>>>>>>>> =C3=A9crit:=0A>>>>>>>>> =0A>>>>>>>>> Hi Rick,=0A>>>>>>>>> th=
anks for your suggestion.=0A>>>>>>>>> For my locking bug, rpc.lockd is st=
ucked in rpcrecv state on=0A>>>>>>>>> the=0A>>>>>>>>> server. kill -9 doe=
sn't affect the=0A>>>>>>>>> process, it's blocked.... (State: Ds)=0A>>>>>=
>>>> =0A>>>>>>>>> for the performances=0A>>>>>>>>> =0A>>>>>>>>> NFSv3: 60=
Mbps=0A>>>>>>>>> NFSv4: 45Mbps=0A>>>>>>>>> Regards,=0A>>>>>>>>> =0A>>>>>>=
>>> Lo=C3=AFc Blot,=0A>>>>>>>>> UNIX Systems, Network and Security Engine=
er=0A>>>>>>>>> http://www.unix-experience.fr=0A>>>>>>>>>; =0A>>>>>>>>> 10 =
d=C3=A9cembre 2014 13:56 "Rick Macklem" <rmacklem@uoguelph.ca>=0A>>>>>>>>=
> a=0A>>>>>>>>> =C3=A9crit:=0A>>>>>>>>> =0A>>>>>>>>> Loic Blot wrote:=0A>=
>>>>>>>> =0A>>>>>>>>>> Hi Rick,=0A>>>>>>>>>> I'm trying NFSv3.=0A>>>>>>>>=
>> Some jails are starting very well but now i have an issue=0A>>>>>>>>>>=
 with=0A>>>>>>>>>> lockd=0A>>>>>>>>>> after some minutes:=0A>>>>>>>>>> =
=0A>>>>>>>>>> nfs server 10.10.X.8:/jails: lockd not responding=0A>>>>>>>=
>>> nfs server 10.10.X.8:/jails lockd is alive again=0A>>>>>>>>>> =0A>>>>=
>>>>>> I look at mbuf, but i seems there is no problem.=0A>>>>>>>>> =0A>>=
>>>>>>> Well, if you need locks to be visible across multiple=0A>>>>>>>>>=
 clients,=0A>>>>>>>>> then=0A>>>>>>>>> I'm afraid you are stuck with usin=
g NFSv4 and the=0A>>>>>>>>> performance=0A>>>>>>>>> you=0A>>>>>>>>> get=
=0A>>>>>>>>> from it. (There is no way to do file handle affinity for=0A>=
>>>>>>>> NFSv4=0A>>>>>>>>> because=0A>>>>>>>>> the read and write ops are=
 buried in the compound RPC and=0A>>>>>>>>> not=0A>>>>>>>>> easily=0A>>>>=
>>>>> recognized.)=0A>>>>>>>>> =0A>>>>>>>>> If the locks don't need to be=
 visible across multiple=0A>>>>>>>>> clients,=0A>>>>>>>>> I'd=0A>>>>>>>>>=
 suggest trying the "nolockd" option with nfsv3.=0A>>>>>>>>> =0A>>>>>>>>>=
> Here is my rc.conf on server:=0A>>>>>>>>>> =0A>>>>>>>>>> nfs_server_ena=
ble=3D"YES"=0A>>>>>>>>>> nfsv4_server_enable=3D"YES"=0A>>>>>>>>>> nfsuser=
d_enable=3D"YES"=0A>>>>>>>>>> nfsd_server_flags=3D"-u -t -n 256"=0A>>>>>>=
>>>> mountd_enable=3D"YES"=0A>>>>>>>>>> mountd_flags=3D"-r"=0A>>>>>>>>>> =
nfsuserd_flags=3D"-usertimeout 0 -force 20"=0A>>>>>>>>>> rpcbind_enable=
=3D"YES"=0A>>>>>>>>>> rpc_lockd_enable=3D"YES"=0A>>>>>>>>>> rpc_statd_ena=
ble=3D"YES"=0A>>>>>>>>>> =0A>>>>>>>>>> Here is the client:=0A>>>>>>>>>> =
=0A>>>>>>>>>> nfsuserd_enable=3D"YES"=0A>>>>>>>>>> nfsuserd_flags=3D"-use=
rtimeout 0 -force 20"=0A>>>>>>>>>> nfscbd_enable=3D"YES"=0A>>>>>>>>>> rpc=
_lockd_enable=3D"YES"=0A>>>>>>>>>> rpc_statd_enable=3D"YES"=0A>>>>>>>>>> =
=0A>>>>>>>>>> Have you got an idea ?=0A>>>>>>>>>> =0A>>>>>>>>>> Regards,=
=0A>>>>>>>>>> =0A>>>>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>>>>> UNIX Systems, Ne=
twork and Security Engineer=0A>>>>>>>>>> http://www.unix-experience.fr=0A=
>>>>>>>>>> =0A>>>>>>>>>> 9 d=C3=A9cembre 2014 04:31 "Rick Macklem" <rmack=
lem@uoguelph.ca>=0A>>>>>>>>>> a=0A>>>>>>>>>> =C3=A9crit:=0A>>>>>>>>>>> Lo=
ic Blot wrote:=0A>>>>>>>>>>> =0A>>>>>>>>>>>> Hi rick,=0A>>>>>>>>>>>> =0A>=
>>>>>>>>>>> I waited 3 hours (no lag at jail launch) and now I do:=0A>>>>=
>>>>>>>> sysrc=0A>>>>>>>>>>>> memcached_flags=3D"-v -m 512"=0A>>>>>>>>>>>=
> Command was very very slow...=0A>>>>>>>>>>>> =0A>>>>>>>>>>>> Here is a =
dd over NFS:=0A>>>>>>>>>>>> =0A>>>>>>>>>>>> 601062912 bytes transferred i=
n 21.060679 secs (28539579=0A>>>>>>>>>>>> bytes/sec)=0A>>>>>>>>>>> =0A>>>=
>>>>>>>> Can you try the same read using an NFSv3 mount?=0A>>>>>>>>>>> (I=
f it runs much faster, you have probably been bitten by=0A>>>>>>>>>>> the=
=0A>>>>>>>>>>> ZFS=0A>>>>>>>>>>> "sequential vs random" read heuristic wh=
ich I've been told=0A>>>>>>>>>>> things=0A>>>>>>>>>>> NFS is doing "rando=
m" reads without file handle affinity.=0A>>>>>>>>>>> File=0A>>>>>>>>>>> h=
andle affinity is very hard to do for NFSv4, so it isn't=0A>>>>>>>>>>> do=
ne.)=0A>>>>>>>>> =0A>>>>>>>>> I was actually suggesting that you try the =
"dd" over nfsv3=0A>>>>>>>>> to=0A>>>>>>>>> see=0A>>>>>>>>> how=0A>>>>>>>>=
> the performance compared with nfsv4. If you do that, please=0A>>>>>>>>>=
 post=0A>>>>>>>>> the=0A>>>>>>>>> comparable results.=0A>>>>>>>>> =0A>>>>=
>>>>> Someday I would like to try and get ZFS's sequential vs=0A>>>>>>>>>=
 random=0A>>>>>>>>> read=0A>>>>>>>>> heuristic modified and any info on w=
hat difference in=0A>>>>>>>>> performance=0A>>>>>>>>> that=0A>>>>>>>>> mi=
ght make for NFS would be useful.=0A>>>>>>>>> =0A>>>>>>>>> rick=0A>>>>>>>=
>> =0A>>>>>>>>> rick=0A>>>>>>>>> =0A>>>>>>>>> This is quite slow...=0A>>>=
>>>>>> =0A>>>>>>>>> You can found some nfsstat below (command isn't finis=
hed=0A>>>>>>>>> yet)=0A>>>>>>>>> =0A>>>>>>>>> nfsstat -c -w 1=0A>>>>>>>>>=
 =0A>>>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir=0A>>>>>=
>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 4 0 0 0 0 0 16 0=0A>>>>>>>>> 2 0 0 0 0 0=
 17 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>=
 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 4 0 0 0 0 4 0=
=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 =
0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 4 0 0 0 0 0 3 0=0A>>>=
>>>>>> 0 0 0 0 0 0 3 0=0A>>>>>>>>> 37 10 0 8 0 0 14 1=0A>>>>>>>>> 18 16 0=
 4 1 2 4 0=0A>>>>>>>>> 78 91 0 82 6 12 30 0=0A>>>>>>>>> 19 18 0 2 2 4 2 0=
=0A>>>>>>>>> 0 0 0 0 2 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> GtAt=
tr Lookup Rdlink Read Write Rename Access Rddir=0A>>>>>>>>> 0 0 0 0 0 0 0=
 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 =
1 0 0 0 0 1 0=0A>>>>>>>>> 4 6 0 0 6 0 3 0=0A>>>>>>>>> 2 0 0 0 0 0 0 0=0A>=
>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 1 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 =
1 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>=
>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 =
0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0=
 0 0 0 0 0 0=0A>>>>>>>>> 6 108 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A=
>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> GtAttr Lookup Rdlink Read Write Ren=
ame Access Rddir=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=
=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 =
0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>=
>>>>>> 98 54 0 86 11 0 25 0=0A>>>>>>>>> 36 24 0 39 25 0 10 1=0A>>>>>>>>> =
67 8 0 63 63 0 41 0=0A>>>>>>>>> 34 0 0 35 34 0 0 0=0A>>>>>>>>> 75 0 0 75 =
77 0 0 0=0A>>>>>>>>> 34 0 0 35 35 0 0 0=0A>>>>>>>>> 75 0 0 74 76 0 0 0=0A=
>>>>>>>>> 33 0 0 34 33 0 0 0=0A>>>>>>>>> 0 0 0 0 5 0 0 0=0A>>>>>>>>> 0 0 =
0 0 0 0 6 0=0A>>>>>>>>> 11 0 0 0 0 0 11 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>=
>>>>>>>> 0 17 0 0 0 0 1 0=0A>>>>>>>>> GtAttr Lookup Rdlink Read Write Ren=
ame Access Rddir=0A>>>>>>>>> 4 5 0 0 0 0 12 0=0A>>>>>>>>> 2 0 0 0 0 0 26 =
0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0=
 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>=
>>>>>>> 0 4 0 0 0 0 4 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0=
 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 4 0 0 0 0 0 2 0=0A>>>>>>>>=
> 2 0 0 0 0 0 24 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 =
0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0=
 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>=
>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir=0A>>>>>>>>> 0=
 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 4 0 0 0 0 0 7 0=0A=
>>>>>>>>> 2 1 0 0 0 0 1 0=0A>>>>>>>>> 0 0 0 0 2 0 0 0=0A>>>>>>>>> 0 0 0 0=
 0 0 0 0=0A>>>>>>>>> 0 0 0 0 6 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>=
>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0=
 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 4 =
6 0 0 0 0 3 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 2 0 0 0 0 0 0 0=0A>=
>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 =
0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> GtAttr Lookup Rdlink Read=
 Write Rename Access Rddir=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 =
0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>=
>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 4 71 0 0 0 0 0 0=0A>>>>>>>>> 0 1 0 0 0 0=
 0 0=0A>>>>>>>>> 2 36 0 0 0 0 1 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>>=
 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=
=0A>>>>>>>>> 1 0 0 0 0 0 1 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 =
0 0 0 0 0 0=0A>>>>>>>>> 79 6 0 79 79 0 2 0=0A>>>>>>>>> 25 0 0 25 26 0 6 0=
=0A>>>>>>>>> 43 18 0 39 46 0 23 0=0A>>>>>>>>> 36 0 0 36 36 0 31 0=0A>>>>>=
>>>> 68 1 0 66 68 0 0 0=0A>>>>>>>>> GtAttr Lookup Rdlink Read Write Renam=
e Access Rddir=0A>>>>>>>>> 36 0 0 36 36 0 0 0=0A>>>>>>>>> 48 0 0 48 49 0 =
0 0=0A>>>>>>>>> 20 0 0 20 20 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>=
> 3 14 0 1 0 0 11 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 0 0 0 0 0 0 0=
 0=0A>>>>>>>>> 0 4 0 0 0 0 4 0=0A>>>>>>>>> 0 0 0 0 0 0 0 0=0A>>>>>>>>> 4 =
22 0 0 0 0 16 0=0A>>>>>>>>> 2 0 0 0 0 0 23 0=0A>>>>>>>>> =0A>>>>>>>>> Reg=
ards,=0A>>>>>>>>> =0A>>>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>>>> UNIX Systems, =
Network and Security Engineer=0A>>>>>>>>> http://www.unix-experience.fr=
=0A>>>>>>>>> =0A>>>>>>>>> 8 d=C3=A9cembre 2014 09:36 "Lo=C3=AFc Blot"=0A>=
>>>>>>>> <loic.blot@unix-experience.fr> a=0A>>>>>>>>> =C3=A9crit:=0A>>>>>=
>>>>> Hi Rick,=0A>>>>>>>>>> I stopped the jails this week-end and started=
 it this=0A>>>>>>>>>> morning,=0A>>>>>>>>>> i'll=0A>>>>>>>>>> give you so=
me stats this week.=0A>>>>>>>>>> =0A>>>>>>>>>> Here is my nfsstat -m outp=
ut (with your rsize/wsize=0A>>>>>>>>>> tweaks)=0A>>>>>>>> =0A>>>>>>>> =0A=
>>>>>>> =0A>>>>>> =0A>>>>> =0A>>>> =0A>>> =0A>> =0A> nfsv4,tcp,resvport,h=
ard,cto,sec=3Dsys,acdirmin=3D3,acdirmax=3D60,acregmin=3D5,acregmax=3D60,n=
ametimeo=3D60,negna=0A>>>>>>>> =0A>>>>>>>>> =0A>>>>>>>> =0A>>>>>>>> =0A>>=
>>>>> =0A>>>>>> =0A>>>>> =0A>>>> =0A>>> =0A>> =0A> etimeo=3D60,rsize=3D32=
768,wsize=3D32768,readdirsize=3D32768,readahead=3D1,wcommitsize=3D773136,=
timeout=3D120,retra=0A>>>>>>>> =0A>>>>>>>>> s=3D2147483647=0A>>>>>>>>> =
=0A>>>>>>>>> On server side my disks are on a raid controller which show =
a=0A>>>>>>>>> 512b=0A>>>>>>>>> volume and write performances=0A>>>>>>>>> =
are very honest (dd if=3D/dev/zero of=3D/jails/test.dd bs=3D4096=0A>>>>>>=
>>> count=3D100000000 =3D> 450MBps)=0A>>>>>>>>> =0A>>>>>>>>> Regards,=0A>=
>>>>>>>> =0A>>>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>>>> UNIX Systems, Network a=
nd Security Engineer=0A>>>>>>>>> http://www.unix-experience.fr=0A>>>>>>>>=
> =0A>>>>>>>>> 5 d=C3=A9cembre 2014 15:14 "Rick Macklem" <rmacklem@uoguel=
ph.ca> a=0A>>>>>>>>> =C3=A9crit:=0A>>>>>>>>> =0A>>>>>>>>> Loic Blot wrote=
:=0A>>>>>>>>> =0A>>>>>>>>> Hi,=0A>>>>>>>>> i'm trying to create a virtual=
isation environment based on=0A>>>>>>>>> jails.=0A>>>>>>>>> Those jails a=
re stored under a big ZFS pool on a FreeBSD=0A>>>>>>>>> 9.3=0A>>>>>>>>> w=
hich=0A>>>>>>>>> export a NFSv4 volume. This NFSv4 volume was mounted on =
a=0A>>>>>>>>> big=0A>>>>>>>>> hypervisor (2 Xeon E5v3 + 128GB memory and =
8 ports (but=0A>>>>>>>>> only 1=0A>>>>>>>>> was=0A>>>>>>>>> used at this =
time).=0A>>>>>>>>> =0A>>>>>>>>> The problem is simple, my hypervisors run=
s 6 jails (used 1%=0A>>>>>>>>> cpu=0A>>>>>>>>> and=0A>>>>>>>>> 10GB RAM a=
pproximatively and less than 1MB bandwidth) and=0A>>>>>>>>> works=0A>>>>>=
>>>> fine at start but the system slows down and after 2-3 days=0A>>>>>>>=
>> become=0A>>>>>>>>> unusable. When i look at top command i see 80-100% =
on=0A>>>>>>>>> system=0A>>>>>>>>> and=0A>>>>>>>>> commands are very very =
slow. Many process are tagged with=0A>>>>>>>>> nfs_cl*.=0A>>>>>>>>> =0A>>=
>>>>>>> To be honest, I would expect the slowness to be because of=0A>>>>=
>>>>> slow=0A>>>>>>>>> response=0A>>>>>>>>> from the NFSv4 server, but if=
 you do:=0A>>>>>>>>> # ps axHl=0A>>>>>>>>> on a client when it is slow an=
d post that, it would give us=0A>>>>>>>>> some=0A>>>>>>>>> more=0A>>>>>>>=
>> information on where the client side processes are sitting.=0A>>>>>>>>=
> If you also do something like:=0A>>>>>>>>> # nfsstat -c -w 1=0A>>>>>>>>=
> and let it run for a while, that should show you how many=0A>>>>>>>>> R=
PCs=0A>>>>>>>>> are=0A>>>>>>>>> being done and which ones.=0A>>>>>>>>> =
=0A>>>>>>>>> # nfsstat -m=0A>>>>>>>>> will show you what your mount is ac=
tually using.=0A>>>>>>>>> The only mount option I can suggest trying is=
=0A>>>>>>>>> "rsize=3D32768,wsize=3D32768",=0A>>>>>>>>> since some networ=
k environments have difficulties with 64K.=0A>>>>>>>>> =0A>>>>>>>>> There=
 are a few things you can try on the NFSv4 server side,=0A>>>>>>>>> if=0A=
>>>>>>>>> it=0A>>>>>>>>> appears=0A>>>>>>>>> that the clients are generat=
ing a large RPC load.=0A>>>>>>>>> - disabling the DRC cache for TCP by se=
tting=0A>>>>>>>>> vfs.nfsd.cachetcp=3D0=0A>>>>>>>>> - If the server is se=
eing a large write RPC load, then=0A>>>>>>>>> "sync=3Ddisabled"=0A>>>>>>>=
>> might help, although it does run a risk of data loss when=0A>>>>>>>>> =
the=0A>>>>>>>>> server=0A>>>>>>>>> crashes.=0A>>>>>>>>> Then there are a =
couple of other ZFS related things (I'm not=0A>>>>>>>>> a=0A>>>>>>>>> ZFS=
=0A>>>>>>>>> guy,=0A>>>>>>>>> but these have shown up on the mailing list=
s).=0A>>>>>>>>> - make sure your volumes are 4K aligned and ashift=3D12 (=
in=0A>>>>>>>>> case a=0A>>>>>>>>> drive=0A>>>>>>>>> that uses 4K sectors =
is pretending to be 512byte sectored)=0A>>>>>>>>> - never run over 70-80%=
 full if write performance is an=0A>>>>>>>>> issue=0A>>>>>>>>> - use a zi=
l on an SSD with good write performance=0A>>>>>>>>> =0A>>>>>>>>> The only=
 NFSv4 thing I can tell you is that it is known that=0A>>>>>>>>> ZFS's=0A=
>>>>>>>>> algorithm for determining sequential vs random I/O fails for=0A=
>>>>>>>>> NFSv4=0A>>>>>>>>> during writing and this can be a performance =
hit. The only=0A>>>>>>>>> workaround=0A>>>>>>>>> is to use NFSv3 mounts, =
since file handle affinity=0A>>>>>>>>> apparently=0A>>>>>>>>> fixes=0A>>>=
>>>>>> the problem and this is only done for NFSv3.=0A>>>>>>>>> =0A>>>>>>=
>>> rick=0A>>>>>>>>> =0A>>>>>>>>> I saw that there are TSO issues with ig=
b then i'm trying to=0A>>>>>>>>> disable=0A>>>>>>>>> it with sysctl but t=
he situation wasn't solved.=0A>>>>>>>>> =0A>>>>>>>>> Someone has got idea=
s ? I can give you more informations if=0A>>>>>>>>> you=0A>>>>>>>>> need.=
=0A>>>>>>>>> =0A>>>>>>>>> Thanks in advance.=0A>>>>>>>>> Regards,=0A>>>>>=
>>>> =0A>>>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>>>> UNIX Systems, Network and S=
ecurity Engineer=0A>>>>>>>>> http://www.unix-experience.fr=0A>>>>>>>>>; __=
_____________________________________________=0A>>>>>>>>> freebsd-fs@free=
bsd.org mailing list=0A>>>>>>>>> http://lists.freebsd.org/mailman/listinf=
o/freebsd-fs=0A>>>>>>>>> To unsubscribe, send any mail to=0A>>>>>>>>> "fr=
eebsd-fs-unsubscribe@freebsd.org"=0A>>>>>>>>> =0A>>>>>>>>> ______________=
_________________________________=0A>>>>>>>>> freebsd-fs@freebsd.org mail=
ing list=0A>>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs=
=0A>>>>>>>>> To unsubscribe, send any mail to=0A>>>>>>>>> "freebsd-fs-uns=
ubscribe@freebsd.org"=0A>>>>>>>>> =0A>>>>>>>>> __________________________=
_____________________=0A>>>>>>>>> freebsd-fs@freebsd.org mailing list=0A>=
>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>>>>>>>>>=
 To unsubscribe, send any mail to=0A>>>>>>>>> "freebsd-fs-unsubscribe@fre=
ebsd.org"=0A>>>>>>>>> =0A>>>>>>>>> ______________________________________=
_________=0A>>>>>>>>> freebsd-fs@freebsd.org mailing list=0A>>>>>>>>> htt=
p://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>>>>>>>>> To unsubscr=
ibe, send any mail to=0A>>>>>>>>> "freebsd-fs-unsubscribe@freebsd.org"=0A=
>>>>>>>>> _______________________________________________=0A>>>>>>>>> fre=
ebsd-fs@freebsd.org mailing list=0A>>>>>>>>> http://lists.freebsd.org/mai=
lman/listinfo/freebsd-fs=0A>>>>>>>>> To unsubscribe, send any mail to=0A>=
>>>>>>>> "freebsd-fs-unsubscribe@freebsd.org"=0A>>>>>>>>> =0A>>>>>>>>> __=
_____________________________________________=0A>>>>>>>>> freebsd-fs@free=
bsd.org mailing list=0A>>>>>>>>> http://lists.freebsd.org/mailman/listinf=
o/freebsd-fs=0A>>>>>>>>> To unsubscribe, send any mail to=0A>>>>>>>>> "fr=
eebsd-fs-unsubscribe@freebsd.org"=0A>>>>>>> =0A>>>>>>> __________________=
_____________________________=0A>>>>>>> freebsd-fs@freebsd.org mailing li=
st=0A>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>>>>>=
>> To unsubscribe, send any mail to=0A>>>>>>> "freebsd-fs-unsubscribe@fre=
ebsd.org"



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4426a915c891b0faf962f1975c4dee1e>