Date: Mon, 22 Dec 2014 08:57:40 +0000 From: "=?utf-8?B?TG/Dr2MgQmxvdA==?=" <loic.blot@unix-experience.fr> To: "Rick Macklem" <rmacklem@uoguelph.ca> Cc: freebsd-fs@freebsd.org Subject: Re: ZFS vnode lock deadlock in zfs_fhtovp was: High Kernel Load with nfsv4 Message-ID: <811d455b0bcaeb43711e8108c96d4f2b@mail.unix-experience.fr> In-Reply-To: <9fcfcbfe720a9b56a995cd6e227b8f9f@mail.unix-experience.fr> References: <9fcfcbfe720a9b56a995cd6e227b8f9f@mail.unix-experience.fr> <2087358136.248078.1419122007097.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi,=0A=0ATo clarify because of our exchanges. Here are the current sysctl= options for server:=0A=0Avfs.nfsd.enable_nobodycheck=3D0=0Avfs.nfsd.enab= le_nogroupcheck=3D0=0A=0Avfs.nfsd.maxthreads=3D200=0Avfs.nfsd.tcphighwate= r=3D10000=0Avfs.nfsd.tcpcachetimeo=3D300=0Avfs.nfsd.server_min_nfsvers=3D= 4=0A=0Akern.maxvnodes=3D10000000=0Akern.ipc.maxsockbuf=3D4194304=0Anet.in= et.tcp.sendbuf_max=3D4194304=0Anet.inet.tcp.recvbuf_max=3D4194304=0A=0Avf= s.lookup_shared=3D0=0A=0ARegards,=0A=0ALo=C3=AFc Blot,=0AUNIX Systems, Ne= twork and Security Engineer=0Ahttp://www.unix-experience.fr=0A=0A22 d=C3= =A9cembre 2014 09:42 "Lo=C3=AFc Blot" <loic.blot@unix-experience.fr> a = =C3=A9crit: =0A=0AHi Rick,=0Amy 5 jails runs this weekend and now i have = some stats on this monday.=0A=0AHopefully deadlock was fixed, yeah, but e= verything isn't good :(=0A=0AOn NFSv4 server (FreeBSD 10.1) system uses 3= 5% CPU=0A=0AAs i can see this is because of nfsd:=0A=0A918 root = 96 20 0 12352K 3372K rpcsvc 6 51.4H 273.68% nfsd: server (nfsd)= =0A=0AIf i look at dmesg i see:=0Anfsd server cache flooded, try increasi= ng vfs.nfsd.tcphighwater=0A=0Avfs.nfsd.tcphighwater was set to 10000, i i= ncrease it to 15000=0A=0AHere is 'nfsstat -s' output:=0A=0AServer Info:= =0AGetattr Setattr Lookup Readlink Read Write Create = Remove=0A12600652 1812 2501097 156 1386423 1983729 = 123 162067=0ARename Link Symlink Mkdir Rmdir Readdi= r RdirPlus Access=0A36762 9 0 0 0 = 3147 0 623524=0AMknod Fsstat Fsinfo PathConf Commi= t=0A0 0 0 0 328117=0AServer Ret-Failed=0A0=0AS= erver Faults=0A0=0AServer Cache Stats:=0AInprog Idem Non-idem Mi= sses=0A0 0 0 12635512=0AServer Write Gathering:=0AWriteO= ps WriteRPC Opsaved=0A1983729 1983729 0=0A=0AAnd here is 'pr= ocstat -kk' for nfsd (server)=0A=0A918 100528 nfsd nfsd: mast= er mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10= _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1 svc_run+0x1de nfsrvd_= nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c amd64_syscall+0x351 Xfast_sy= scall+0xfb =0A918 100568 nfsd nfsd: service mi_switch+0xe1= sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run= _internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe = =0A918 100569 nfsd nfsd: service mi_switch+0xe1 sleepq_cat= ch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0= x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 10057= 0 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0= xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100571 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_w= ait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0x= b fork_exit+0x9a fork_trampoline+0xe =0A918 100572 nfsd nfsd:= service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf = _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a fork_trampoline+0xe =0A918 100573 nfsd nfsd: service = mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig= +0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_tr= ampoline+0xe =0A918 100574 nfsd nfsd: service mi_switch+0x= e1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_r= un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe= =0A918 100575 nfsd nfsd: service mi_switch+0xe1 sleepq_ca= tch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+= 0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 1005= 76 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+= 0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_th= read_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100577 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_= wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0= xb fork_exit+0x9a fork_trampoline+0xe =0A918 100578 nfsd nfsd= : service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf= _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a fork_trampoline+0xe =0A918 100579 nfsd nfsd: service = mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si= g+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_t= rampoline+0xe =0A918 100580 nfsd nfsd: service mi_switch+0= xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_= run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0x= e =0A918 100581 nfsd nfsd: service mi_switch+0xe1 sleepq_c= atch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100= 582 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals= +0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_t= hread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100583 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq= _wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100584 nfsd nfs= d: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0x= f _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exi= t+0x9a fork_trampoline+0xe =0A918 100585 nfsd nfsd: service = mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_s= ig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_= trampoline+0xe =0A918 100586 nfsd nfsd: service mi_switch+= 0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0= xe =0A918 100587 nfsd nfsd: service mi_switch+0xe1 sleepq_= catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_interna= l+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 10= 0588 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signal= s+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_= thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100589 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleep= q_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start= +0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100590 nfsd nf= sd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0= xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_ex= it+0x9a fork_trampoline+0xe =0A918 100591 nfsd nfsd: service = mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_= sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork= _trampoline+0xe =0A918 100592 nfsd nfsd: service mi_switch= +0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a sv= c_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+= 0xe =0A918 100593 nfsd nfsd: service mi_switch+0xe1 sleepq= _catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 1= 00594 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signa= ls+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc= _thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100595 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab slee= pq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_star= t+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100596 nfsd n= fsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a fork_trampoline+0xe =0A918 100597 nfsd nfsd: service= mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait= _sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a for= k_trampoline+0xe =0A918 100598 nfsd nfsd: service mi_switc= h+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a s= vc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline= +0xe =0A918 100599 nfsd nfsd: service mi_switch+0xe1 sleep= q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 = 100600 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_sign= als+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e sv= c_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100601 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sle= epq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_sta= rt+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100602 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig= +0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_= exit+0x9a fork_trampoline+0xe =0A918 100603 nfsd nfsd: servic= e mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wai= t_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fo= rk_trampoline+0xe =0A918 100604 nfsd nfsd: service mi_swit= ch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a = svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampolin= e+0xe =0A918 100605 nfsd nfsd: service mi_switch+0xe1 slee= pq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_inte= rnal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918= 100606 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_sig= nals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e s= vc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100607 nfsd= nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sl= eepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_st= art+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100608 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_si= g+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork= _exit+0x9a fork_trampoline+0xe =0A918 100609 nfsd nfsd: servi= ce mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wa= it_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a f= ork_trampoline+0xe =0A918 100610 nfsd nfsd: service mi_swi= tch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a= svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoli= ne+0xe =0A918 100611 nfsd nfsd: service mi_switch+0xe1 sle= epq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A91= 8 100612 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e = svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100613 nfs= d nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab s= leepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_s= tart+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100614 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_s= ig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb for= k_exit+0x9a fork_trampoline+0xe =0A918 100615 nfsd nfsd: serv= ice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_w= ait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a = fork_trampoline+0xe =0A918 100616 nfsd nfsd: service mi_sw= itch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16= a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampol= ine+0xe =0A918 100617 nfsd nfsd: service mi_switch+0xe1 sl= eepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A9= 18 100618 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_s= ignals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e= svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100619 nf= sd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab = sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100620 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_= sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fo= rk_exit+0x9a fork_trampoline+0xe =0A918 100621 nfsd nfsd: ser= vice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_= wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= fork_trampoline+0xe =0A918 100622 nfsd nfsd: service mi_s= witch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x1= 6a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampo= line+0xe =0A918 100623 nfsd nfsd: service mi_switch+0xe1 s= leepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_i= nternal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A= 918 100624 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_= signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87= e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100625 n= fsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab= sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread= _start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100626 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait= _sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb f= ork_exit+0x9a fork_trampoline+0xe =0A918 100627 nfsd nfsd: se= rvice mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv= _wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9= a fork_trampoline+0xe =0A918 100628 nfsd nfsd: service mi_= switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x= 16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_tramp= oline+0xe =0A918 100629 nfsd nfsd: service mi_switch+0xe1 = sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_= internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe = =0A918 100630 nfsd nfsd: service mi_switch+0xe1 sleepq_cat= ch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0= x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 10063= 1 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0= xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100632 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_w= ait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0x= b fork_exit+0x9a fork_trampoline+0xe =0A918 100633 nfsd nfsd:= service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf = _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a fork_trampoline+0xe =0A918 100634 nfsd nfsd: service = mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig= +0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_tr= ampoline+0xe =0A918 100635 nfsd nfsd: service mi_switch+0x= e1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_r= un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe= =0A918 100636 nfsd nfsd: service mi_switch+0xe1 sleepq_ca= tch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+= 0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 1006= 37 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+= 0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_th= read_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100638 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_= wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0= xb fork_exit+0x9a fork_trampoline+0xe =0A918 100639 nfsd nfsd= : service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf= _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit= +0x9a fork_trampoline+0xe =0A918 100640 nfsd nfsd: service = mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_si= g+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_t= rampoline+0xe =0A918 100641 nfsd nfsd: service mi_switch+0= xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_= run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0x= e =0A918 100642 nfsd nfsd: service mi_switch+0xe1 sleepq_c= atch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100= 643 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals= +0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_t= hread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100644 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq= _wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100645 nfsd nfs= d: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0x= f _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exi= t+0x9a fork_trampoline+0xe =0A918 100646 nfsd nfsd: service = mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_s= ig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_= trampoline+0xe =0A918 100647 nfsd nfsd: service mi_switch+= 0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc= _run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0= xe =0A918 100648 nfsd nfsd: service mi_switch+0xe1 sleepq_= catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_interna= l+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 10= 0649 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signal= s+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_= thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100650 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleep= q_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start= +0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100651 nfsd nf= sd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0= xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_ex= it+0x9a fork_trampoline+0xe =0A918 100652 nfsd nfsd: service = mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_= sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork= _trampoline+0xe =0A918 100653 nfsd nfsd: service mi_switch= +0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a sv= c_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+= 0xe =0A918 100654 nfsd nfsd: service mi_switch+0xe1 sleepq= _catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_intern= al+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 1= 00655 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signa= ls+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc= _thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100656 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab slee= pq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_star= t+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100657 nfsd n= fsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+= 0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_e= xit+0x9a fork_trampoline+0xe =0A918 100658 nfsd nfsd: service= mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait= _sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a for= k_trampoline+0xe =0A918 100659 nfsd nfsd: service mi_switc= h+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a s= vc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline= +0xe =0A918 100660 nfsd nfsd: service mi_switch+0xe1 sleep= q_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_inter= nal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 = 100661 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_sign= als+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e sv= c_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe =0A918 100662 nfsd = nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sle= epq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_sta= rt+0xb fork_exit+0x9a fork_trampoline+0xe=0A---=0A=0ANow if we look at cl= ient (FreeBSD 9.3)=0A=0AWe see system was very busy and do many and many = interrupts=0A=0ACPU: 0.0% user, 0.0% nice, 37.8% system, 51.2% interrup= t, 11.0% idle=0A=0AA look at process list shows that there are many sendm= ail process in state nfstry=0A=0Anfstry 18 32:27 0.88% sendmail: Queue= runner@00:30:00 for /var/spool/clientm=0A=0AHere is 'nfsstat -c' output:= =0A=0AClient Info:=0ARpc Counts:=0AGetattr Setattr Lookup Readlink = Read Write Create Remove=0A1051347 1724 2494481 = 118 903902 1901285 162676 161899=0ARename Link Symli= nk Mkdir Rmdir Readdir RdirPlus Access=0A36744 2 = 0 114 40 3131 0 544136=0AMknod Fsst= at Fsinfo PathConf Commit=0A9 0 0 0 245= 821=0ARpc Info:=0ATimedOut Invalid X Replies Retries Requests=0A0 = 0 0 0 8356557=0ACache Info:=0AAttr Hits Misses= Lkup Hits Misses BioR Hits Misses BioW Hits Misses=0A108754455 = 491475 54229224 2437229 46814561 821723 5132123 1871871=0AB= ioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits M= isses=0A144035 118 53736 2753 27813 1 5723883= 9 544205=0A=0AIf you need more things, tell me, i let the PoC in this = state.=0A=0AThanks=0A=0ARegards,=0A=0ALo=C3=AFc Blot,=0AUNIX Systems, Net= work and Security Engineer=0Ahttp://www.unix-experience.fr=0A=0A21 d=C3= =A9cembre 2014 01:33 "Rick Macklem" <rmacklem@uoguelph.ca> a =C3=A9crit: = =0A=0A=0ALoic Blot wrote:=0A=0A> Hi Rick,=0A> ok, i don't need locallocks= , i haven't understand option was for that=0A> usage, i removed it.=0A> I= do more tests on monday.=0A> Thanks for the deadlock fix, for other peop= le :)=0A=0AGood. Please let us know if running with vfs.nfsd.enable_local= locks=3D0=0Agets rid of the deadlocks? (I think it fixes the one you saw.= )=0A=0AOn the performance side, you might also want to try different valu= es of=0Areadahead, if the Linux client has such a mount option. (With the= =0ANFSv4-ZFS sequential vs random I/O heuristic, I have no idea what the= =0Aoptimal readahead value would be.)=0A=0AGood luck with it and please l= et us know how it goes, rick=0Aps: I now have a patch to fix the deadlock= when vfs.nfsd.enable_locallocks=3D1=0Ais set. I'll post it for anyone wh= o is interested after I put it=0Athrough some testing.=0A=0A=0A--=0ABest = regards,=0ALo=C3=AFc BLOT,=0AUNIX systems, security and network engineer= =0Ahttp://www.unix-experience.fr=0A=0ALe jeudi 18 d=C3=A9cembre 2014 =C3= =A0 19:46 -0500, Rick Macklem a =C3=A9crit : =0A=0ALoic Blot wrote: =0A> = Hi rick,=0A> i tried to start a LXC container on Debian Squeeze from my= =0A> freebsd=0A> ZFS+NFSv4 server and i also have a deadlock on nfsd=0A> = (vfs.lookup_shared=3D0). Deadlock procs each time i launch a=0A> squeeze= =0A> container, it seems (3 tries, 3 fails).=0A=0AWell, I`ll take a look = at this `procstat -kk`, but the only thing=0AI`ve seen posted w.r.t. avoi= ding deadlocks in ZFS is to not use=0Anullfs. (I have no idea if you are = using any nullfs mounts, but=0Aif so, try getting rid of them.)=0A=0AHere= `s a high level post about the ZFS and vnode locking problem,=0Abut there= is no patch available, as far as I know.=0A=0Ahttp://docs.FreeBSD.org/cg= i/mid.cgi?54739F41.8030407=0A=0Arick=0A=0A=0A921 - D 0:00.02 nfsd:= server (nfsd)=0A=0AHere is the procstat -kk=0A=0APID TID COMM = TDNAME KSTACK=0A921 100538 nfsd nfsd: master = mi_switch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e= =0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Anfsvno_advlock+0x1= 19 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad=0Anfsrvd_locku+0x283 nfsrvd_d= orpc+0xec6 nfssvc_program+0x554=0Asvc_run_internal+0xc77 svc_run+0x1de nf= srvd_nfsd+0x1ca=0Anfssvc_nfsd+0x107 sys_nfssvc+0x9c=0A921 100572 nfsd = nfsd: service mi_switch+0xe1=0Asleepq_catch_signals+0xab slee= pq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_= start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100573 nfsd = nfsd: service mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wa= it_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start= +0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100574 nfsd n= fsd: service mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_si= g+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100575 nfsd nfsd: = service mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_= exit+0x9a=0Afork_trampoline+0xe=0A921 100576 nfsd nfsd: servi= ce mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_c= v_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0Afork_trampoline+0xe=0A921 100577 nfsd nfsd: service = mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wai= t_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0Afork_trampoline+0xe=0A921 100578 nfsd nfsd: service mi_= switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_si= g+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afo= rk_trampoline+0xe=0A921 100579 nfsd nfsd: service mi_switc= h+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x1= 6a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_tr= ampoline+0xe=0A921 100580 nfsd nfsd: service mi_switch+0xe= 1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0A= svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampol= ine+0xe=0A921 100581 nfsd nfsd: service mi_switch+0xe1=0As= leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_r= un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0= xe=0A921 100582 nfsd nfsd: service mi_switch+0xe1=0Asleepq= _catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A= 921 100583 nfsd nfsd: service mi_switch+0xe1=0Asleepq_catc= h_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_interna= l+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 1= 00584 nfsd nfsd: service mi_switch+0xe1=0Asleepq_catch_sig= nals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100585= nfsd nfsd: service mi_switch+0xe1=0Asleepq_catch_signals+= 0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e sv= c_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100586 nfsd= nfsd: service mi_switch+0xe1=0Asleepq_catch_signals+0xab = sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100587 nfsd = nfsd: service mi_switch+0xe1=0Asleepq_catch_signals+0xab sleep= q_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_s= tart+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100588 nfsd = nfsd: service mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100589 nfsd nf= sd: service mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig= +0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb f= ork_exit+0x9a=0Afork_trampoline+0xe=0A921 100590 nfsd nfsd: s= ervice mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_= exit+0x9a=0Afork_trampoline+0xe=0A921 100591 nfsd nfsd: servi= ce mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_c= v_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0Afork_trampoline+0xe=0A921 100592 nfsd nfsd: service = mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wai= t_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0Afork_trampoline+0xe=0A921 100593 nfsd nfsd: service mi_= switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_si= g+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afo= rk_trampoline+0xe=0A921 100594 nfsd nfsd: service mi_switc= h+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x1= 6a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_tr= ampoline+0xe=0A921 100595 nfsd nfsd: service mi_switch+0xe= 1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0A= svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampol= ine+0xe=0A921 100596 nfsd nfsd: service mi_switch+0xe1=0As= leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_r= un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0= xe=0A921 100597 nfsd nfsd: service mi_switch+0xe1=0Asleepq= _catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A= 921 100598 nfsd nfsd: service mi_switch+0xe1=0Asleepq_catc= h_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_interna= l+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 1= 00599 nfsd nfsd: service mi_switch+0xe1=0Asleepq_catch_sig= nals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100600= nfsd nfsd: service mi_switch+0xe1=0Asleepq_catch_signals+= 0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e sv= c_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100601 nfsd= nfsd: service mi_switch+0xe1=0Asleepq_catch_signals+0xab = sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100602 nfsd = nfsd: service mi_switch+0xe1=0Asleepq_catch_signals+0xab sleep= q_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_s= tart+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100603 nfsd = nfsd: service mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100604 nfsd nf= sd: service mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig= +0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb f= ork_exit+0x9a=0Afork_trampoline+0xe=0A921 100605 nfsd nfsd: s= ervice mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_= exit+0x9a=0Afork_trampoline+0xe=0A921 100606 nfsd nfsd: servi= ce mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_c= v_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0Afork_trampoline+0xe=0A921 100607 nfsd nfsd: service = mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wai= t_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0Afork_trampoline+0xe=0A921 100608 nfsd nfsd: service mi_= switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_si= g+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afo= rk_trampoline+0xe=0A921 100609 nfsd nfsd: service mi_switc= h+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x1= 6a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_tr= ampoline+0xe=0A921 100610 nfsd nfsd: service mi_switch+0xe= 1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0A= svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampol= ine+0xe=0A921 100611 nfsd nfsd: service mi_switch+0xe1=0As= leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_r= un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0= xe=0A921 100612 nfsd nfsd: service mi_switch+0xe1=0Asleepq= _catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A= 921 100613 nfsd nfsd: service mi_switch+0xe1=0Asleepq_catc= h_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_interna= l+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 1= 00614 nfsd nfsd: service mi_switch+0xe1=0Asleepq_catch_sig= nals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100615= nfsd nfsd: service mi_switch+0xe1=0Asleepq_catch_signals+= 0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e sv= c_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100616 nfsd= nfsd: service mi_switch+0xe1=0Asleepq_wait+0x3a _sleep+0x= 287 nfsmsleep+0x66 nfsv4_lock+0x9b=0Anfsrv_getlockfile+0x179 nfsrv_lockct= rl+0x21f nfsrvd_lock+0x5b1=0Anfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_= run_internal+0xc77=0Asvc_thread_start+0xb fork_exit+0x9a fork_trampoline+= 0xe=0A921 100617 nfsd nfsd: service mi_switch+0xe1=0Asleep= q_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_i= nternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe= =0A921 100618 nfsd nfsd: service mi_switch+0xe1=0Asleepq_w= ait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0Anfsrvd_dorpc+0x316= nfssvc_program+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0xb fork_= exit+0x9a fork_trampoline+0xe=0A921 100619 nfsd nfsd: service= mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_= wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x= 9a=0Afork_trampoline+0xe=0A921 100620 nfsd nfsd: service m= i_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_= sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0A= fork_trampoline+0xe=0A921 100621 nfsd nfsd: service mi_swi= tch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0= x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_= trampoline+0xe=0A921 100622 nfsd nfsd: service mi_switch+0= xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a= =0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_tram= poline+0xe=0A921 100623 nfsd nfsd: service mi_switch+0xe1= =0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0As= vc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoli= ne+0xe=0A921 100624 nfsd nfsd: service mi_switch+0xe1=0Asl= eepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_ru= n_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0x= e=0A921 100625 nfsd nfsd: service mi_switch+0xe1=0Asleepq_= catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_int= ernal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A9= 21 100626 nfsd nfsd: service mi_switch+0xe1=0Asleepq_catch= _signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal= +0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 10= 0627 nfsd nfsd: service mi_switch+0xe1=0Asleepq_catch_sign= als+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87= e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100628 = nfsd nfsd: service mi_switch+0xe1=0Asleepq_catch_signals+0= xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc= _thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100629 nfsd = nfsd: service mi_switch+0xe1=0Asleepq_catch_signals+0xab s= leepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thre= ad_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100630 nfsd = nfsd: service mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq= _wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_st= art+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100631 nfsd = nfsd: service mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait= _sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0= xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100632 nfsd nfs= d: service mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+= 0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fo= rk_exit+0x9a=0Afork_trampoline+0xe=0A921 100633 nfsd nfsd: se= rvice mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_= exit+0x9a=0Afork_trampoline+0xe=0A921 100634 nfsd nfsd: servi= ce mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_c= v_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0Afork_trampoline+0xe=0A921 100635 nfsd nfsd: service = mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wai= t_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0Afork_trampoline+0xe=0A921 100636 nfsd nfsd: service mi_= switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_si= g+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afo= rk_trampoline+0xe=0A921 100637 nfsd nfsd: service mi_switc= h+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x1= 6a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_tr= ampoline+0xe=0A921 100638 nfsd nfsd: service mi_switch+0xe= 1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0A= svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampol= ine+0xe=0A921 100639 nfsd nfsd: service mi_switch+0xe1=0As= leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_r= un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0= xe=0A921 100640 nfsd nfsd: service mi_switch+0xe1=0Asleepq= _catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A= 921 100641 nfsd nfsd: service mi_switch+0xe1=0Asleepq_catc= h_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_interna= l+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 1= 00642 nfsd nfsd: service mi_switch+0xe1=0Asleepq_catch_sig= nals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100643= nfsd nfsd: service mi_switch+0xe1=0Asleepq_catch_signals+= 0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e sv= c_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100644 nfsd= nfsd: service mi_switch+0xe1=0Asleepq_catch_signals+0xab = sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100645 nfsd = nfsd: service mi_switch+0xe1=0Asleepq_catch_signals+0xab sleep= q_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_s= tart+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100646 nfsd = nfsd: service mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100647 nfsd nf= sd: service mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig= +0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb f= ork_exit+0x9a=0Afork_trampoline+0xe=0A921 100648 nfsd nfsd: s= ervice mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_= exit+0x9a=0Afork_trampoline+0xe=0A921 100649 nfsd nfsd: servi= ce mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_c= v_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0Afork_trampoline+0xe=0A921 100650 nfsd nfsd: service = mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wai= t_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0Afork_trampoline+0xe=0A921 100651 nfsd nfsd: service mi_= switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_si= g+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afo= rk_trampoline+0xe=0A921 100652 nfsd nfsd: service mi_switc= h+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x1= 6a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_tr= ampoline+0xe=0A921 100653 nfsd nfsd: service mi_switch+0xe= 1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0A= svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampol= ine+0xe=0A921 100654 nfsd nfsd: service mi_switch+0xe1=0As= leepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_r= un_internal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0= xe=0A921 100655 nfsd nfsd: service mi_switch+0xe1=0Asleepq= _catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_in= ternal+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A= 921 100656 nfsd nfsd: service mi_switch+0xe1=0Asleepq_catc= h_signals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_interna= l+0x87e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 1= 00657 nfsd nfsd: service mi_switch+0xe1=0Asleepq_catch_sig= nals+0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100658= nfsd nfsd: service mi_switch+0xe1=0Asleepq_catch_signals+= 0xab sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e sv= c_thread_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100659 nfsd= nfsd: service mi_switch+0xe1=0Asleepq_catch_signals+0xab = sleepq_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100660 nfsd = nfsd: service mi_switch+0xe1=0Asleepq_catch_signals+0xab sleep= q_wait_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_s= tart+0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100661 nfsd = nfsd: service mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wai= t_sig+0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a=0Afork_trampoline+0xe=0A921 100662 nfsd nf= sd: service mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig= +0xf=0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb f= ork_exit+0x9a=0Afork_trampoline+0xe=0A921 100663 nfsd nfsd: s= ervice mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf= =0A_cv_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_= exit+0x9a=0Afork_trampoline+0xe=0A921 100664 nfsd nfsd: servi= ce mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_c= v_wait_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+= 0x9a=0Afork_trampoline+0xe=0A921 100665 nfsd nfsd: service = mi_switch+0xe1=0Asleepq_catch_signals+0xab sleepq_wait_sig+0xf=0A_cv_wai= t_sig+0x16a=0Asvc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a= =0Afork_trampoline+0xe=0A921 100666 nfsd nfsd: service mi_= switch+0xe1=0Asleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9= b=0Anfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8=0Anfsrvd_dorpc+0xc76=0A= nfssvc_program+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0xb=0Afork= _exit+0x9a fork_trampoline+0xe=0A=0ARegards,=0A=0ALo=C3=AFc Blot,=0AUNIX = Systems, Network and Security Engineer=0Ahttp://www.unix-experience.fr=0A= =0A15 d=C3=A9cembre 2014 15:18 "Rick Macklem" <rmacklem@uoguelph.ca> a=0A= =C3=A9crit: =0A=0ALoic Blot wrote:=0A=0A> For more informations, here is = procstat -kk on nfsd, if you=0A> need=0A> more=0A> hot datas, tell me.=0A= > =0A> Regards, PID TID COMM TDNAME KSTACK=0A> = 918 100529 nfsd nfsd: master mi_switch+0xe1=0A> sleepq_wa= it+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1= _APV+0xab _vn_lock+0x43=0A> zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_= fhtovp+0xc8 nfsrvd_dorpc+0x917=0A> nfssvc_program+0x554 svc_run_internal+= 0xc77 svc_run+0x1de=0A> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x= 9c=0A> amd64_syscall+0x351=0A=0AWell, most of the threads are stuck like = this one, waiting for=0Aa=0Avnode=0Alock in ZFS. All of them appear to be= in zfs_fhtovp().=0AI`m not a ZFS guy, so I can`t help much. I`ll try cha= nging the=0Asubject line=0Ato include ZFS vnode lock, so maybe the ZFS gu= ys will take a=0Alook.=0A=0AThe only thing I`ve seen suggested is trying:= =0Asysctl vfs.lookup_shared=3D0=0Ato disable shared vop_lookup()s. Appare= ntly zfs_lookup()=0Adoesn`t=0Aobey the vnode locking rules for lookup and= rename, according=0Ato=0Athe posting I saw.=0A=0AI`ve added a couple of = comments about the other threads below,=0Abut=0Athey are all either waiti= ng for an RPC request or waiting for=0Athe=0Athreads stuck on the ZFS vno= de lock to complete.=0A=0Arick=0A=0A> 918 100564 nfsd nfsd: s= ervice mi_switch+0xe1=0A> sleepq_catch_signals+0xab sleepq_wait_sig+0x= f=0A> _cv_wait_sig+0x16a=0A> svc_run_internal+0x87e svc_thread_start+0xb = fork_exit+0x9a=0A> fork_trampoline+0xe=0A=0AFyi, this thread is just wait= ing for an RPC to arrive. (Normal)=0A=0A> 918 100565 nfsd nfs= d: service mi_switch+0xe1=0A> sleepq_catch_signals+0xab sleepq_wait_si= g+0xf=0A> _cv_wait_sig+0x16a=0A> svc_run_internal+0x87e svc_thread_start+= 0xb fork_exit+0x9a=0A> fork_trampoline+0xe=0A> 918 100566 nfsd = nfsd: service mi_switch+0xe1=0A> sleepq_catch_signals+0xab sleepq_wa= it_sig+0xf=0A> _cv_wait_sig+0x16a=0A> svc_run_internal+0x87e svc_thread_s= tart+0xb fork_exit+0x9a=0A> fork_trampoline+0xe=0A> 918 100567 nfsd = nfsd: service mi_switch+0xe1=0A> sleepq_catch_signals+0xab slee= pq_wait_sig+0xf=0A> _cv_wait_sig+0x16a=0A> svc_run_internal+0x87e svc_thr= ead_start+0xb fork_exit+0x9a=0A> fork_trampoline+0xe=0A> 918 100568 nfsd = nfsd: service mi_switch+0xe1=0A> sleepq_catch_signals+0xab= sleepq_wait_sig+0xf=0A> _cv_wait_sig+0x16a=0A> svc_run_internal+0x87e sv= c_thread_start+0xb fork_exit+0x9a=0A> fork_trampoline+0xe=0A> 918 100569 = nfsd nfsd: service mi_switch+0xe1=0A> sleepq_catch_signals= +0xab sleepq_wait_sig+0xf=0A> _cv_wait_sig+0x16a=0A> svc_run_internal+0x8= 7e svc_thread_start+0xb fork_exit+0x9a=0A> fork_trampoline+0xe=0A> 918 10= 0570 nfsd nfsd: service mi_switch+0xe1=0A> sleepq_catch_si= gnals+0xab sleepq_wait_sig+0xf=0A> _cv_wait_sig+0x16a=0A> svc_run_interna= l+0x87e svc_thread_start+0xb fork_exit+0x9a=0A> fork_trampoline+0xe=0A> 9= 18 100571 nfsd nfsd: service mi_switch+0xe1=0A> sleepq_wai= t+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A> nfsrvd_dorpc+0x316= nfssvc_program+0x554 svc_run_internal+0xc77=0A> svc_thread_start+0xb for= k_exit+0x9a fork_trampoline+0xe=0A> 918 100572 nfsd nfsd: ser= vice mi_switch+0xe1=0A> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 n= fsv4_lock+0x9b=0A> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8=0A> nfsr= vd_dorpc+0xc76=0A> nfssvc_program+0x554 svc_run_internal+0xc77=0A> svc_th= read_start+0xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A=0AThis one (and = a few others) are waiting for the nfsv4_lock.=0AThis=0Ahappens=0Abecause = other threads are stuck with RPCs in progress. (ie. The=0Aones=0Awaiting = on the vnode lock in zfs_fhtovp().)=0AFor these, the RPC needs to lock ou= t other threads to do the=0Aoperation,=0Aso it waits for the nfsv4_lock()= which can exclusively lock the=0ANFSv4=0Adata structures once all other = nfsd threads complete their RPCs=0Ain=0Aprogress.=0A=0A> 918 100573 nfsd = nfsd: service mi_switch+0xe1=0A> sleepq_wait+0x3a _sleep+0= x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0A> nfsrvd_dorpc+0x316 nfssvc_program= +0x554 svc_run_internal+0xc77=0A> svc_thread_start+0xb fork_exit+0x9a for= k_trampoline+0xe=0A=0ASame as above.=0A=0A> 918 100574 nfsd n= fsd: service mi_switch+0xe1=0A> sleepq_wait+0x3a sleeplk+0x15d __lockm= gr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A> z= fs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x91= 7=0A> nfssvc_program+0x554 svc_run_internal+0xc77=0A> svc_thread_start+0x= b=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 918 100575 nfsd = nfsd: service mi_switch+0xe1=0A> sleepq_wait+0x3a sleeplk+0x15d __lock= mgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A> = zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x9= 17=0A> nfssvc_program+0x554 svc_run_internal+0xc77=0A> svc_thread_start+0= xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 918 100576 nfsd = nfsd: service mi_switch+0xe1=0A> sleepq_wait+0x3a sleeplk+0x15d __loc= kmgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A>= zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x= 917=0A> nfssvc_program+0x554 svc_run_internal+0xc77=0A> svc_thread_start+= 0xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 918 100577 nfsd = nfsd: service mi_switch+0xe1=0A> sleepq_wait+0x3a sleeplk+0x15d __lo= ckmgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A= > zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0= x917=0A> nfssvc_program+0x554 svc_run_internal+0xc77=0A> svc_thread_start= +0xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 918 100578 nfsd = nfsd: service mi_switch+0xe1=0A> sleepq_wait+0x3a sleeplk+0x15d __l= ockmgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43= =0A> zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorp= c+0x917=0A> nfssvc_program+0x554 svc_run_internal+0xc77=0A> svc_thread_st= art+0xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 918 100579 nfsd = nfsd: service mi_switch+0xe1=0A> sleepq_wait+0x3a sleeplk+0x15d = __lockmgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x4= 3=0A> zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dor= pc+0x917=0A> nfssvc_program+0x554 svc_run_internal+0xc77=0A> svc_thread_s= tart+0xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 918 100580 nfsd = nfsd: service mi_switch+0xe1=0A> sleepq_wait+0x3a sleeplk+0x15d= __lockmgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x= 43=0A> zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_do= rpc+0x917=0A> nfssvc_program+0x554 svc_run_internal+0xc77=0A> svc_thread_= start+0xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 918 100581 nfsd = nfsd: service mi_switch+0xe1=0A> sleepq_wait+0x3a sleeplk+0x15= d __lockmgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0= x43=0A> zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_d= orpc+0x917=0A> nfssvc_program+0x554 svc_run_internal+0xc77=0A> svc_thread= _start+0xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 918 100582 nfsd = nfsd: service mi_switch+0xe1=0A> sleepq_wait+0x3a sleeplk+0x1= 5d __lockmgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+= 0x43=0A> zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_= dorpc+0x917=0A> nfssvc_program+0x554 svc_run_internal+0xc77=0A> svc_threa= d_start+0xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 918 100583 nfsd = nfsd: service mi_switch+0xe1=0A> sleepq_wait+0x3a sleeplk+0x= 15d __lockmgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock= +0x43=0A> zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd= _dorpc+0x917=0A> nfssvc_program+0x554 svc_run_internal+0xc77=0A> svc_thre= ad_start+0xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 918 100584 nfsd = nfsd: service mi_switch+0xe1=0A> sleepq_wait+0x3a sleeplk+0= x15d __lockmgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_loc= k+0x43=0A> zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrv= d_dorpc+0x917=0A> nfssvc_program+0x554 svc_run_internal+0xc77=0A> svc_thr= ead_start+0xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 918 100585 nfsd = nfsd: service mi_switch+0xe1=0A> sleepq_wait+0x3a sleeplk+= 0x15d __lockmgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lo= ck+0x43=0A> zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsr= vd_dorpc+0x917=0A> nfssvc_program+0x554 svc_run_internal+0xc77=0A> svc_th= read_start+0xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 918 100586 nfsd= nfsd: service mi_switch+0xe1=0A> sleepq_wait+0x3a sleeplk= +0x15d __lockmgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_l= ock+0x43=0A> zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfs= rvd_dorpc+0x917=0A> nfssvc_program+0x554 svc_run_internal+0xc77=0A> svc_t= hread_start+0xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 918 100587 nfs= d nfsd: service mi_switch+0xe1=0A> sleepq_wait+0x3a sleepl= k+0x15d __lockmgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_= lock+0x43=0A> zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nf= srvd_dorpc+0x917=0A> nfssvc_program+0x554 svc_run_internal+0xc77=0A> svc_= thread_start+0xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 918 100588 nf= sd nfsd: service mi_switch+0xe1=0A> sleepq_wait+0x3a sleep= lk+0x15d __lockmgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn= _lock+0x43=0A> zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 n= fsrvd_dorpc+0x917=0A> nfssvc_program+0x554 svc_run_internal+0xc77=0A> svc= _thread_start+0xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 918 100589 n= fsd nfsd: service mi_switch+0xe1=0A> sleepq_wait+0x3a slee= plk+0x15d __lockmgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _v= n_lock+0x43=0A> zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 = nfsrvd_dorpc+0x917=0A> nfssvc_program+0x554 svc_run_internal+0xc77=0A> sv= c_thread_start+0xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 918 100590 = nfsd nfsd: service mi_switch+0xe1=0A> sleepq_wait+0x3a sle= eplk+0x15d __lockmgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _= vn_lock+0x43=0A> zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8= nfsrvd_dorpc+0x917=0A> nfssvc_program+0x554 svc_run_internal+0xc77=0A> s= vc_thread_start+0xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 918 100591= nfsd nfsd: service mi_switch+0xe1=0A> sleepq_wait+0x3a sl= eeplk+0x15d __lockmgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_APV+0xab = _vn_lock+0x43=0A> zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc= 8 nfsrvd_dorpc+0x917=0A> nfssvc_program+0x554 svc_run_internal+0xc77=0A> = svc_thread_start+0xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 918 10059= 2 nfsd nfsd: service mi_switch+0xe1=0A> sleepq_wait+0x3a s= leeplk+0x15d __lockmgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_APV+0xab= _vn_lock+0x43=0A> zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fhtovp+0x= c8 nfsrvd_dorpc+0x917=0A> nfssvc_program+0x554 svc_run_internal+0xc77=0A>= svc_thread_start+0xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 918 1005= 93 nfsd nfsd: service mi_switch+0xe1=0A> sleepq_wait+0x3a = sleeplk+0x15d __lockmgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_APV+0xa= b _vn_lock+0x43=0A> zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fhtovp+0= xc8 nfsrvd_dorpc+0x917=0A> nfssvc_program+0x554 svc_run_internal+0xc77=0A= > svc_thread_start+0xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 918 100= 594 nfsd nfsd: service mi_switch+0xe1=0A> sleepq_wait+0x3a= sleeplk+0x15d __lockmgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_APV+0x= ab _vn_lock+0x43=0A> zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fhtovp+= 0xc8 nfsrvd_dorpc+0x917=0A> nfssvc_program+0x554 svc_run_internal+0xc77= =0A> svc_thread_start+0xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 918 = 100595 nfsd nfsd: service mi_switch+0xe1=0A> sleepq_wait+0= x3a sleeplk+0x15d __lockmgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_APV= +0xab _vn_lock+0x43=0A> zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fhto= vp+0xc8 nfsrvd_dorpc+0x917=0A> nfssvc_program+0x554 svc_run_internal+0xc7= 7=0A> svc_thread_start+0xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 918= 100596 nfsd nfsd: service mi_switch+0xe1=0A> sleepq_wait+= 0x3a sleeplk+0x15d __lockmgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_AP= V+0xab _vn_lock+0x43=0A> zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fht= ovp+0xc8 nfsrvd_dorpc+0x917=0A> nfssvc_program+0x554 svc_run_internal+0xc= 77=0A> svc_thread_start+0xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 91= 8 100597 nfsd nfsd: service mi_switch+0xe1=0A> sleepq_wait= +0x3a sleeplk+0x15d __lockmgr_args+0x902=0A> vop_stdlock+0x3c VOP_LOCK1_A= PV+0xab _vn_lock+0x43=0A> zfs_fhtovp+0x38d=0A> nfsvno_fhtovp+0x7c nfsd_fh= tovp+0xc8 nfsrvd_dorpc+0x917=0A> nfssvc_program+0x554 svc_run_internal+0x= c77=0A> svc_thread_start+0xb=0A> fork_exit+0x9a fork_trampoline+0xe=0A> 9= 18 100598 nfsd=20=20=20=20=20=20=20 nfsd: service mi_switch+0xe1= =0A> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A> vop_stdlock+= 0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A> zfs_fhtovp+0x38d=0A> nfsvno_fht= ovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A> nfssvc_program+0x554 svc= _run_internal+0xc77=0A> svc_thread_start+0xb=0A> fork_exit+0x9a fork_tram= poline+0xe=0A> 918 100599 nfsd nfsd: service mi_switch+0xe= 1=0A> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A> vop_stdlock= +0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A> zfs_fhtovp+0x38d=0A> nfsvno_fh= tovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A> nfssvc_program+0x554 sv= c_run_internal+0xc77=0A> svc_thread_start+0xb=0A> fork_exit+0x9a fork_tra= mpoline+0xe=0A> 918 100600 nfsd nfsd: service mi_switch+0x= e1=0A> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A> vop_stdloc= k+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A> zfs_fhtovp+0x38d=0A> nfsvno_f= htovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A> nfssvc_program+0x554 s= vc_run_internal+0xc77=0A> svc_thread_start+0xb=0A> fork_exit+0x9a fork_tr= ampoline+0xe=0A> 918 100601 nfsd nfsd: service mi_switch+0= xe1=0A> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A> vop_stdlo= ck+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A> zfs_fhtovp+0x38d=0A> nfsvno_= fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A> nfssvc_program+0x554 = svc_run_internal+0xc77=0A> svc_thread_start+0xb=0A> fork_exit+0x9a fork_t= rampoline+0xe=0A> 918 100602 nfsd nfsd: service mi_switch+= 0xe1=0A> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A> vop_stdl= ock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A> zfs_fhtovp+0x38d=0A> nfsvno= _fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A> nfssvc_program+0x554= svc_run_internal+0xc77=0A> svc_thread_start+0xb=0A> fork_exit+0x9a fork_= trampoline+0xe=0A> 918 100603 nfsd nfsd: service mi_switch= +0xe1=0A> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A> vop_std= lock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A> zfs_fhtovp+0x38d=0A> nfsvn= o_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A> nfssvc_program+0x55= 4 svc_run_internal+0xc77=0A> svc_thread_start+0xb=0A> fork_exit+0x9a fork= _trampoline+0xe=0A> 918 100604 nfsd nfsd: service mi_switc= h+0xe1=0A> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A> vop_st= dlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A> zfs_fhtovp+0x38d=0A> nfsv= no_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A> nfssvc_program+0x5= 54 svc_run_internal+0xc77=0A> svc_thread_start+0xb=0A> fork_exit+0x9a for= k_trampoline+0xe=0A> 918 100605 nfsd nfsd: service mi_swit= ch+0xe1=0A> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A> vop_s= tdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A> zfs_fhtovp+0x38d=0A> nfs= vno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A> nfssvc_program+0x= 554 svc_run_internal+0xc77=0A> svc_thread_start+0xb=0A> fork_exit+0x9a fo= rk_trampoline+0xe=0A> 918 100606 nfsd nfsd: service mi_swi= tch+0xe1=0A> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A> vop_= stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A> zfs_fhtovp+0x38d=0A> nf= svno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A> nfssvc_program+0= x554 svc_run_internal+0xc77=0A> svc_thread_start+0xb=0A> fork_exit+0x9a f= ork_trampoline+0xe=0A> 918 100607 nfsd nfsd: service mi_sw= itch+0xe1=0A> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0A> vop= _stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0A> zfs_fhtovp+0x38d=0A> n= fsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A> nfssvc_program+= 0x554 svc_run_internal+0xc77=0A> svc_thread_start+0xb=0A> fork_exit+0x9a = fork_trampoline+0xe=0A=0ALots more waiting for the ZFS vnode lock in zfs_= fhtovp().=0A=0A=0A918 100608 nfsd nfsd: service mi_switch+= 0xe1=0Asleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0Anfs= rv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1=0Anfsrvd_dorp= c+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0x= b fork_exit+0x9a fork_trampoline+0xe=0A918 100609 nfsd nfsd: = service mi_switch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lockmgr_args= +0x902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azfs_fhtovp+0= x38d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0Anfssvc_pr= ogram+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0xb=0Afork_exit+0x9= a fork_trampoline+0xe=0A918 100610 nfsd nfsd: service mi_s= witch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e=0Avop_st= dlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Anfsvno_advlock+0x119 nfsrv_= dolocal+0x84 nfsrv_lockctrl+0x14ad=0Anfsrvd_locku+0x283 nfsrvd_dorpc+0xec= 6 nfssvc_program+0x554=0Asvc_run_internal+0xc77 svc_thread_start+0xb fork= _exit+0x9a=0Afork_trampoline+0xe=0A918 100611 nfsd nfsd: serv= ice mi_switch+0xe1=0Asleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv= 4_lock+0x9b=0Anfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0x= c77=0Asvc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A918 10061= 2 nfsd nfsd: service mi_switch+0xe1=0Asleepq_wait+0x3a _sl= eep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0Anfsrvd_dorpc+0x316 nfssvc_prog= ram+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0xb fork_exit+0x9a fo= rk_trampoline+0xe=0A918 100613 nfsd nfsd: service mi_switc= h+0xe1=0Asleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0An= fsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77=0Asvc_threa= d_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A918 100614 nfsd = nfsd: service mi_switch+0xe1=0Asleepq_wait+0x3a _sleep+0x287 nfsms= leep+0x66 nfsv4_lock+0x9b=0Anfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_r= un_internal+0xc77=0Asvc_thread_start+0xb fork_exit+0x9a fork_trampoline+0= xe=0A918 100615 nfsd nfsd: service mi_switch+0xe1=0Asleepq= _wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0Anfsrvd_dorpc+0x3= 16 nfssvc_program+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0xb for= k_exit+0x9a fork_trampoline+0xe=0A918 100616 nfsd nfsd: servi= ce mi_switch+0xe1=0Asleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4= _lock+0x9b=0Anfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc= 77=0Asvc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A918 100617= nfsd nfsd: service mi_switch+0xe1=0Asleepq_wait+0x3a _sle= ep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0Anfsrvd_dorpc+0x316 nfssvc_progr= am+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0xb fork_exit+0x9a for= k_trampoline+0xe=0A918 100618 nfsd nfsd: service mi_switch= +0xe1=0Asleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0Anf= srvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77=0Asvc_thread= _start+0xb fork_exit+0x9a fork_trampoline+0xe=0A918 100619 nfsd = nfsd: service mi_switch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __loc= kmgr_args+0x902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azfs= _fhtovp+0x38d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0A= nfssvc_program+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0xb=0Afork= _exit+0x9a fork_trampoline+0xe=0A918 100620 nfsd nfsd: servic= e mi_switch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902= =0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azfs_fhtovp+0x38d= =0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0Anfssvc_progra= m+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0xb=0Afork_exit+0x9a fo= rk_trampoline+0xe=0A918 100621 nfsd nfsd: service mi_switc= h+0xe1=0Asleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b=0An= fsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77=0Asvc_threa= d_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A918 100622 nfsd = nfsd: service mi_switch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lo= ckmgr_args+0x902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azf= s_fhtovp+0x38d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917= =0Anfssvc_program+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0xb=0Af= ork_exit+0x9a fork_trampoline+0xe=0A918 100623 nfsd nfsd: ser= vice mi_switch+0xe1=0Asleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfs= v4_lock+0x9b=0Anfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0= xc77=0Asvc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe=0A918 1006= 24 nfsd nfsd: service mi_switch+0xe1=0Asleepq_wait+0x3a sl= eeplk+0x15d __lockmgr_args+0x902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _v= n_lock+0x43=0Azfs_fhtovp+0x38d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsr= vd_dorpc+0x917=0Anfssvc_program+0x554 svc_run_internal+0xc77=0Asvc_thread= _start+0xb=0Afork_exit+0x9a fork_trampoline+0xe=0A918 100625 nfsd = nfsd: service mi_switch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __l= ockmgr_args+0x902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Az= fs_fhtovp+0x38d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917= =0Anfssvc_program+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0xb=0Af= ork_exit+0x9a fork_trampoline+0xe=0A918 100626 nfsd nfsd: ser= vice mi_switch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x= 902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azfs_fhtovp+0x38= d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0Anfssvc_progr= am+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0xb=0Afork_exit+0x9a f= ork_trampoline+0xe=0A918 100627 nfsd nfsd: service mi_swit= ch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0Avop_stdlo= ck+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azfs_fhtovp+0x38d=0Anfsvno_fhto= vp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0Anfssvc_program+0x554 svc_ru= n_internal+0xc77=0Asvc_thread_start+0xb=0Afork_exit+0x9a fork_trampoline+= 0xe=0A918 100628 nfsd nfsd: service mi_switch+0xe1=0Asleep= q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0Avop_stdlock+0x3c VOP_LOC= K1_APV+0xab _vn_lock+0x43=0Azfs_fhtovp+0x38d=0Anfsvno_fhtovp+0x7c nfsd_fh= tovp+0xc8 nfsrvd_dorpc+0x917=0Anfssvc_program+0x554 svc_run_internal+0xc7= 7=0Asvc_thread_start+0xb=0Afork_exit+0x9a fork_trampoline+0xe=0A918 10062= 9 nfsd nfsd: service mi_switch+0xe1=0Asleepq_wait+0x3a sle= eplk+0x15d __lockmgr_args+0x902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn= _lock+0x43=0Azfs_fhtovp+0x38d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrv= d_dorpc+0x917=0Anfssvc_program+0x554 svc_run_internal+0xc77=0Asvc_thread_= start+0xb=0Afork_exit+0x9a fork_trampoline+0xe=0A918 100630 nfsd = nfsd: service mi_switch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lo= ckmgr_args+0x902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azf= s_fhtovp+0x38d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917= =0Anfssvc_program+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0xb=0Af= ork_exit+0x9a fork_trampoline+0xe=0A918 100631 nfsd nfsd: ser= vice mi_switch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x= 902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azfs_fhtovp+0x38= d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0Anfssvc_progr= am+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0xb=0Afork_exit+0x9a f= ork_trampoline+0xe=0A918 100632 nfsd nfsd: service mi_swit= ch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0Avop_stdlo= ck+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azfs_fhtovp+0x38d=0Anfsvno_fhto= vp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0Anfssvc_program+0x554 svc_ru= n_internal+0xc77=0Asvc_thread_start+0xb=0Afork_exit+0x9a fork_trampoline+= 0xe=0A918 100633 nfsd nfsd: service mi_switch+0xe1=0Asleep= q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0Avop_stdlock+0x3c VOP_LOC= K1_APV+0xab _vn_lock+0x43=0Azfs_fhtovp+0x38d=0Anfsvno_fhtovp+0x7c nfsd_fh= tovp+0xc8 nfsrvd_dorpc+0x917=0Anfssvc_program+0x554 svc_run_internal+0xc7= 7=0Asvc_thread_start+0xb=0Afork_exit+0x9a fork_trampoline+0xe=0A918 10063= 4 nfsd nfsd: service mi_switch+0xe1=0Asleepq_wait+0x3a sle= eplk+0x15d __lockmgr_args+0x902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn= _lock+0x43=0Azfs_fhtovp+0x38d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrv= d_dorpc+0x917=0Anfssvc_program+0x554 svc_run_internal+0xc77=0Asvc_thread_= start+0xb=0Afork_exit+0x9a fork_trampoline+0xe=0A918 100635 nfsd = nfsd: service mi_switch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lo= ckmgr_args+0x902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azf= s_fhtovp+0x38d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917= =0Anfssvc_program+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0xb=0Af= ork_exit+0x9a fork_trampoline+0xe=0A918 100636 nfsd nfsd: ser= vice mi_switch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x= 902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azfs_fhtovp+0x38= d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0Anfssvc_progr= am+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0xb=0Afork_exit+0x9a f= ork_trampoline+0xe=0A918 100637 nfsd nfsd: service mi_swit= ch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0Avop_stdlo= ck+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azfs_fhtovp+0x38d=0Anfsvno_fhto= vp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0Anfssvc_program+0x554 svc_ru= n_internal+0xc77=0Asvc_thread_start+0xb=0Afork_exit+0x9a fork_trampoline+= 0xe=0A918 100638 nfsd nfsd: service mi_switch+0xe1=0Asleep= q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0Avop_stdlock+0x3c VOP_LOC= K1_APV+0xab _vn_lock+0x43=0Azfs_fhtovp+0x38d=0Anfsvno_fhtovp+0x7c nfsd_fh= tovp+0xc8 nfsrvd_dorpc+0x917=0Anfssvc_program+0x554 svc_run_internal+0xc7= 7=0Asvc_thread_start+0xb=0Afork_exit+0x9a fork_trampoline+0xe=0A918 10063= 9 nfsd nfsd: service mi_switch+0xe1=0Asleepq_wait+0x3a sle= eplk+0x15d __lockmgr_args+0x902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn= _lock+0x43=0Azfs_fhtovp+0x38d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrv= d_dorpc+0x917=0Anfssvc_program+0x554 svc_run_internal+0xc77=0Asvc_thread_= start+0xb=0Afork_exit+0x9a fork_trampoline+0xe=0A918 100640 nfsd = nfsd: service mi_switch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lo= ckmgr_args+0x902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azf= s_fhtovp+0x38d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917= =0Anfssvc_program+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0xb=0Af= ork_exit+0x9a fork_trampoline+0xe=0A918 100641 nfsd nfsd: ser= vice mi_switch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x= 902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azfs_fhtovp+0x38= d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0Anfssvc_progr= am+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0xb=0Afork_exit+0x9a f= ork_trampoline+0xe=0A918 100642 nfsd nfsd: service mi_swit= ch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0Avop_stdlo= ck+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azfs_fhtovp+0x38d=0Anfsvno_fhto= vp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0Anfssvc_program+0x554 svc_ru= n_internal+0xc77=0Asvc_thread_start+0xb=0Afork_exit+0x9a fork_trampoline+= 0xe=0A918 100643 nfsd nfsd: service mi_switch+0xe1=0Asleep= q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0Avop_stdlock+0x3c VOP_LOC= K1_APV+0xab _vn_lock+0x43=0Azfs_fhtovp+0x38d=0Anfsvno_fhtovp+0x7c nfsd_fh= tovp+0xc8 nfsrvd_dorpc+0x917=0Anfssvc_program+0x554 svc_run_internal+0xc7= 7=0Asvc_thread_start+0xb=0Afork_exit+0x9a fork_trampoline+0xe=0A918 10064= 4 nfsd nfsd: service mi_switch+0xe1=0Asleepq_wait+0x3a sle= eplk+0x15d __lockmgr_args+0x902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn= _lock+0x43=0Azfs_fhtovp+0x38d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrv= d_dorpc+0x917=0Anfssvc_program+0x554 svc_run_internal+0xc77=0Asvc_thread_= start+0xb=0Afork_exit+0x9a fork_trampoline+0xe=0A918 100645 nfsd = nfsd: service mi_switch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lo= ckmgr_args+0x902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azf= s_fhtovp+0x38d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917= =0Anfssvc_program+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0xb=0Af= ork_exit+0x9a fork_trampoline+0xe=0A918 100646 nfsd nfsd: ser= vice mi_switch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x= 902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azfs_fhtovp+0x38= d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0Anfssvc_progr= am+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0xb=0Afork_exit+0x9a f= ork_trampoline+0xe=0A918 100647 nfsd nfsd: service mi_swit= ch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0Avop_stdlo= ck+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azfs_fhtovp+0x38d=0Anfsvno_fhto= vp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0Anfssvc_program+0x554 svc_ru= n_internal+0xc77=0Asvc_thread_start+0xb=0Afork_exit+0x9a fork_trampoline+= 0xe=0A918 100648 nfsd nfsd: service mi_switch+0xe1=0Asleep= q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0Avop_stdlock+0x3c VOP_LOC= K1_APV+0xab _vn_lock+0x43=0Azfs_fhtovp+0x38d=0Anfsvno_fhtovp+0x7c nfsd_fh= tovp+0xc8 nfsrvd_dorpc+0x917=0Anfssvc_program+0x554 svc_run_internal+0xc7= 7=0Asvc_thread_start+0xb=0Afork_exit+0x9a fork_trampoline+0xe=0A918 10064= 9 nfsd nfsd: service mi_switch+0xe1=0Asleepq_wait+0x3a sle= eplk+0x15d __lockmgr_args+0x902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn= _lock+0x43=0Azfs_fhtovp+0x38d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrv= d_dorpc+0x917=0Anfssvc_program+0x554 svc_run_internal+0xc77=0Asvc_thread_= start+0xb=0Afork_exit+0x9a fork_trampoline+0xe=0A918 100650 nfsd = nfsd: service mi_switch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lo= ckmgr_args+0x902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azf= s_fhtovp+0x38d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917= =0Anfssvc_program+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0xb=0Af= ork_exit+0x9a fork_trampoline+0xe=0A918 100651 nfsd nfsd: ser= vice mi_switch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x= 902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azfs_fhtovp+0x38= d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0Anfssvc_progr= am+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0xb=0Afork_exit+0x9a f= ork_trampoline+0xe=0A918 100652 nfsd nfsd: service mi_swit= ch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0Avop_stdlo= ck+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azfs_fhtovp+0x38d=0Anfsvno_fhto= vp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0Anfssvc_program+0x554 svc_ru= n_internal+0xc77=0Asvc_thread_start+0xb=0Afork_exit+0x9a fork_trampoline+= 0xe=0A918 100653 nfsd nfsd: service mi_switch+0xe1=0Asleep= q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0Avop_stdlock+0x3c VOP_LOC= K1_APV+0xab _vn_lock+0x43=0Azfs_fhtovp+0x38d=0Anfsvno_fhtovp+0x7c nfsd_fh= tovp+0xc8 nfsrvd_dorpc+0x917=0Anfssvc_program+0x554 svc_run_internal+0xc7= 7=0Asvc_thread_start+0xb=0Afork_exit+0x9a fork_trampoline+0xe=0A918 10065= 4 nfsd nfsd: service mi_switch+0xe1=0Asleepq_wait+0x3a sle= eplk+0x15d __lockmgr_args+0x902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn= _lock+0x43=0Azfs_fhtovp+0x38d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrv= d_dorpc+0x917=0Anfssvc_program+0x554 svc_run_internal+0xc77=0Asvc_thread_= start+0xb=0Afork_exit+0x9a fork_trampoline+0xe=0A918 100655 nfsd = nfsd: service mi_switch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lo= ckmgr_args+0x902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azf= s_fhtovp+0x38d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917= =0Anfssvc_program+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0xb=0Af= ork_exit+0x9a fork_trampoline+0xe=0A918 100656 nfsd nfsd: ser= vice mi_switch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x= 902=0Avop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azfs_fhtovp+0x38= d=0Anfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0Anfssvc_progr= am+0x554 svc_run_internal+0xc77=0Asvc_thread_start+0xb=0Afork_exit+0x9a f= ork_trampoline+0xe=0A918 100657 nfsd nfsd: service mi_swit= ch+0xe1=0Asleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0Avop_stdlo= ck+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43=0Azfs_fhtovp+0x38d=0Anfsvno_fhto= vp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917=0Anfssvc_program+0x554 svc_ru= n_internal+0xc77=0Asvc_thread_start+0xb=0Afork_exit+0x9a fork_trampoline+= 0xe=0A918 100658 nfsd nfsd: service mi_switch+0xe1=0Asleep= q_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902=0Avop_stdlock+0x3c VOP_LOC= K1_APV+0xab _vn_lock+0x43=0Azfs_fhtovp+0x38d=0Anfsvno_fhtovp+0x7c nfsd_fh= tovp+0xc8 nfsrvd_dorpc+0x917=0Anfssvc_program+0x554 svc_run_internal+0xc7= 7=0Asvc_thread_start+0xb=0Afork_exit+0x9a fork_trampoline+0xe=0A=0ALo=C3= =AFc Blot,=0AUNIX Systems, Network and Security Engineer=0Ahttp://www.uni= x-experience.fr=0A=0A15 d=C3=A9cembre 2014 13:29 "Lo=C3=AFc Blot"=0A<loic= .blot@unix-experience.fr>=0Aa=0A=C3=A9crit: =0A=0AHmmm...=0Anow i'm exper= iencing a deadlock.=0A=0A0 918 915 0 21 0 12352 3372 zfs D - 1:48.64 nfsd= : server=0A(nfsd)=0A=0Athe only issue was to reboot the server, but after= rebooting=0Adeadlock arrives a second time when i=0Astart my jails over = NFS.=0A=0ARegards,=0A=0ALo=C3=AFc Blot,=0AUNIX Systems, Network and Secur= ity Engineer=0Ahttp://www.unix-experience.fr=0A=0A15 d=C3=A9cembre 2014 1= 0:07 "Lo=C3=AFc Blot"=0A<loic.blot@unix-experience.fr>=0Aa=0A=C3=A9crit:= =0A=0AHi Rick,=0Aafter talking with my N+1, NFSv4 is required on our=0Ain= frastructure.=0AI tried to upgrade NFSv4+ZFS=0Aserver from 9.3 to 10.1, i= hope this will resolve some=0Aissues...=0A=0ARegards,=0A=0ALo=C3=AFc Blo= t,=0AUNIX Systems, Network and Security Engineer=0Ahttp://www.unix-experi= ence.fr=0A=0A10 d=C3=A9cembre 2014 15:36 "Lo=C3=AFc Blot"=0A<loic.blot@un= ix-experience.fr>=0Aa=0A=C3=A9crit:=0A=0AHi Rick,=0Athanks for your sugge= stion.=0AFor my locking bug, rpc.lockd is stucked in rpcrecv state on=0At= he=0Aserver. kill -9 doesn't affect the=0Aprocess, it's blocked.... (Stat= e: Ds)=0A=0Afor the performances=0A=0ANFSv3: 60Mbps=0ANFSv4: 45Mbps=0AReg= ards,=0A=0ALo=C3=AFc Blot,=0AUNIX Systems, Network and Security Engineer= =0Ahttp://www.unix-experience.fr=0A=0A10 d=C3=A9cembre 2014 13:56 "Rick M= acklem" <rmacklem@uoguelph.ca>=0Aa=0A=C3=A9crit:=0A=0A=0ALoic Blot wrote:= =0A=0A> Hi Rick,=0A> I'm trying NFSv3.=0A> Some jails are starting very w= ell but now i have an issue=0A> with=0A> lockd=0A> after some minutes:=0A= > =0A> nfs server 10.10.X.8:/jails: lockd not responding=0A> nfs server 1= 0.10.X.8:/jails lockd is alive again=0A> =0A> I look at mbuf, but i seems= there is no problem.=0A=0AWell, if you need locks to be visible across m= ultiple=0Aclients,=0Athen=0AI'm afraid you are stuck with using NFSv4 and= the=0Aperformance=0Ayou=0Aget=0Afrom it. (There is no way to do file han= dle affinity for=0ANFSv4=0Abecause=0Athe read and write ops are buried in= the compound RPC and=0Anot=0Aeasily=0Arecognized.)=0A=0AIf the locks don= 't need to be visible across multiple=0Aclients,=0AI'd=0Asuggest trying t= he "nolockd" option with nfsv3.=0A=0A> Here is my rc.conf on server:=0A> = =0A> nfs_server_enable=3D"YES"=0A> nfsv4_server_enable=3D"YES"=0A> nfsuse= rd_enable=3D"YES"=0A> nfsd_server_flags=3D"-u -t -n 256"=0A> mountd_enabl= e=3D"YES"=0A> mountd_flags=3D"-r"=0A> nfsuserd_flags=3D"-usertimeout 0 -f= orce 20"=0A> rpcbind_enable=3D"YES"=0A> rpc_lockd_enable=3D"YES"=0A> rpc_= statd_enable=3D"YES"=0A> =0A> Here is the client:=0A> =0A> nfsuserd_enabl= e=3D"YES"=0A> nfsuserd_flags=3D"-usertimeout 0 -force 20"=0A> nfscbd_enab= le=3D"YES"=0A> rpc_lockd_enable=3D"YES"=0A> rpc_statd_enable=3D"YES"=0A> = =0A> Have you got an idea ?=0A> =0A> Regards,=0A> =0A> Lo=C3=AFc Blot,=0A= > UNIX Systems, Network and Security Engineer=0A> http://www.unix-experie= nce.fr=0A> =0A> 9 d=C3=A9cembre 2014 04:31 "Rick Macklem" <rmacklem@uogue= lph.ca>=0A> a=0A> =C3=A9crit: =0A>> Loic Blot wrote:=0A>> =0A>>> Hi rick,= =0A>>> =0A>>> I waited 3 hours (no lag at jail launch) and now I do:=0A>>= > sysrc=0A>>> memcached_flags=3D"-v -m 512"=0A>>> Command was very very s= low...=0A>>> =0A>>> Here is a dd over NFS:=0A>>> =0A>>> 601062912 bytes t= ransferred in 21.060679 secs (28539579=0A>>> bytes/sec)=0A>> =0A>> Can yo= u try the same read using an NFSv3 mount?=0A>> (If it runs much faster, y= ou have probably been bitten by=0A>> the=0A>> ZFS=0A>> "sequential vs ran= dom" read heuristic which I've been told=0A>> things=0A>> NFS is doing "r= andom" reads without file handle affinity.=0A>> File=0A>> handle affinity= is very hard to do for NFSv4, so it isn't=0A>> done.)=0A=0AI was actuall= y suggesting that you try the "dd" over nfsv3=0Ato=0Asee=0Ahow=0Athe perf= ormance compared with nfsv4. If you do that, please=0Apost=0Athe=0Acompar= able results.=0A=0ASomeday I would like to try and get ZFS's sequential v= s=0Arandom=0Aread=0Aheuristic modified and any info on what difference in= =0Aperformance=0Athat=0Amight make for NFS would be useful.=0A=0Arick=0A= =0A=0A=0A=0Arick=0A=0A=0AThis is quite slow...=0A=0AYou can found some nf= sstat below (command isn't finished=0Ayet)=0A=0Anfsstat -c -w 1=0A=0AGtAt= tr Lookup Rdlink Read Write Rename Access Rddir=0A0 0 0 0 0 0 0 0=0A4 0 0= 0 0 0 16 0=0A2 0 0 0 0 0 17 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 = 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 4 0 0 0 0 4 0=0A0 0 0 0 0 0 0 0=0A0 0 0= 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A4 0 0 0 0 0 3 0=0A0 0 0 = 0 0 0 3 0=0A37 10 0 8 0 0 14 1=0A18 16 0 4 1 2 4 0=0A78 91 0 82 6 12 30 0= =0A19 18 0 2 2 4 2 0=0A0 0 0 0 2 0 0 0=0A0 0 0 0 0 0 0 0=0AGtAttr Lookup = Rdlink Read Write Rename Access Rddir=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0= =0A0 0 0 0 0 0 0 0=0A0 1 0 0 0 0 1 0=0A4 6 0 0 6 0 3 0=0A2 0 0 0 0 0 0 0= =0A0 0 0 0 0 0 0 0=0A1 0 0 0 0 0 0 0=0A0 0 0 0 1 0 0 0=0A0 0 0 0 0 0 0 0= =0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0= =0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A6 108 0 0 0 0 0 = 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0AGtAttr Lookup Rdlink Read Write R= ename Access Rddir=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0= =0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0= =0A98 54 0 86 11 0 25 0=0A36 24 0 39 25 0 10 1=0A67 8 0 63 63 0 41 0=0A34= 0 0 35 34 0 0 0=0A75 0 0 75 77 0 0 0=0A34 0 0 35 35 0 0 0=0A75 0 0 74 76= 0 0 0=0A33 0 0 34 33 0 0 0=0A0 0 0 0 5 0 0 0=0A0 0 0 0 0 0 6 0=0A11 0 0 = 0 0 0 11 0=0A0 0 0 0 0 0 0 0=0A0 17 0 0 0 0 1 0=0AGtAttr Lookup Rdlink Re= ad Write Rename Access Rddir=0A4 5 0 0 0 0 12 0=0A2 0 0 0 0 0 26 0=0A0 0 = 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0= 0 0 0 0 0=0A0 4 0 0 0 0 4 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0 = 0 0 0 0 0=0A4 0 0 0 0 0 2 0=0A2 0 0 0 0 0 24 0=0A0 0 0 0 0 0 0 0=0A0 0 0 = 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0= 0 0 0 0=0A0 0 0 0 0 0 0 0=0AGtAttr Lookup Rdlink Read Write Rename Acces= s Rddir=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A4 0 0 0 0 0 7 0=0A2 1 0 0 0= 0 1 0=0A0 0 0 0 2 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 6 0 0 0=0A0 0 0 0 0 = 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0= 0 0=0A0 0 0 0 0 0 0 0=0A4 6 0 0 0 0 3 0=0A0 0 0 0 0 0 0 0=0A2 0 0 0 0 0 = 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0= 0=0AGtAttr Lookup Rdlink Read Write Rename Access Rddir=0A0 0 0 0 0 0 0 = 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0= =0A4 71 0 0 0 0 0 0=0A0 1 0 0 0 0 0 0=0A2 36 0 0 0 0 1 0=0A0 0 0 0 0 0 0 = 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A1 0 0 0 0 0 1 0= =0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A79 6 0 79 79 0 2 0=0A25 0 0 25 26 = 0 6 0=0A43 18 0 39 46 0 23 0=0A36 0 0 36 36 0 31 0=0A68 1 0 66 68 0 0 0= =0AGtAttr Lookup Rdlink Read Write Rename Access Rddir=0A36 0 0 36 36 0 0= 0=0A48 0 0 48 49 0 0 0=0A20 0 0 20 20 0 0 0=0A0 0 0 0 0 0 0 0=0A3 14 0 1= 0 0 11 0=0A0 0 0 0 0 0 0 0=0A0 0 0 0 0 0 0 0=0A0 4 0 0 0 0 4 0=0A0 0 0 0= 0 0 0 0=0A4 22 0 0 0 0 16 0=0A2 0 0 0 0 0 23 0=0A=0ARegards,=0A=0ALo=C3= =AFc Blot,=0AUNIX Systems, Network and Security Engineer=0Ahttp://www.uni= x-experience.fr=0A=0A8 d=C3=A9cembre 2014 09:36 "Lo=C3=AFc Blot"=0A<loic.= blot@unix-experience.fr> a=0A=C3=A9crit: =0A> Hi Rick,=0A> I stopped the = jails this week-end and started it this=0A> morning,=0A> i'll=0A> give yo= u some stats this week.=0A> =0A> Here is my nfsstat -m output (with your = rsize/wsize=0A> tweaks)=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A= =0A=0A=0A=0A=0A=0Anfsv4,tcp,resvport,hard,cto,sec=3Dsys,acdirmin=3D3,acdi= rmax=3D60,acregmin=3D5,acregmax=3D60,nametimeo=3D60,negna =0A=0A=0A=0A=0A= =0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A= =0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0Aetimeo=3D60,rsize=3D32768= ,wsize=3D32768,readdirsize=3D32768,readahead=3D1,wcommitsize=3D773136,tim= eout=3D120,retra =0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0As=3D2147483647=0A=0A= On server side my disks are on a raid controller which show a=0A512b=0Avo= lume and write performances=0Aare very honest (dd if=3D/dev/zero of=3D/ja= ils/test.dd bs=3D4096=0Acount=3D100000000 =3D> 450MBps)=0A=0ARegards,=0A= =0ALo=C3=AFc Blot,=0AUNIX Systems, Network and Security Engineer=0Ahttp:/= /www.unix-experience.fr=0A=0A5 d=C3=A9cembre 2014 15:14 "Rick Macklem" <r= macklem@uoguelph.ca> a=0A=C3=A9crit:=0A=0A=0ALoic Blot wrote:=0A=0A=0AHi,= =0Ai'm trying to create a virtualisation environment based on=0Ajails.=0A= Those jails are stored under a big ZFS pool on a FreeBSD=0A9.3=0Awhich=0A= export a NFSv4 volume. This NFSv4 volume was mounted on a=0Abig=0Ahypervi= sor (2 Xeon E5v3 + 128GB memory and 8 ports (but=0Aonly 1=0Awas=0Aused at= this time).=0A=0AThe problem is simple, my hypervisors runs 6 jails (use= d 1%=0Acpu=0Aand=0A10GB RAM approximatively and less than 1MB bandwidth) = and=0Aworks=0Afine at start but the system slows down and after 2-3 days= =0Abecome=0Aunusable. When i look at top command i see 80-100% on=0Asyste= m=0Aand=0Acommands are very very slow. Many process are tagged with=0Anfs= _cl*.=0A=0A=0ATo be honest, I would expect the slowness to be because of= =0Aslow=0Aresponse=0Afrom the NFSv4 server, but if you do:=0A# ps axHl=0A= on a client when it is slow and post that, it would give us=0Asome=0Amore= =0Ainformation on where the client side processes are sitting.=0AIf you a= lso do something like:=0A# nfsstat -c -w 1=0Aand let it run for a while, = that should show you how many=0ARPCs=0Aare=0Abeing done and which ones.= =0A=0A# nfsstat -m=0Awill show you what your mount is actually using.=0AT= he only mount option I can suggest trying is=0A"rsize=3D32768,wsize=3D327= 68",=0Asince some network environments have difficulties with 64K.=0A=0AT= here are a few things you can try on the NFSv4 server side,=0Aif=0Ait=0Aa= ppears=0Athat the clients are generating a large RPC load.=0A- disabling = the DRC cache for TCP by setting=0Avfs.nfsd.cachetcp=3D0=0A- If the serve= r is seeing a large write RPC load, then=0A"sync=3Ddisabled"=0Amight help= , although it does run a risk of data loss when=0Athe=0Aserver=0Acrashes.= =0AThen there are a couple of other ZFS related things (I'm not=0Aa=0AZFS= =0Aguy,=0Abut these have shown up on the mailing lists).=0A- make sure yo= ur volumes are 4K aligned and ashift=3D12 (in=0Acase a=0Adrive=0Athat use= s 4K sectors is pretending to be 512byte sectored)=0A- never run over 70-= 80% full if write performance is an=0Aissue=0A- use a zil on an SSD with = good write performance=0A=0AThe only NFSv4 thing I can tell you is that i= t is known that=0AZFS's=0Aalgorithm for determining sequential vs random = I/O fails for=0ANFSv4=0Aduring writing and this can be a performance hit.= The only=0Aworkaround=0Ais to use NFSv3 mounts, since file handle affini= ty=0Aapparently=0Afixes=0Athe problem and this is only done for NFSv3.=0A= =0Arick=0A=0A=0AI saw that there are TSO issues with igb then i'm trying = to=0Adisable=0Ait with sysctl but the situation wasn't solved.=0A=0ASomeo= ne has got ideas ? I can give you more informations if=0Ayou=0Aneed.=0A= =0AThanks in advance.=0ARegards,=0A=0ALo=C3=AFc Blot,=0AUNIX Systems, Net= work and Security Engineer=0Ahttp://www.unix-experience.fr=0A____________= ___________________________________=0Afreebsd-fs@freebsd.org mailing list= =0Ahttp://lists.freebsd.org/mailman/listinfo/freebsd-fs=0ATo unsubscribe,= send any mail to=0A"freebsd-fs-unsubscribe@freebsd.org"=0A=0A=0A=0A=0A__= _____________________________________________=0Afreebsd-fs@freebsd.org ma= iling list=0Ahttp://lists.freebsd.org/mailman/listinfo/freebsd-fs=0ATo un= subscribe, send any mail to=0A"freebsd-fs-unsubscribe@freebsd.org"=0A=0A_= ______________________________________________=0Afreebsd-fs@freebsd.org m= ailing list=0Ahttp://lists.freebsd.org/mailman/listinfo/freebsd-fs=0ATo u= nsubscribe, send any mail to=0A"freebsd-fs-unsubscribe@freebsd.org"=0A=0A= _______________________________________________=0Afreebsd-fs@freebsd.org = mailing list=0Ahttp://lists.freebsd.org/mailman/listinfo/freebsd-fs=0ATo = unsubscribe, send any mail to=0A"freebsd-fs-unsubscribe@freebsd.org"=0A__= _____________________________________________=0Afreebsd-fs@freebsd.org ma= iling list=0Ahttp://lists.freebsd.org/mailman/listinfo/freebsd-fs=0ATo un= subscribe, send any mail to=0A"freebsd-fs-unsubscribe@freebsd.org"=0A=0A= =0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A__________________________________= _____________=0Afreebsd-fs@freebsd.org mailing list=0Ahttp://lists.freebs= d.org/mailman/listinfo/freebsd-fs=0ATo unsubscribe, send any mail to "fre= ebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?811d455b0bcaeb43711e8108c96d4f2b>