Date: Wed, 11 Aug 2010 19:00:27 -0700 (PDT) From: alan bryan <alan.bryan@yahoo.com> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: freebsd-stable@freebsd.org Subject: Re: FreeBSD 8.1-Release NFSD hang in rc_lo state Message-ID: <719967.84603.qm@web50506.mail.re2.yahoo.com> In-Reply-To: <1347461186.514707.1281494117995.JavaMail.root@erie.cs.uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
--- On Tue, 8/10/10, Rick Macklem <rmacklem@uoguelph.ca> wrote:=0A=0A> From=
: Rick Macklem <rmacklem@uoguelph.ca>=0A> Subject: Re: FreeBSD 8.1-Release =
NFSD hang in rc_lo state=0A> To: "alan bryan" <alan.bryan@yahoo.com>=0A> Cc=
: freebsd-stable@freebsd.org=0A> Date: Tuesday, August 10, 2010, 7:35 PM=0A=
> > I'm doing some testing with=0A> loading up a new storage server. I have=
=0A> > couple ZFS filesystems exported via NFS over UDP.=0A> > =0A> > I hav=
e a client machine (8.1 also) that has mounted=0A> those filesystems=0A> > =
along with some test PHP scripts that are doing a ton=0A> of=0A> > read/wri=
te/fstat operations to load it up.=0A> > =0A> > If I ctrl-C to kill the scr=
ipts on the client I found=0A> that I can end=0A> > up with nfsd on the ser=
ver stuck at 100% CPU and in=0A> the rc_lo state.=0A> > /etc/rc.d/nfsd rest=
art does nothing.=0A> > =0A> > # nfsstat -s -w 1 -W=0A> > GtAttr Lookup Rdl=
ink Read Write Rename Access Rddir=0A> > 0 0 0 0 0 0 0 0=0A> > 0 0 0 0 0 0 =
0 0=0A> > 0 0 0 0 0 0 0 0=0A> > =0A> > =0A> > Client complaining with:=0A> =
> kernel: nfs server 192.168.1.2:/tank/alantest2: not=0A> responding=0A> > =
=0A> > =0A> > =0A> > Top on the server:=0A> > =0A> > last pid: 4776; load a=
verages: 1.24, 1.15, 1.15 up=0A> 0+23:04:46 18:25:26=0A> > 53 processes: 2 =
running, 35 sleeping, 16 lock=0A> > CPU: 0.0% user, 0.0% nice, 25.0% system=
, 0.0%=0A> interrupt, 75.0% idle=0A> > Mem: 21M Active, 5812K Inact, 2029M =
Wired, 88K Cache,=0A> 8592K Buf, 5857M=0A> > Free=0A> > Swap: 3942M Total, =
3942M Free=0A> > =0A> > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU=0A=
> COMMAND=0A> > 922 root 44 0 5804K 1820K CPU2 2 63:52 100.00% {nfsd:=0A> s=
ervice}=0A> > 922 root 45 0 5804K 1820K *rc_lo 1 30:23 0.00% {nfsd:=0A> ser=
vice}=0A> > 922 root 48 0 5804K 1820K rpcsvc 1 30:20 0.00% {nfsd:=0A> servi=
ce}=0A> > 922 root 46 0 5804K 1820K *rc_lo 1 30:17 0.00% {nfsd:=0A> service=
}=0A> > 922 root 48 0 5804K 1820K rpcsvc 1 30:17 0.00% {nfsd:=0A> service}=
=0A> > 922 root 46 0 5804K 1820K rpcsvc 1 30:13 0.00% {nfsd:=0A> service}=
=0A> > 922 root 45 0 5804K 1820K *rc_lo 2 30:13 0.00% {nfsd:=0A> service}=
=0A> > 922 root 45 0 5804K 1820K *rc_lo 3 30:12 0.00% {nfsd:=0A> service}=
=0A> > 922 root 44 0 5804K 1820K *rc_lo 0 30:10 0.00% {nfsd:=0A> service}=
=0A> > 922 root 46 0 5804K 1820K *rc_lo 2 30:07 0.00% {nfsd:=0A> service}=
=0A> > 922 root 46 0 5804K 1820K *rc_lo 0 30:03 0.00% {nfsd:=0A> service}=
=0A> > 922 root 45 0 5804K 1820K *rc_lo 1 30:03 0.00% {nfsd:=0A> service}=
=0A> > 922 root 46 0 5804K 1820K *rc_lo 2 30:03 0.00% {nfsd:=0A> service}=
=0A> > 922 root 45 0 5804K 1820K *rc_lo 2 30:02 0.00% {nfsd:=0A> service}=
=0A> > 922 root 45 0 5804K 1820K *rc_lo 1 29:59 0.00% {nfsd:=0A> master}=0A=
> > 922 root 45 0 5804K 1820K *rc_lo 2 29:54 0.00% {nfsd:=0A> service}=0A> =
> 922 root 46 0 5804K 1820K *rc_lo 0 29:54 0.00% {nfsd:=0A> service}=0A> > =
922 root 46 0 5804K 1820K *rc_lo 0 29:52 0.00% {nfsd:=0A> service}=0A> > 92=
2 root 47 0 5804K 1820K *rc_lo 2 29:46 0.00% {nfsd:=0A> service}=0A> > 922 =
root 46 0 5804K 1820K *rc_lo 1 29:44 0.00% {nfsd:=0A> service}=0A> > =0A> =
=0A> I have a patch that I think might fix this. (Essentially=0A> the same=
=0A> bug was posted on another list recently.) I'll post it to=0A> Alan=0A>=
separately, but if anyone else wants to try it, it will be=0A> at=0A> =A0=
=A0=A0http://people.freebsd.org/~rmacklem/freebsd8.1-patches/replay.patch=
=0A> in a few minutes.=0A> =0A> rick=0A> =0A> =0A> ________________________=
_______________________=0A> freebsd-stable@freebsd.org=0A> mailing list=0A>=
http://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A>; To unsubscrib=
e, send any mail to "freebsd-stable-unsubscribe@freebsd.org"=0A> =0A=0A=0A=
=0A=0A=0AI applied this patch and have been unable to reproduce the hang si=
nce.=0A=0ARick, Thanks for an awesomely quick response!=0A=0AThis will go i=
nto 8-Stable at some point?=0A=0A--Alan=0A=0A=0A=0A=0A=0A
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?719967.84603.qm>