Date: Wed, 05 May 2010 09:20:52 +0800 (CST) From: "Cheng-Lin Yang" <yuwen@exodus.cs.ccu.edu.tw> To: "rmacklem" <rmacklem@uoguelph.ca> Cc: freebsd-fs <freebsd-fs@freebsd.org>, lab <lab@cs.ccu.edu.tw> Subject: Re: Struggling on NFS problem Message-ID: <1273022452.30713.yuwen@exodus.cs.ccu.edu.tw> In-Reply-To: <Pine.GSO.4.63.1005041019180.14588@muncher.cs.uoguelph.ca> References: <1272960060.34062.yuwen@exodus.cs.ccu.edu.tw> <Pine.GSO.4.63.1005041019180.14588@muncher.cs.uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi=20Rick, Unfortunately,=20the=20machine=20which=20shows=20negative=20number,=20it's=20= amd64=20not=20i386: FreeBSD=20csie0.cs.ccu.edu.tw=208.0-RELEASE-p2=20FreeBSD=208.0-RELEASE-p2=20= #0:=20Tue=20Jan=20=205=2021:11:58=20UTC=202010=20=20=20=20=20root@amd64-buil= der.daemonology.net:/usr/obj/usr/src/sys/GENERIC=20=20amd64 I'll=20try=20to=20apply=20all=20the=20patches=20on=20my=20NFS=20server=20to=20= see=20if=20it=20can=20fix=20the=20problem.=20Also,=20I've=20replyed=20a=20ma= il=20which=20contains=20more=20information=20about=20NFS=20server.=20Could=20= you=20also=20take=20a=20look=20on=20it=20to=20see=20if=20there's=20anything=20= wrong? If=20the=20problem=20shows=20up=20again=20(I'm=20sure=20it=20will),=20I'll=20= try=20to=20capture=20the=20NFS=20traffic=20between=20server=20and=20clients.= =20Thank=20you!=20:) Regards, Cheng-Lin=20Yang -----Original=20message----- From:Rick=20Macklem=20<rmacklem@uoguelph.ca> To:Cheng-Lin=20Yang=20<yuwen@exodus.cs.ccu.edu.tw> Cc:freebsd-fs=20<freebsd-fs@freebsd.org>,lab=20<lab@cs.ccu.edu.tw> Date:Tue,=204=20May=202010=2010:34:02=20-0400=20(EDT) Subject:Re:=20Struggling=20on=20NFS=20problem On=20Tue,=204=20May=202010,=20Cheng-Lin=20Yang=20wrote: >=20Dear=20all, >=20Currently,=20we=20have=20a=20NFS=20server=20which=20runs=20FreeBSD8=20wi= th=20ZFS=20and=20few=20workstations=20as=20NFS=20client=20(2=20*=20FreeBSD8=20= amd64=20+=201=20*=20FreeBSD7.2=20i386=20+=202=20*=20Fedora=20+=20Debian).=20= We=20spotted=20that=20NFS=20performs=20weirdly=20on=20FreeBSD=20clients,=20w= hich=20will=20significantly=20slow=20down=20the=20system=20response.=20The=20= only=20solution=20to=20it=20is=20to=20reboot=20the=20clients=20(Linux=20clie= nt=20runs=20smoothly).=20So=20we=20try=20to=20use=20"nfsstat=20-c"=20on=20Fr= eeBSD=20client=20to=20dig=20into=20the=20problem=20and=20found=20strange=20r= esult=20(http://pastebin.com/K71qpEDG)=20: >=20csie0[~]#=20nfsstat=20-c [stuff=20snipped] > >=20As=20you=20can=20see,=20the=20value=20of=20"BioW=20Hits"=20is=20a=20nega= tive=20number.=20Shouldn't=20it=20be=20equal=20or=20larger=20than=20zero?=20= We=20have=20totally=20no=20idea=20on=20this=20issue.=20Please=20kindly=20hel= p=20us=20on=20investigating=20the=20problem.=20Any=20suggestion=20is=20extre= mely=20welcomed.=20Thank=20you. > I=20suspect=20that=20the=20negative=20value=20is=20just=20a=20wrap=20around=20= (assuming=20you're on=20a=2032bit=20arch)=20and=20hust=20means=20lottsa=20hits.=20If=20that=20i= s=20the=20case,=20it suggests=20a=20fairly=20heavy=20write=20load,=20which=20can=20be=20an=20issu= e=20for=20servers using=20ZFS=20(as=20others=20have=20already=20posted=20about). There=20are=20a=20#=20of=20patches=20for=20FreeBSD8.0=20related=20to=20NFS=20= (one=20specifically w.r.t.=20the=20server=20using=20ZFS)=20at: =20=09http://people.freebsd.org/~rmacklem If=20you=20are=20using=20FreeBSD8.0=20for=20the=20server,=20it=20would=20be=20= worth=20trying these=20patches=20(they=20are=20all=20independent,=20in=20that=20any=20of=20= them=20can=20be applied,=20in=20any=20order).=20(If=20you=20are=20using=20a=20recent=20stabl= e/8,=20then you=20should=20already=20have=20the=20patches.) In=20particular,=20one=20of=20them=20fixes=20a=20case=20where=20FreeBSD=20cl= ients=20will get=20stuck=20looping=20trying=20to=20access=20a=20file=20after=20it=20has=20= been=20deleted on=20the=20server,=20because=20the=20server=20reported=20EIO=20instead=20of=20= ESTALE=20for this=20case. If=20the=20patches=20don't=20help,=20please=20try=20to=20collect=20more=20in= formation from=20both=20the=20slow=20clients=20and=20server.=20"ps=20axl"=20on=20them=20= all=20can=20be useful.=20Also,=20you=20can=20use=20"tcpdump=20-s=200=20-w=20<file>=20host=20= <clienthost>" to=20capture=20traffic=20between=20the=20slow=20client=20and=20server=20whic= h=20can=20be looked=20at=20via=20wireshark.=20(tcpdump=20doesn't=20decode=20NFS=20traffic= =20well, but=20a=20binary=20capture=20from=20tcpdump=20goes=20into=20wireshark=20ok=20= and=20it=20does understand=20NFS=20traffic)=20If=20you=20get=20to=20this=20point,=20you=20ca= n=20email=20me the=20"<file>"=20as=20an=20attachment=20and=20I=20can=20take=20a=20look=20at= =20it.=20If=20you look=20at=20it,=20one=20scenario=20that=20is=20of=20interest=20is=20where=20= the=20client just=20keeps=20retrying=20the=20same=20NFS=20RPC. Good=20luck=20with=20it=20and=20let=20us=20know=20how=20it=20goes,=20rick ---------------------------------- Cheng-Lin=20Yang Sun=20Certified=20Java=20Programmer High=20Speed=20Network=20Group=20Lab=20(HSNG) Institute=20of=20Computer=20Science=20&=20Info.=20Engineering, National=20Chung=20Cheng=20Univerisity,=20Taiwan E-mail:=20yuwen@exodus.cs.ccu.edu.tw
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1273022452.30713.yuwen>