Date: Wed, 17 Mar 2021 21:48:41 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: Alan Somers <asomers@freebsd.org> Cc: freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: Understanding the vfs.nfsd.request_space* sysctls Message-ID: <YQXPR0101MB0968373DFF89B4FDB6000E4DDD6A9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <CAOtMX2ih9cLq3-Mw9xw3T9ovBiZ%2BY3PG82pN0s2eVusRooRF_g@mail.gmail.com> References: <CAOtMX2geF7Tv7VYc9WtK23a%2BnDzJzD2ehVKp94=xANjTA9S9Tg@mail.gmail.com> <YQXPR0101MB0968CD521F2A19F44488367ADD6B9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM>, <CAOtMX2ih9cLq3-Mw9xw3T9ovBiZ%2BY3PG82pN0s2eVusRooRF_g@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Alan Somers wrote:=0A=
>Rick Macklem wrote:=0A=
[stuff snipped]=0A=
>>Anytime used > high, it throttles, which means it leaves=0A=
>>the RPC messages on the socket receive queue.=0A=
>>This indicates the server is not keeping up with requests.=0A=
>>(ie Overloaded or ???)=0A=
>=0A=
>That all makes sense. So having a high request_space_used basically just =
means that >my storage is too slow.=0A=
One other thing you might want to look at is rcv interrupt moderation=0A=
in the net device driver.=0A=
--> This *might* result in received RPC requests being "batched", resulting=
in=0A=
a burst of RPCs that overloads the server. (Except for write requests=
, each=0A=
RPC is a small message typically in a single small packet.)=0A=
--> Disabling rcv interrupt moderation will result in more interrupts=
, but=0A=
then each RPC request will be handled right away. This might r=
esult=0A=
in more evenly spaced RPCs (in a temporal sense).=0A=
Also, delaying processing of RPC requests increases RPC RTT an=
d that=0A=
always has a negative impact on NFS performance.=0A=
=0A=
rick=0A=
=0A=
> And how can I either=0A=
>increase the available space or decrease the stuff that's using it?=0A=
Increasing vfs.nfsd.request_space_high avoids the throttling,=0A=
but it is hard to say that is a good idea, since there won't be=0A=
backpressure applied to the clients via TCP windows, etc.=0A=
=0A=
Fixing the server so that it isn't overloaded would be better,=0A=
I think?=0A=
--> I'm the last guy to take ZFS advice from, but I think there=0A=
is a tunable w.r.t. how much arc is used for metadata.=0A=
Getattrs will need cached (metadata) to reply quickly.=0A=
Lookups also depend on cached attributes for good=0A=
perf. as well.=0A=
=0A=
I've added an L2ARC since I sent that original email. We'll see how well i=
t works. I expect it to take 48 hours before results are apparent.=0A=
=0A=
--> Make sure you have lots of nfsd threads. They are=0A=
just kernel threads, so they don't use a lot of resources=0A=
and having too many is much better than not enough.=0A=
-->I can't remember what the upper limit is these days,=0A=
but I'd set it to that for a busy nfs server.=0A=
For NFSv4.1, each client can send a maximum of=0A=
session_slot_table_size concurrent RPCs. FreeBSD=0A=
uses a single 64slot table for each mount. I think=0A=
Linux uses a 32slot table, but supports trunking.=0A=
I don't know if the Linux client uses a separate=0A=
session table for each trunk or not?=0A=
--> Something like #clients * 32 (or 64) nfsd=0A=
threads running on the server.=0A=
=0A=
Experimentally, I determined that 768 threads is sufficient. But your form=
ula would suggest > 2500, which is a lot of threads! I'll keep it in mind =
if I ever reach 768, though.=0A=
=0A=
=0A=
rick=0A=
=0A=
-Alan=0A=
_______________________________________________=0A=
freebsd-fs@freebsd.org<mailto:freebsd-fs@freebsd.org> mailing list=0A=
https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A=
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org<mailto=
:freebsd-fs-unsubscribe@freebsd.org>"=0A=
=0A=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YQXPR0101MB0968373DFF89B4FDB6000E4DDD6A9>