Date: Tue, 16 Mar 2021 19:32:16 -0600 From: Alan Somers <asomers@freebsd.org> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: Understanding the vfs.nfsd.request_space* sysctls Message-ID: <CAOtMX2ih9cLq3-Mw9xw3T9ovBiZ%2BY3PG82pN0s2eVusRooRF_g@mail.gmail.com> In-Reply-To: <YQXPR0101MB0968CD521F2A19F44488367ADD6B9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> References: <CAOtMX2geF7Tv7VYc9WtK23a%2BnDzJzD2ehVKp94=xANjTA9S9Tg@mail.gmail.com> <YQXPR0101MB0968CD521F2A19F44488367ADD6B9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Mar 16, 2021 at 6:06 PM Rick Macklem <rmacklem@uoguelph.ca> wrote: > Alan Somers wrote: > >Could somebody help me understand the meaning of the various > >vfs.nfsd.request_space* sysctls, and what implication they have for tuning > >NFS? > > > >I have several NFS servers. Most are performing well, but one with a > >metadata-heavy workload (NFS 4.1, no delegations, lots of Sequence, > GetAttr > >and Lookup ops) is not. Total throughput is a not-unreasonable several > >hundred MB/s, but some clients are limited to very slow speeds, sometimes > >writing at < 10 MB/s. > > > >The request_space sysctls show some pretty stark divergence between the > >well-performing and poor-performing servers: > > > >sysctl well-performing poor-performing > >request_space_used < 6 M varies, but > >currently 40 M > >request_space_used_highest < 39 M 24 G > >request_space_throttle_count 0 35 > > > >So what does request_space_used measure, anyway? > It's the number of bytes of request message(s) received, > but not yet being processed by nfsd threads. > (They're actually in sys/rpc/svc.c, although named for the > nfs server.) > > Anytime used > high, it throttles, which means it leaves > the RPC messages on the socket receive queue. > This indicates the server is not keeping up with requests. > (ie Overloaded or ???) > That all makes sense. So having a high request_space_used basically just means that my storage is too slow. > > > And how can I either > >increase the available space or decrease the stuff that's using it? > Increasing vfs.nfsd.request_space_high avoids the throttling, > but it is hard to say that is a good idea, since there won't be > backpressure applied to the clients via TCP windows, etc. > > Fixing the server so that it isn't overloaded would be better, > I think? > --> I'm the last guy to take ZFS advice from, but I think there > is a tunable w.r.t. how much arc is used for metadata. > Getattrs will need cached (metadata) to reply quickly. > Lookups also depend on cached attributes for good > perf. as well. > I've added an L2ARC since I sent that original email. We'll see how well it works. I expect it to take 48 hours before results are apparent. > --> Make sure you have lots of nfsd threads. They are > just kernel threads, so they don't use a lot of resources > and having too many is much better than not enough. > -->I can't remember what the upper limit is these days, > but I'd set it to that for a busy nfs server. > For NFSv4.1, each client can send a maximum of > session_slot_table_size concurrent RPCs. FreeBSD > uses a single 64slot table for each mount. I think > Linux uses a 32slot table, but supports trunking. > I don't know if the Linux client uses a separate > session table for each trunk or not? > --> Something like #clients * 32 (or 64) nfsd > threads running on the server. > Experimentally, I determined that 768 threads is sufficient. But your formula would suggest > 2500, which is a lot of threads! I'll keep it in mind if I ever reach 768, though. > > rick > > -Alan > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2ih9cLq3-Mw9xw3T9ovBiZ%2BY3PG82pN0s2eVusRooRF_g>