Date: Tue, 28 Nov 2017 23:18:57 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: Emmanuel Vadot <manu@bidouilliste.com> Cc: Konstantin Belousov <kostikbel@gmail.com>, FreeBSD Current <freebsd-current@freebsd.org>, "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org>, Rick Macklem <rmacklem@freebsd.org> Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? Message-ID: <YTOPR0101MB21721E82EF605249D38064A1DD3A0@YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <YTOPR0101MB217218369849273146801292DD3A0@YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM> References: <20171128114136.10e44d5bcfd563e9b4ab5453@bidouilliste.com> <20171128110428.GN2272@kib.kiev.ua> <20171128142610.6ab535b0b8c6b5d372cd3f27@bidouilliste.com> <20171128134009.GQ2272@kib.kiev.ua> <YTOPR0101MB2172BE15A55BCA881F805AE6DD3A0@YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM>, <20171128164132.290bf07218b16164db613242@bidouilliste.com>, <YTOPR0101MB217255A72DD5E8555DF001E9DD3A0@YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM>, <YTOPR0101MB217218369849273146801292DD3A0@YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM>
next in thread | previous in thread | raw e-mail | index | archive | help
--_002_YTOPR0101MB21721E82EF605249D38064A1DD3A0YTOPR0101MB2172_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Did my usual and forgot to attach it. Here's the patch, rick ________________________________________ From: Rick Macklem <rmacklem@uoguelph.ca> Sent: Tuesday, November 28, 2017 6:17:13 PM To: Emmanuel Vadot Cc: Konstantin Belousov; FreeBSD Current; freebsd-fs@freebsd.org; Rick Mack= lem Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? I think the attached patch should fix your performance problem. It simply checks for nfsrv_delegatecnt !=3D 0 before doing all the locking stuff in nfsrv_checkgetattr(). If this fixes your performance problem (with delegations disabled), I'll probably add a separate counter for write delegations and use atomics to increment/decrement it so that it is SMP safe without acquiring any lock. If you can test this, please let me know how it goes? rick ________________________________________ From: Rick Macklem <rmacklem@uoguelph.ca> Sent: Tuesday, November 28, 2017 2:09:51 PM To: Emmanuel Vadot Cc: Konstantin Belousov; FreeBSD Current; freebsd-fs@freebsd.org; Rick Mack= lem Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? Emmanuel Vadot wrote: >I wrote: >> Since it defaults to "disabled", I don't see why a tunable would be nece= ssary? >> (Just do nothing and delegations don't happen. If you want the server >> to issue delegations, then use the sysctl to turn them on. If you want = to turn >> them off again at the server, reboot the server without setting the sys= ctl.) > >If you need to reboot to make things working again without delegation >this shouldn't be a sysctl. Turning them off without rebooting doesn't break anything. I just wrote it that way since you seemed to want to use a tunable, which implied rebooting was your preference. > > > > The reason behind it is recent problem at work on some on our file= r > > > > related to NFS. > > > > We use NFSv4 without delegation as we never been able to have good > > > > performance with FreeBSD server and Linux client (we need to do tes= t > > > > again but that's for later). > Delegations are almost never useful, especially with Linux clients. Emmanuel Vadot wrote: >Can you elaborate ? Reading what delegation are I see that this is >exactly what I'm looking for to have better performance with NFS for my >workload (I only have one client per mount point). Delegations allow the client to do Opens locally without contacting the server. Unless, the delegation is a write delegation, this only applied to read-only delegations. Also, since most implementors couldn`t agree on how to check permissions via the delegation, most client implementations still do an Access check at open, which is almost as much overhead as the Open RPC. (For example, Solaris servers always specified an empty ACE in th= e delegation, forcing the client to do an Access. I have no idea what the current Linux server=E9client does. If you capture packets when a Linux client is mounted with delegations enabled, you could look for RPCs like Ac= cess when are Opened multiple times. If you see them, then delegations aren`t saving = RPCs. Also, they are `per file`, so are only useful if the client is Opening the same file multiple times. Further, if another client Opens the same file and the first client got a W= rite delegation, then the write delegation must be recalled, which is a lot of overhead and one of the few cases where the FreeBSD server must exclusively lock the state lists, forcing all other RPCs related to Open, Close to wait= . They sounded good in theory and might have worked well if the implementors had agreed to do them, but that didn=E8t happen. (Companies pay for servers= , but the clients don=E8t get funded so delegation support in the clients are lacking= . I tried to make them useful in the FreeBSD client, but I=E8m not sure I succeeded.) > [stuff snipped] If I understood your original post, you have a performance problem caused by lock contention, where the server grabs the state lock to check for dele= gations for every Getattr from a client. As below, I think the fix is to add code to check for no delegations issued= that does not require acquisition of the state lock. Btw, large numbers of Getattrs will not perform well with delegations. (Again, the client should be able to do Getattr operations locally in the client when it has a delegation for the file, but if the client is not doi= ng that...) I wrote: > > Having a per-mount version of this would be overkill, I think. It would h= ave to > disable callbacks on the mount point, since there is no way for a client = to say > "don't give me delegations" except disabling callbacks, which the server > requires for delegation issue. > [stuff snipped] > The case where there has never been delegations issued will result in an > empty delegation queue. Maybe this case can be handled without acquiring > the "global NFSv4 state lock", which is what sounds like is causing the > performance issue. Maybe a global counter of how many delegations are > issued that is handled by atomic ops. > --> If it is 0, nfsrv_checkgetattr() can just return without acquiring th= e lock. > > I'll take a look at this, rick > rick --_002_YTOPR0101MB21721E82EF605249D38064A1DD3A0YTOPR0101MB2172_ Content-Type: application/octet-stream; name="checkgetattr.patch" Content-Description: checkgetattr.patch Content-Disposition: attachment; filename="checkgetattr.patch"; size=393; creation-date="Tue, 28 Nov 2017 23:18:53 GMT"; modification-date="Tue, 28 Nov 2017 23:18:53 GMT" Content-Transfer-Encoding: base64 LS0tIGZzL25mc3NlcnZlci9uZnNfbmZzZHN0YXRlLmMuc2F2CTIwMTctMTEtMjggMDk6NTY6Mjcu NzA4NTk5MDAwIC0wNTAwCisrKyBmcy9uZnNzZXJ2ZXIvbmZzX25mc2RzdGF0ZS5jCTIwMTctMTEt MjggMDk6NTc6MzEuMTEzMTYzMDAwIC0wNTAwCkBAIC01Mjk4LDcgKzUyOTgsNyBAQCBuZnNydl9j aGVja2dldGF0dHIoc3RydWN0IG5mc3J2X2Rlc2NyaXB0CiAJdV9xdWFkX3QgZGVsZWdmaWxlcmV2 OwogCiAJTkZTQ0JHRVRBVFRSX0FUVFJCSVQoYXR0cmJpdHAsICZjYmJpdHMpOwotCWlmICghTkZT Tk9OWkVST19BVFRSQklUKCZjYmJpdHMpKQorCWlmICghTkZTTk9OWkVST19BVFRSQklUKCZjYmJp dHMpIHx8IG5mc3J2X2RlbGVnYXRlY250ID09IDApCiAJCWdvdG8gb3V0OwogCiAJLyoK --_002_YTOPR0101MB21721E82EF605249D38064A1DD3A0YTOPR0101MB2172_--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YTOPR0101MB21721E82EF605249D38064A1DD3A0>