Date: Fri, 12 Oct 2018 00:37:55 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: Peter Eriksson <peter@ifm.liu.se>, Felix Winterhalter <felix@audiofair.de> Cc: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: NFSv4 Kerberos mount from Linux Message-ID: <YTOPR0101MB1820439E0BFBF57DB2572E92DDE20@YTOPR0101MB1820.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <33A0F0BC-4AD8-4DE3-B484-42B7FB208B6A@ifm.liu.se> References: <30f6446c-6fed-4b1e-9cae-9c417974ec46@audiofair.de> <YTOPR0101MB1820A5756D172342AF441C25DDEA0@YTOPR0101MB1820.CANPRD01.PROD.OUTLOOK.COM> <c1ffda48-3809-bb4c-6d97-451765b0e25e@audiofair.de> <YTOPR0101MB18207F35A3973F26C6A58F6ADDE00@YTOPR0101MB1820.CANPRD01.PROD.OUTLOOK.COM>, <33A0F0BC-4AD8-4DE3-B484-42B7FB208B6A@ifm.liu.se>
next in thread | previous in thread | raw e-mail | index | archive | help
From: Peter Eriksson wrote: >Just a few comments (random brain-dump) in case someone else is having pro= blems >with NFS & Kerberos. > > >We=92ve been using NFSv4 with Kerberos from Linux clients here for many ye= ars (with >Solaris-based NFS servers and MIT Kerberos) and lately using Fre= eBSD as the NFS >server OS (in an Microsoft AD Kerberos environment). I agree with the other post that this is a very useful document and it woul= d be nice to keep it somewhere where it is easier to find than in the mailing li= st archive. I don't know where that could be, so hopefully someone else in FreeBSD-land can suggest a good place? The one area you don't discuss (and maybe isn't really a problem?) is what ticket encryption type(s) you use. Kerberized NFS still uses DES (someday this may change, but I think that re= quires implementation of RPCSEC_GSS V3), so it needs an 8byte session key. (I have never seen a documented way to convert a session key of greater tha= n 8bytes into an 8byte session key for RPCSEC_GSS to use. As such, I have no= idea what happens if you choose a ticket encryption type that results in a grea= ter than 8byte key.) Here's a couple of minor comments: [lots of good stuff snipped] >4. rc.conf (we have a lot of users in our AD so we have to use a large num= ber for >usermax, replace =93liu.se<http://liu.se>=94 with your NFSv4 =93do= main=94 for >nfsuserd_flags) > >gssd_enable=3D"YES" >nfs_server_enable=3D"YES" >nfsv4_server_enable=3D"YES" >nfscbd_enable=3D"YES" Since the nfscbd is only used by the client (and only when delegations or p= NFS is enabled in the server), I believe this is harmless, but unnecessary for th= e server. >mountd_enable=3D"YES" >nfsuserd_enable=3D"YES" >nfsuserd_flags=3D"-manage-gids -domain liu.se<http://liu.se>; -usertimeout = 10 -usermax 100000 16" > Btw, if you have many clients doing NFSv4.0 mounts, tuning of your DRC is a= dvised. (The defaults are for extremely small NFS servers.) NFSv4.1 mounts don't us= e the DRC, so if that is what your clients are doing, this isn't necessary. It's a little off topic, but for NFSv4.0 (and/or NFSv3) mounts and a reason= able sized NFS server, reasonable settings in /etc/sysctl.conf might be: vfs.nfsd.tcpcachetimeo=3D600 vfs.nfsd.tcphighwater=3D100000 vfs.nfsd.v4statelimit=3D10000000 vfs.nfsd.clienthashsize=3D10000 vfs.nfsd.statehashsize=3D10000 - If most/all of your mounts are NFSv4.1, then instead of the above, you mi= ght want: vfs.nfsd.sessionhashsize=3D10000 >5. Make sure you use NTPD so the clock is correct. > > >* All clients (Solaris 10, OmniOS, MacOS 10.12-10.14, FreeBSD 11.0-11.2, C= entOS 7, >Debian 9, Ubuntu 17-18 tested): > >1. Make sure FQDN is in /etc/hosts > >2. Make sure you use NTPD so the clock is correct. > >3. Have a =93host/FQDN@REALM=94 Kerberos host principal in /etc/krb5.keyta= b (nfs or >root is not needed for NFS-mounting to work) I'm guessing that you use the "gssname=3Dhost" mount option for your FreeBS= D clients? (The name after "gssname=3D" is whatever name you have before "/FQDN@REALM" for this principal name.) This is what I referred to as the host-based initiator credential. Thanks for posting this, rick 4. We use a fairly default /etc/krb5.conf, sort of like: [libdefaults] default_realm =3D REALM dns_lookup_realm =3D true ticket_lifetime =3D 24h renew_lifetime =3D 7d forwardable =3D true default_ccache_name =3D KEYRING:persistent:%{uid} KEYRING probably only works on Linux and there are some problems with KEYRI= NG in Debian & Ubuntu since not everything in it supports it due to them us= ing Heimdal instead of MIT (like for smbclient) but it mostly works. Works = fine in CentOS 7 though - in general CentOS 7 feels more =93enterprise=94-r= eady than Debian & Ubuntu. The old classic FILE-ccaches should work fine th= ough. For mounting we use the automounter and a "executable map=94 (perl script) = that looks up records in DNS (Hesiod-style) since the built-in Hesiod suppo= rt in most automounters is a bit.. lacking. Works quite well. You can find = the scripts we use here: http://www.grebo.net/~peter/nfs (The dns-update scripts use data from an SQL database so probably isn=92t d= irectly usable to anybody else. We use the same SQL database to populate a = locally developed BerkeleyDB-based NSS-database on each FreeBSD server in o= rder to speed things up since AD/LDAP-looks with ~90k users and silly amoun= ts of AD groups takes forever, even with cacheing). Some Linux-specific stuff: Packages needed: CentOS: - nfs-utils - libnfsidmap - nfs4-acl-tools - autofs Debian: - keyutils - nfs-kernel-server # rpc.idmapd needs this due to a bug in Debian Ubuntu: - keyutils Other nice-to have packages: - hesiod - autofs-hesiod Some settings to check for: /etc/default/nfs-common: NEED_IDMAPD=3Dyes NEED_GSSD=3Dyes /etc/idmapd.conf (replace =93liu.se<http://liu.se>=94 with your NFSv4 =93= domain=94): Domain=3Dliu.se<http://liu.se>; /etc/request-key.d/id_resolver.conf (should be there already if using a m= odern Linux and you=92ve added the packages above): create id_resolver * * /usr/sbin/nfsidmap %k %d MacOS: Basically require the latest - 10.14 (Mojave) - for things to work smoothly= . In theory 10.12 & 10.13 should work but there is some bug in them that ca= uses the OS to panic when you try to use NFS & Kerberos. 10.11 and earlier = doesn=92t support good enough encryption for us=85 But with 10.14 you just= need to get a Kerberos ticket and then you can mount things just fine. /etc/nfs.conf should contain (replace =93liu.se<http://liu.se>=94 with your= NFSv4 =93domain=94): nfs.client.default_nfs4domain=3Dliu.se<http://liu.se>; (There are a lot of problems you can run into with Microsofts AD implementa= tion of Kerberos too that we=92ve had to be fighting with, but that=92s a w= hole other topic) - Peter On 10 Oct 2018, at 23:47, Rick Macklem <rmacklem@uoguelph.ca<mailto:rmackle= m@uoguelph.ca>> wrote: Felix Winterhalter wrote: On 10/4/18 5:21 PM, Rick Macklem wrote: [stuff snipped] I am now trying to mount this directory as root first without having to deal with user keytabs or tickets. This works fine with -sec=3Dsys and nfsv4.1 and nfsv3 and -sec=3Dkrb5p. This does not however work with nfsv4 and krb5p or any other krb5 flavor. Sorry, I'm not sure what you are saying here. Is it 1 - no version of NFS works for krb5p or 2 - NFSv4.1 works for krb5p, but NFSv4.0 does not or 3 - only nfsv3 works for krb5p [snipped lots of text] #3 is indeed what was happening. I could mount with krb5p for nfsv3 (which I was not aware was even doable) however nfsv4 would stubbornly refuse to do any mounting. Yes, RPCSEC_GSS was done by Sun for NFSv3 and it was a good fit, since NFSv= 3 does not have any server state to maintain. As such, all RPCs are atomic op= erations done by users (which for Kerberized mounts must have a TGT in a credential = cache). NFSv4 wasn't really a good fit for the model, because the NFSv4 server main= tains lock state (NFSv4 Opens are a form of lock used by Windows at file open tim= e). There are "state maintenance" operations that must be done by the user doin= g the mount (usually root), where they typically don't have a TGT in a creden= tial cache. --> The ugly solution for this is typically a host-based client credential = in a keytab on the client. (Usually a principal like "root/client-host.domain@REAL= M" or "host/client-host.domain@REALM" or "nfs/client-host.domain@REALM" in the default keytab on the client.) I have now after a lot of try and error figured out what I need to do in order to make it work. To start with I have kerberos credentials with both host/ and nfs/ on both client and server. Mounting nfsv4 shares with krb5p from a linux server has also worked in this context. Yes, I'm assuming that satisfied the host-based client credential as I desc= ribed above. I leave you to judge whether what I found out is intended behaviour or if something weird is going on. Yes, sounds like intended behaviour, since the client must have a Kerberos credential to use for the "state maintenance" operations that are not done = on behalf of a user. My exports file originally looked something like this: /nfsTests/ /nfsTests/testexport /nfsTests/otherexport -maproot=3Droot -sec=3Dkrb5p clients V4: /nfsTests -sec=3Dkrb5p clients Which allowed me to do nfsv3 krb5p mounts but not nfsv4 krb5p mounts. Changing the exports file to this: /nfsTests/ /nfsTests/testexport /nfsTests/otherexport -maproot=3Droot -sec=3Dkrb5p clients V4: /nfsTests -sec=3Dkrb5p,krb5i clients This suggests that there is a bug in the client, where it uses krb5i instea= d of krb5p at some point in the mounting process. (I have also seen cases where the cl= ient erroneously falls back on using sys at some point in the mounting process.) (You did mention before you were using the Linux client. If you are using a= FreeBSD client, I would be interested in looking at this.) Allows nfsv4 krb5p mounts to work for some reason I do not understand. Not setting the -sec option on the V4 line apparently defaults to -sec=3Dsys and doesn't allow any krb5 mounts. I'm not sure that this is a good default as I wasn't even aware that the -sec option needed to be set on this line. In FreeBSD, defaults are meant to maintain backwards compatibility. This me= ans that AUTH_SYS should work by default. Also, AUTH_SYS is what 99.9999% of FreeBSD NFS mounts still use, from what I've seen.) I've got packet traces of the nfsv3 krb5 and krb5i mounts and I'll make traces of the two nfsv4 mount attempts and send them to you if you're interested. I'm still not sure what exactly is happening here. The successful one for NFSv4 might be interesting. If you look at it in wireshark, I suspect you'd find somewhere during the mount that it did RPCs which were not krb5p and that would show why the addition of krb5i made it work. I did suggest you start with -sec=3Dsys:krb5:krb5i:krb5p and, once that wor= ks, remove the security flavours one at a time until the mount doesn't work. (Then you capture packets for the minimal case that does work and look at what security flavours the client is using for all RPCs done during the mou= nt.) You now know why almost no one uses Kerberized NFSv4 mounts. Unfortunately, the NFSv4 working group has never gotten around to a better solution. Discussion of a host based encryption technique using something like SSL has happened, but no one has gone beyond that. rick _______________________________________________ freebsd-fs@freebsd.org<mailto:freebsd-fs@freebsd.org> mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org<mailto= :freebsd-fs-unsubscribe@freebsd.org>"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YTOPR0101MB1820439E0BFBF57DB2572E92DDE20>