Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 3 Oct 2012 18:35:41 -0400 (EDT)
From:      Rick Macklem <rmacklem@uoguelph.ca>
To:        Ulysse 31 <ulysse31@gmail.com>
Cc:        freebsd-fs@freebsd.org
Subject:   Re: nfsv4 kerberized and gssname=root and allgsname
Message-ID:  <1483416316.1685354.1349303741302.JavaMail.root@erie.cs.uoguelph.ca>
In-Reply-To: <CAFSDvD3v3piN65uCoVxNCRkm8bs9Hz4c6=KZZU6gn1=a-JzhcQ@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_1685353_717231508.1349303741299
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Ulysse 31 wrote:
> 2012/9/29 Rick Macklem <rmacklem@uoguelph.ca>:
> > Ulysse 31 wrote:
> >> Hi all,
> >>
> >> I am actually working on a freebsd 9 backup server.
> >> this server would backup the production server via kerberized nfs4
> >> (since the old backup server, a linux one, was doing so).
> >> we used on the old backup server a root/<fqdn> kerberos identity,
> >> which allows the backup server to access all the data.
> >> I have followed the documentation found at :
> >>
> >> http://code.google.com/p/macnfsv4/wiki/FreeBSD8KerberizedNFSSetup
> >>
> >> done :
> >> - added to kernel :
> >>
> >> options KGSSAPI
> >> device crypto
> >>
> >> - added to rc.conf :
> >>
> >> nfs_client_enable="YES"
> >> rpc_lockd_enable="YES"
> >> rpc_statd_enable="YES"
> >> rpcbind_enable="YES"
> >> devfs_enable="YES"
> >> gssd_enable="YES"
> >>
> >> - have done sysctl vfs.rpcsec.keytab_enctype=1 and added it to
> >> /etc/sysctl.conf
> >>
> >> We used MIT kerberos implementation, since it is the one used on
> >> all
> >> our servers (mostly linux), and we have created and
> >> /etc/krb5.keytab
> >> containing the following keys :
> >> host/<fqdn>
> >> nfs/<fqdn>
> >> root/<fqdn>
> >>
> >> and, of course, i have used the available patch at :
> >> http://people.freebsd.org/~rmacklem/rpcsec_gss-9.patch
> >>
> >> When i try to mount with the (B) method (the one of the google
> >> wiki),
> >> it works as expected, i mean, with a correct user credential, i can
> >> access to the user data.
> >> But, when i try to access via the (C) method (the one that i need
> >> in
> >> order to do a full backup of the production storage server) i get a
> >> systematic kernel panic when launch the mount command.
> >> The mount command looks to something like : mount -t nfs -o
> >> nfsv4,sec=krb5i,gssname=root,allgssname <production server
> >> fqdn>:<export_path> <local_path_where_to_mount>
Just to confirm it, you are saying that exactly the same mount command,
except without the "allgssname" option, doesn't crash?

That is weird, since when I look at the code, there shouldn't be any
difference between the two mounts, up to the point where it crashes.

The crash seems to indicate that nr_auth is bogus, but I can't see
how/why that would happen.

I have attached a patch which changes the way nr_auth is set and "might"
help, although I doubt it. (It is untested, but if you want to try it,
good luck with it.)

I'll email again if I get something more solid figured out, rick

> >> I have activated the kernel debugging stuff to get some infos, here
> >> is
> >> the message :
> >>
> >>
> >> Fatal trap 12: page fault while in kernel mode
> >> cpuid = 0; apic id = 00
> >> fault virtual address = 0x368
> >> fault code = supervisor read data, page not present
> >> instruction pointer = 0x20:0xffffffff80866ab7
> >> stack pointer = 0x28:0xffffff804aa39ce0
> >> frame pointer = 0x28:0xffffff804aa39d30
> >> code segment = base 0x0, limit 0xfffff, type 0x1b
> >> = DPL 0, pres 1, long 1, def32 0, gran 1
> >> processor eflags = interrupt enabled, resume, IOPL = 0
> >> current process = 701 (mount_nfs)
> >> trap number = 12
> >> panic: page fault
> >> cpuid = 0
> >> KDB: stack backtrace:
> >> #0 0xffffffff808ae486 at kdb_backtrace+0x66
> >> #1 0xffffffff8087885e at panic+0x1ce
> >> #2 0xffffffff80b82380 at trap_fatal+0x290
> >> #3 0xffffffff80b826b8 at trap_pfault+0x1e8
> >> #4 0xffffffff80b82cbe at trap+0x3be
> >> #5 0xffffffff80b6c57f at calltrap+0x8
> >> #6 0xffffffff80a78eda at rpc_gss_init+0x72a
> >> #7 0xffffffff80a79cd6 at rpc_gss_refresh_auth+0x46
> >> #8 0xffffffff807a5a53 at newnfs_request+0x163
> >> #9 0xffffffff807bf0f7 at nfsrpc_getattrnovp+0xd7
> >> #10 0xffffffff807d9b29 at mountnfs+0x4e9
> >> #11 0xffffffff807db60a at nfs_mount+0x13ba
> >> #12 0xffffffff809068fb at vfs_donmount+0x100b
> >> #13 0xffffffff80907086 at sys_nmount+0x66
> >> #14 0xffffffff80b81c60 at amd64_syscall+0x540
> >> #15 0xffffffff80b6c867 at Xfast_syscall+0xf7
> >> Uptime: 2m31s
> >> Dumping 97 out of 1002 MB:..17%..33%..50%..66%..83%..99%
> >>
> >> ------------------------------------------------------------------------
> >>
> >> Does anyone as experience something similar ? is their a way to
> >> correct that ?
> >> Thanks for the help.
> >>
> > Well, you're probably the first person to try doing this in years. I
> > did
> > have it working about 4-5years ago. Welcome to the bleeding edge;-)
> >
> > Could you do the following w.r.t. above kernel:
> > # cd /boot/nkernel (or wherever the kernel lives)
> > # nm kernel | grep rpc_gss_init
> > - add the offset 0x72a to the address for rpc_gss_init
> > # addr2line -e kernel.symbols
> > 0xXXX - the hex number above (address of rpc_gss_init+0x72a)
> > - email me what it prints out, so I know where the crash is
> > occurring
> >
> > You could also run the following command on the Linux server to
> > capture
> > packets during the mount attempt, then email me the xxx.pcap file so
> > I
> > can look at it in wireshark, to see what is happening before the
> > crash.
> > (I'm guessing nr_auth is somehow bogus, but that's just a guess.:-)
> > # tcpdump -s 0 -w xxx.pcap host <freebsd-client>
> 
> Hi,
> 
> Sorry for the delay i was on travel and no working network connection.
> Back online for the rest of the week ^^.
> Thanks for your help, here is what it prints out :
> 
> root@bsdenc:/boot/kernel # nm kernel | grep rpc_gss_init
> ffffffff80df07b0 r __set_sysinit_set_sym_svc_rpc_gss_init_sys_init
> ffffffff80a787b0 t rpc_gss_init
> ffffffff80a7a580 t svc_rpc_gss_init
> ffffffff81127530 d svc_rpc_gss_init_sys_init
> ffffffff80a7a3b0 T xdr_rpc_gss_init_res
> root@bsdenc:/boot/kernel # addr2line -e kernel.symbols
> 0xffffffff80a78eda
> /usr/src/sys/rpc/rpcsec_gss/rpcsec_gss.c:772
> 
> 
> for the tcpdump from the linux server, i think you may are doing
> reference to the production nfs server ?
> if yes, unfortunately it is not linux, it is a netapp filer, so no
> "real" root access on it (so no tcpdump available :s ).
> if you were mentioning the old backup server (which is linux but nfs
> client), i cannot do unmount/mount on it since its production
> (mountpoint always busy), but i can made a quick VM/testmachine that
> acts like the linux backup server and do a tcpdump from it.
> Just let me know. Thanks again.
> 
> --
> Ulysse31
> 
> >
> > rick
> >
> >> --
> >> Ulysse31
> >> _______________________________________________
> >> freebsd-fs@freebsd.org mailing list
> >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >> To unsubscribe, send any mail to
> >> "freebsd-fs-unsubscribe@freebsd.org"

------=_Part_1685353_717231508.1349303741299
Content-Type: text/x-patch; name=rpcsec-crash.patch
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename=rpcsec-crash.patch

LS0tIGZzL25mcy9uZnNfY29tbW9ua3JwYy5jLnNhdgkyMDEyLTEwLTAzIDE3OjU4OjE1LjAwMDAw
MDAwMCAtMDQwMAorKysgZnMvbmZzL25mc19jb21tb25rcnBjLmMJMjAxMi0xMC0wMyAxODoyMjow
MS4wMDAwMDAwMDAgLTA0MDAKQEAgLTQ1OCwxMyArNDU4LDE0IEBAIG5ld25mc19yZXF1ZXN0KHN0
cnVjdCBuZnNydl9kZXNjcmlwdCAqbmQKIAl1X2ludCB0cnlsYXRlcl9kZWxheSA9IDE7CiAJc3Ry
dWN0IG5mc19mZWVkYmFja19hcmcgbmY7CiAJc3RydWN0IHRpbWV2YWwgdGltbywgbm93OwotCUFV
VEggKmF1dGg7CisJQVVUSCAqYXV0aCwgKnNhdmF1dGg7CiAJc3RydWN0IHJwY19jYWxsZXh0cmEg
ZXh0OwogCWVudW0gY2xudF9zdGF0IHN0YXQ7CiAJc3RydWN0IG5mc3JlcSAqcmVwID0gTlVMTDsK
IAljaGFyICpzcnZfcHJpbmNpcGFsID0gTlVMTCwgKmNsbnRfcHJpbmNpcGFsID0gTlVMTDsKIAlz
aWdzZXRfdCBvbGRzZXQ7CiAJc3RydWN0IHVjcmVkICphdXRoY3JlZDsKKwlpbnQgZGVzdHJveV9h
dXRoID0gMTsKIAogCWlmICh4aWRwICE9IE5VTEwpCiAJCSp4aWRwID0gMDsKQEAgLTU4NywxMiAr
NTg4LDI5IEBAIG5ld25mc19yZXF1ZXN0KHN0cnVjdCBuZnNydl9kZXNjcmlwdCAqbmQKIAkJICog
bmZzc29ja3JlcSBzdHJ1Y3R1cmUsIHNvIGRvbid0IHJlbGVhc2UgdGhlIHJlZmVyZW5jZSBjb3Vu
dAogCQkgKiBoZWxkIG9uIGl0LiAtLT4gRG9uJ3QgQVVUSF9ERVNUUk9ZKCkgaXQgaW4gdGhpcyBm
dW5jdGlvbi4KIAkJICovCi0JCWlmIChucnAtPm5yX2F1dGggPT0gTlVMTCkKLQkJCW5ycC0+bnJf
YXV0aCA9IG5mc19nZXRhdXRoKG5ycCwgc2VjZmxhdm91ciwKKwkJZGVzdHJveV9hdXRoID0gMDsK
KwkJTkZTTE9DS1NPQ0tSRVEobnJwKTsKKwkJaWYgKG5ycC0+bnJfYXV0aCA9PSBOVUxMKSB7CisJ
CQlORlNVTkxPQ0tTT0NLUkVRKG5ycCk7CisJCQlhdXRoID0gbmZzX2dldGF1dGgobnJwLCBzZWNm
bGF2b3VyLAogCQkJICAgIGNsbnRfcHJpbmNpcGFsLCBzcnZfcHJpbmNpcGFsLCBOVUxMLCBhdXRo
Y3JlZCk7Ci0JCWVsc2UKLQkJCXJwY19nc3NfcmVmcmVzaF9hdXRoX2NhbGwobnJwLT5ucl9hdXRo
KTsKLQkJYXV0aCA9IG5ycC0+bnJfYXV0aDsKKwkJCU5GU0xPQ0tTT0NLUkVRKG5ycCk7CisJCQlp
ZiAobnJwLT5ucl9hdXRoID09IE5VTEwpIHsKKwkJCQlucnAtPm5yX2F1dGggPSBhdXRoOworCQkJ
CU5GU1VOTE9DS1NPQ0tSRVEobnJwKTsKKwkJCX0gZWxzZSB7CisJCQkJc2F2YXV0aCA9IGF1dGg7
CisJCQkJYXV0aCA9IG5ycC0+bnJfYXV0aDsKKwkJCQlORlNVTkxPQ0tTT0NLUkVRKG5ycCk7CisJ
CQkJaWYgKHNhdmF1dGggIT0gTlVMTCkKKwkJCQkJREVTVFJPWV9BVVRIKHNhdmF1dGgpOworCQkJ
fQorCQl9IGVsc2UgeworCQkJYXV0aCA9IG5ycC0+bnJfYXV0aDsKKwkJCU5GU1VOTE9DS1NPQ0tS
RVEobnJwKTsKKwkJfQorCQlpZiAoYXV0aCAhPSBOVUxMKQorCQkJcnBjX2dzc19yZWZyZXNoX2F1
dGhfY2FsbChhdXRoKTsKIAl9IGVsc2UKIAkJYXV0aCA9IG5mc19nZXRhdXRoKG5ycCwgc2VjZmxh
dm91ciwgTlVMTCwKIAkJICAgIHNydl9wcmluY2lwYWwsIE5VTEwsIGF1dGhjcmVkKTsKQEAgLTc0
MSw3ICs3NTksNyBAQCB0cnlhZ2FpbjoKIAl9CiAJaWYgKGVycm9yKSB7CiAJCW1fZnJlZW0obmQt
Pm5kX21yZXEpOwotCQlpZiAodXNlZ3NzbmFtZSA9PSAwKQorCQlpZiAoZGVzdHJveV9hdXRoICE9
IDApCiAJCQlBVVRIX0RFU1RST1koYXV0aCk7CiAJCWlmIChyZXAgIT0gTlVMTCkKIAkJCUZSRUUo
KGNhZGRyX3QpcmVwLCBNX05GU0RSRVEpOwpAQCAtODk5LDcgKzkxNyw3IEBAIHRyeWFnYWluOgog
I2VuZGlmCiAKIAltX2ZyZWVtKG5kLT5uZF9tcmVxKTsKLQlpZiAodXNlZ3NzbmFtZSA9PSAwKQor
CWlmIChkZXN0cm95X2F1dGggIT0gMCkKIAkJQVVUSF9ERVNUUk9ZKGF1dGgpOwogCWlmIChyZXAg
IT0gTlVMTCkKIAkJRlJFRSgoY2FkZHJfdClyZXAsIE1fTkZTRFJFUSk7CkBAIC05MDksNyArOTI3
LDcgQEAgdHJ5YWdhaW46CiBuZnNtb3V0OgogCW1idWZfZnJlZW0obmQtPm5kX21yZXApOwogCW1i
dWZfZnJlZW0obmQtPm5kX21yZXEpOwotCWlmICh1c2Vnc3NuYW1lID09IDApCisJaWYgKGRlc3Ry
b3lfYXV0aCAhPSAwKQogCQlBVVRIX0RFU1RST1koYXV0aCk7CiAJaWYgKHJlcCAhPSBOVUxMKQog
CQlGUkVFKChjYWRkcl90KXJlcCwgTV9ORlNEUkVRKTsK
------=_Part_1685353_717231508.1349303741299--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1483416316.1685354.1349303741302.JavaMail.root>