From owner-freebsd-fs@FreeBSD.ORG Wed Oct 3 22:35:48 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C2AA1106566B for ; Wed, 3 Oct 2012 22:35:48 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 4FF948FC0C for ; Wed, 3 Oct 2012 22:35:48 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqAEALy8bFCDaFvO/2dsb2JhbAA8CRaFebl3giABAQEDAQEBASAEJyALGxgRGQIEJQEJJgYIBwQBHASFcIFuBgulQJJZiyUPBAUGhQiBEgOOboNKgQSCLYEVjxaDCYE/CDQ X-IronPort-AV: E=Sophos;i="4.80,530,1344225600"; d="scan'208";a="181851197" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 03 Oct 2012 18:35:41 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 4FE94B4032; Wed, 3 Oct 2012 18:35:41 -0400 (EDT) Date: Wed, 3 Oct 2012 18:35:41 -0400 (EDT) From: Rick Macklem To: Ulysse 31 Message-ID: <1483416316.1685354.1349303741302.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_1685353_717231508.1349303741299" X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org Subject: Re: nfsv4 kerberized and gssname=root and allgsname X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Oct 2012 22:35:49 -0000 ------=_Part_1685353_717231508.1349303741299 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Ulysse 31 wrote: > 2012/9/29 Rick Macklem : > > Ulysse 31 wrote: > >> Hi all, > >> > >> I am actually working on a freebsd 9 backup server. > >> this server would backup the production server via kerberized nfs4 > >> (since the old backup server, a linux one, was doing so). > >> we used on the old backup server a root/ kerberos identity, > >> which allows the backup server to access all the data. > >> I have followed the documentation found at : > >> > >> http://code.google.com/p/macnfsv4/wiki/FreeBSD8KerberizedNFSSetup > >> > >> done : > >> - added to kernel : > >> > >> options KGSSAPI > >> device crypto > >> > >> - added to rc.conf : > >> > >> nfs_client_enable="YES" > >> rpc_lockd_enable="YES" > >> rpc_statd_enable="YES" > >> rpcbind_enable="YES" > >> devfs_enable="YES" > >> gssd_enable="YES" > >> > >> - have done sysctl vfs.rpcsec.keytab_enctype=1 and added it to > >> /etc/sysctl.conf > >> > >> We used MIT kerberos implementation, since it is the one used on > >> all > >> our servers (mostly linux), and we have created and > >> /etc/krb5.keytab > >> containing the following keys : > >> host/ > >> nfs/ > >> root/ > >> > >> and, of course, i have used the available patch at : > >> http://people.freebsd.org/~rmacklem/rpcsec_gss-9.patch > >> > >> When i try to mount with the (B) method (the one of the google > >> wiki), > >> it works as expected, i mean, with a correct user credential, i can > >> access to the user data. > >> But, when i try to access via the (C) method (the one that i need > >> in > >> order to do a full backup of the production storage server) i get a > >> systematic kernel panic when launch the mount command. > >> The mount command looks to something like : mount -t nfs -o > >> nfsv4,sec=krb5i,gssname=root,allgssname >> fqdn>: Just to confirm it, you are saying that exactly the same mount command, except without the "allgssname" option, doesn't crash? That is weird, since when I look at the code, there shouldn't be any difference between the two mounts, up to the point where it crashes. The crash seems to indicate that nr_auth is bogus, but I can't see how/why that would happen. I have attached a patch which changes the way nr_auth is set and "might" help, although I doubt it. (It is untested, but if you want to try it, good luck with it.) I'll email again if I get something more solid figured out, rick > >> I have activated the kernel debugging stuff to get some infos, here > >> is > >> the message : > >> > >> > >> Fatal trap 12: page fault while in kernel mode > >> cpuid = 0; apic id = 00 > >> fault virtual address = 0x368 > >> fault code = supervisor read data, page not present > >> instruction pointer = 0x20:0xffffffff80866ab7 > >> stack pointer = 0x28:0xffffff804aa39ce0 > >> frame pointer = 0x28:0xffffff804aa39d30 > >> code segment = base 0x0, limit 0xfffff, type 0x1b > >> = DPL 0, pres 1, long 1, def32 0, gran 1 > >> processor eflags = interrupt enabled, resume, IOPL = 0 > >> current process = 701 (mount_nfs) > >> trap number = 12 > >> panic: page fault > >> cpuid = 0 > >> KDB: stack backtrace: > >> #0 0xffffffff808ae486 at kdb_backtrace+0x66 > >> #1 0xffffffff8087885e at panic+0x1ce > >> #2 0xffffffff80b82380 at trap_fatal+0x290 > >> #3 0xffffffff80b826b8 at trap_pfault+0x1e8 > >> #4 0xffffffff80b82cbe at trap+0x3be > >> #5 0xffffffff80b6c57f at calltrap+0x8 > >> #6 0xffffffff80a78eda at rpc_gss_init+0x72a > >> #7 0xffffffff80a79cd6 at rpc_gss_refresh_auth+0x46 > >> #8 0xffffffff807a5a53 at newnfs_request+0x163 > >> #9 0xffffffff807bf0f7 at nfsrpc_getattrnovp+0xd7 > >> #10 0xffffffff807d9b29 at mountnfs+0x4e9 > >> #11 0xffffffff807db60a at nfs_mount+0x13ba > >> #12 0xffffffff809068fb at vfs_donmount+0x100b > >> #13 0xffffffff80907086 at sys_nmount+0x66 > >> #14 0xffffffff80b81c60 at amd64_syscall+0x540 > >> #15 0xffffffff80b6c867 at Xfast_syscall+0xf7 > >> Uptime: 2m31s > >> Dumping 97 out of 1002 MB:..17%..33%..50%..66%..83%..99% > >> > >> ------------------------------------------------------------------------ > >> > >> Does anyone as experience something similar ? is their a way to > >> correct that ? > >> Thanks for the help. > >> > > Well, you're probably the first person to try doing this in years. I > > did > > have it working about 4-5years ago. Welcome to the bleeding edge;-) > > > > Could you do the following w.r.t. above kernel: > > # cd /boot/nkernel (or wherever the kernel lives) > > # nm kernel | grep rpc_gss_init > > - add the offset 0x72a to the address for rpc_gss_init > > # addr2line -e kernel.symbols > > 0xXXX - the hex number above (address of rpc_gss_init+0x72a) > > - email me what it prints out, so I know where the crash is > > occurring > > > > You could also run the following command on the Linux server to > > capture > > packets during the mount attempt, then email me the xxx.pcap file so > > I > > can look at it in wireshark, to see what is happening before the > > crash. > > (I'm guessing nr_auth is somehow bogus, but that's just a guess.:-) > > # tcpdump -s 0 -w xxx.pcap host > > Hi, > > Sorry for the delay i was on travel and no working network connection. > Back online for the rest of the week ^^. > Thanks for your help, here is what it prints out : > > root@bsdenc:/boot/kernel # nm kernel | grep rpc_gss_init > ffffffff80df07b0 r __set_sysinit_set_sym_svc_rpc_gss_init_sys_init > ffffffff80a787b0 t rpc_gss_init > ffffffff80a7a580 t svc_rpc_gss_init > ffffffff81127530 d svc_rpc_gss_init_sys_init > ffffffff80a7a3b0 T xdr_rpc_gss_init_res > root@bsdenc:/boot/kernel # addr2line -e kernel.symbols > 0xffffffff80a78eda > /usr/src/sys/rpc/rpcsec_gss/rpcsec_gss.c:772 > > > for the tcpdump from the linux server, i think you may are doing > reference to the production nfs server ? > if yes, unfortunately it is not linux, it is a netapp filer, so no > "real" root access on it (so no tcpdump available :s ). > if you were mentioning the old backup server (which is linux but nfs > client), i cannot do unmount/mount on it since its production > (mountpoint always busy), but i can made a quick VM/testmachine that > acts like the linux backup server and do a tcpdump from it. > Just let me know. Thanks again. > > -- > Ulysse31 > > > > > rick > > > >> -- > >> Ulysse31 > >> _______________________________________________ > >> freebsd-fs@freebsd.org mailing list > >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs > >> To unsubscribe, send any mail to > >> "freebsd-fs-unsubscribe@freebsd.org" ------=_Part_1685353_717231508.1349303741299 Content-Type: text/x-patch; name=rpcsec-crash.patch Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename=rpcsec-crash.patch LS0tIGZzL25mcy9uZnNfY29tbW9ua3JwYy5jLnNhdgkyMDEyLTEwLTAzIDE3OjU4OjE1LjAwMDAw MDAwMCAtMDQwMAorKysgZnMvbmZzL25mc19jb21tb25rcnBjLmMJMjAxMi0xMC0wMyAxODoyMjow MS4wMDAwMDAwMDAgLTA0MDAKQEAgLTQ1OCwxMyArNDU4LDE0IEBAIG5ld25mc19yZXF1ZXN0KHN0 cnVjdCBuZnNydl9kZXNjcmlwdCAqbmQKIAl1X2ludCB0cnlsYXRlcl9kZWxheSA9IDE7CiAJc3Ry dWN0IG5mc19mZWVkYmFja19hcmcgbmY7CiAJc3RydWN0IHRpbWV2YWwgdGltbywgbm93OwotCUFV VEggKmF1dGg7CisJQVVUSCAqYXV0aCwgKnNhdmF1dGg7CiAJc3RydWN0IHJwY19jYWxsZXh0cmEg ZXh0OwogCWVudW0gY2xudF9zdGF0IHN0YXQ7CiAJc3RydWN0IG5mc3JlcSAqcmVwID0gTlVMTDsK IAljaGFyICpzcnZfcHJpbmNpcGFsID0gTlVMTCwgKmNsbnRfcHJpbmNpcGFsID0gTlVMTDsKIAlz aWdzZXRfdCBvbGRzZXQ7CiAJc3RydWN0IHVjcmVkICphdXRoY3JlZDsKKwlpbnQgZGVzdHJveV9h dXRoID0gMTsKIAogCWlmICh4aWRwICE9IE5VTEwpCiAJCSp4aWRwID0gMDsKQEAgLTU4NywxMiAr NTg4LDI5IEBAIG5ld25mc19yZXF1ZXN0KHN0cnVjdCBuZnNydl9kZXNjcmlwdCAqbmQKIAkJICog bmZzc29ja3JlcSBzdHJ1Y3R1cmUsIHNvIGRvbid0IHJlbGVhc2UgdGhlIHJlZmVyZW5jZSBjb3Vu dAogCQkgKiBoZWxkIG9uIGl0LiAtLT4gRG9uJ3QgQVVUSF9ERVNUUk9ZKCkgaXQgaW4gdGhpcyBm dW5jdGlvbi4KIAkJICovCi0JCWlmIChucnAtPm5yX2F1dGggPT0gTlVMTCkKLQkJCW5ycC0+bnJf YXV0aCA9IG5mc19nZXRhdXRoKG5ycCwgc2VjZmxhdm91ciwKKwkJZGVzdHJveV9hdXRoID0gMDsK KwkJTkZTTE9DS1NPQ0tSRVEobnJwKTsKKwkJaWYgKG5ycC0+bnJfYXV0aCA9PSBOVUxMKSB7CisJ CQlORlNVTkxPQ0tTT0NLUkVRKG5ycCk7CisJCQlhdXRoID0gbmZzX2dldGF1dGgobnJwLCBzZWNm bGF2b3VyLAogCQkJICAgIGNsbnRfcHJpbmNpcGFsLCBzcnZfcHJpbmNpcGFsLCBOVUxMLCBhdXRo Y3JlZCk7Ci0JCWVsc2UKLQkJCXJwY19nc3NfcmVmcmVzaF9hdXRoX2NhbGwobnJwLT5ucl9hdXRo KTsKLQkJYXV0aCA9IG5ycC0+bnJfYXV0aDsKKwkJCU5GU0xPQ0tTT0NLUkVRKG5ycCk7CisJCQlp ZiAobnJwLT5ucl9hdXRoID09IE5VTEwpIHsKKwkJCQlucnAtPm5yX2F1dGggPSBhdXRoOworCQkJ CU5GU1VOTE9DS1NPQ0tSRVEobnJwKTsKKwkJCX0gZWxzZSB7CisJCQkJc2F2YXV0aCA9IGF1dGg7 CisJCQkJYXV0aCA9IG5ycC0+bnJfYXV0aDsKKwkJCQlORlNVTkxPQ0tTT0NLUkVRKG5ycCk7CisJ CQkJaWYgKHNhdmF1dGggIT0gTlVMTCkKKwkJCQkJREVTVFJPWV9BVVRIKHNhdmF1dGgpOworCQkJ fQorCQl9IGVsc2UgeworCQkJYXV0aCA9IG5ycC0+bnJfYXV0aDsKKwkJCU5GU1VOTE9DS1NPQ0tS RVEobnJwKTsKKwkJfQorCQlpZiAoYXV0aCAhPSBOVUxMKQorCQkJcnBjX2dzc19yZWZyZXNoX2F1 dGhfY2FsbChhdXRoKTsKIAl9IGVsc2UKIAkJYXV0aCA9IG5mc19nZXRhdXRoKG5ycCwgc2VjZmxh dm91ciwgTlVMTCwKIAkJICAgIHNydl9wcmluY2lwYWwsIE5VTEwsIGF1dGhjcmVkKTsKQEAgLTc0 MSw3ICs3NTksNyBAQCB0cnlhZ2FpbjoKIAl9CiAJaWYgKGVycm9yKSB7CiAJCW1fZnJlZW0obmQt Pm5kX21yZXEpOwotCQlpZiAodXNlZ3NzbmFtZSA9PSAwKQorCQlpZiAoZGVzdHJveV9hdXRoICE9 IDApCiAJCQlBVVRIX0RFU1RST1koYXV0aCk7CiAJCWlmIChyZXAgIT0gTlVMTCkKIAkJCUZSRUUo KGNhZGRyX3QpcmVwLCBNX05GU0RSRVEpOwpAQCAtODk5LDcgKzkxNyw3IEBAIHRyeWFnYWluOgog I2VuZGlmCiAKIAltX2ZyZWVtKG5kLT5uZF9tcmVxKTsKLQlpZiAodXNlZ3NzbmFtZSA9PSAwKQor CWlmIChkZXN0cm95X2F1dGggIT0gMCkKIAkJQVVUSF9ERVNUUk9ZKGF1dGgpOwogCWlmIChyZXAg IT0gTlVMTCkKIAkJRlJFRSgoY2FkZHJfdClyZXAsIE1fTkZTRFJFUSk7CkBAIC05MDksNyArOTI3 LDcgQEAgdHJ5YWdhaW46CiBuZnNtb3V0OgogCW1idWZfZnJlZW0obmQtPm5kX21yZXApOwogCW1i dWZfZnJlZW0obmQtPm5kX21yZXEpOwotCWlmICh1c2Vnc3NuYW1lID09IDApCisJaWYgKGRlc3Ry b3lfYXV0aCAhPSAwKQogCQlBVVRIX0RFU1RST1koYXV0aCk7CiAJaWYgKHJlcCAhPSBOVUxMKQog CQlGUkVFKChjYWRkcl90KXJlcCwgTV9ORlNEUkVRKTsK ------=_Part_1685353_717231508.1349303741299--