Date: Fri, 23 Sep 2011 20:54:24 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Clinton Adams <clinton.adams@gmail.com> Cc: freebsd-fs@freebsd.org Subject: Re: kernel panics with RPCSEC_GSS Message-ID: <1498466253.1940252.1316825664747.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <CAEuopLYJ6fvZ_7hCHQdjKgdNC39kzK1DwyGib7XvvrqE5Pqozw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_1940251_322460862.1316825664744 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Clinton Adams wrote: > Hi, > > On 8.2-RELEASE-p2, kernel occassionaly panics during relatively high > nfs usage (usually morning logins). Frequency of crashes have > decreased as we have reduced the number of clients, about twice a week > with 10 clients versus daily with 15. > > Server is running nfsv4 with mit kerberos, clients are linux (ubuntu > 10.04). > > Backtraces from last 2 cores - > > #1 0xffffffff805cbb5e in boot (howto=260) > at /usr/src/sys/kern/kern_shutdown.c:419 > #2 0xffffffff805cbf91 in panic (fmt=Variable "fmt" is not available. > ) at /usr/src/sys/kern/kern_shutdown.c:592 > #3 0xffffffff808d25c0 in trap_fatal (frame=0xc, eva=Variable "eva" is > not available. > ) > at /usr/src/sys/amd64/amd64/trap.c:783 > #4 0xffffffff808d299f in trap_pfault (frame=0xffffff8096bb7790, > usermode=0) > at /usr/src/sys/amd64/amd64/trap.c:699 > #5 0xffffffff808d2e7f in trap (frame=0xffffff8096bb7790) > at /usr/src/sys/amd64/amd64/trap.c:449 > #6 0xffffffff808baf74 in calltrap () > at /usr/src/sys/amd64/amd64/exception.S:224 > #7 0xffffffff807db8d8 in svc_rpc_gss_forget_client (client=0x0) > at /usr/src/sys/rpc/rpcsec_gss/svc_rpcsec_gss.c:616 > #8 0xffffffff807dc1c3 in svc_rpc_gss (rqst=0xffffff005708c000, > msg=0xffffff8096bb7b20) at > /usr/src/sys/rpc/rpcsec_gss/svc_rpcsec_gss.c:642 Well, here's the code snippet... while (svc_rpc_gss_client_count > CLIENT_MAX) 642 svc_rpc_gss_forget_client(TAILQ_LAST(&svc_rpc_gss_clients, 643 svc_rpc_gss_client_list)); >From the above, it looks like the "client" returned by TAILQ_LAST() is bogus. A quick look at the code shows that all changes to that tailq and the value of svc_rpc_gss_client_count are protected by a sx lock, however this lock isn't held here. (svc_rpc_gss_client_count is decremented in svc_rpc_gss_forget_client().) svc_rpc_gss_client_count only seems to be incremented when an entry is added to the tailq and decremented in svc_rpc_gss_forget_client() when an entry is removed from the tailq, so I can't see how it's value would get messed up? All I can think of is trying adding locking to the above. Could you please try the attached patch. rick > #9 0xffffffff807d49d3 in svc_run_internal (pool=0xffffff003d03d600, > ismaster=0) at /usr/src/sys/rpc/svc.c:837 > #10 0xffffffff807d518b in svc_thread_start (arg=Variable "arg" is not > available. > ) > at /usr/src/sys/rpc/svc.c:1200 > #11 0xffffffff805a2798 in fork_exit ( > callout=0xffffffff807d5180 <svc_thread_start>, arg=0xffffff003d03d600, > frame=0xffffff8096bb7c40) at /usr/src/sys/kern/kern_fork.c:845 > #12 0xffffffff808bb43e in fork_trampoline () > at /usr/src/sys/amd64/amd64/exception.S:565 > > > #1 0xffffffff805cbabe in boot (howto=260) > at /usr/src/sys/kern/kern_shutdown.c:419 > #2 0xffffffff805cbed3 in panic (fmt=0x0) > at /usr/src/sys/kern/kern_shutdown.c:592 > #3 0xffffffff808d239d in trap_fatal (frame=0xffffff0004c89460, > eva=Variable "eva" is not available. > ) > at /usr/src/sys/amd64/amd64/trap.c:783 > #4 0xffffffff808d275f in trap_pfault (frame=0xffffff8096c0d790, > usermode=0) > at /usr/src/sys/amd64/amd64/trap.c:699 > #5 0xffffffff808d2b5f in trap (frame=0xffffff8096c0d790) > at /usr/src/sys/amd64/amd64/trap.c:449 > #6 0xffffffff808bada4 in calltrap () > at /usr/src/sys/amd64/amd64/exception.S:224 > #7 0xffffffff807db856 in svc_rpc_gss_forget_client > (client=0xffffff001c015200) > at atomic.h:158 > #8 0xffffffff807dc0e3 in svc_rpc_gss (rqst=0xffffff0004a24000, > msg=0xffffff8096c0db20) at > /usr/src/sys/rpc/rpcsec_gss/svc_rpcsec_gss.c:642 > #9 0xffffffff807d48f3 in svc_run_internal (pool=0xffffff0004ca6200, > ismaster=0) at /usr/src/sys/rpc/svc.c:837 > #10 0xffffffff807d50ab in svc_thread_start (arg=Variable "arg" is not > available. > ) > at /usr/src/sys/rpc/svc.c:1200 > #11 0xffffffff805a26f8 in fork_exit ( > callout=0xffffffff807d50a0 <svc_thread_start>, arg=0xffffff0004ca6200, > frame=0xffffff8096c0dc40) at /usr/src/sys/kern/kern_fork.c:845 > #12 0xffffffff808bb26e in fork_trampoline () > at /usr/src/sys/amd64/amd64/exception.S:565 > > Kernel is generic except for > device crypto > options KGSSAPI. > > Ash /etc/make.conf > WITHOUT_X11=yes > KRB5_HOME=/usr/local > KRB5_IMPL=mit > # added by use.perl 2011-09-02 11:38:57 > PERL_VERSION=5.10.1 > > I'm happy to provide any additional info. > > Thanks for any help, > Clinton > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" ------=_Part_1940251_322460862.1316825664744 Content-Type: text/x-patch; name=svcrpcsec.patch Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename=svcrpcsec.patch LS0tIHJwYy9ycGNzZWNfZ3NzL3N2Y19ycGNzZWNfZ3NzLmMuc2F2CTIwMTEtMDktMjMgMjA6MTQ6 MDcuMDAwMDAwMDAwIC0wNDAwCisrKyBycGMvcnBjc2VjX2dzcy9zdmNfcnBjc2VjX2dzcy5jCTIw MTEtMDktMjMgMjA6MjE6MjguMDAwMDAwMDAwIC0wNDAwCkBAIC02MzgsMTYgKzYzOCwyNSBAQCBz dmNfcnBjX2dzc190aW1lb3V0X2NsaWVudHModm9pZCkKIAkgKiBGaXJzdCBlbmZvcmNlIHRoZSBt YXggY2xpZW50IGxpbWl0LiBXZSBrZWVwCiAJICogc3ZjX3JwY19nc3NfY2xpZW50cyBpbiBMUlUg b3JkZXIuCiAJICovCi0Jd2hpbGUgKHN2Y19ycGNfZ3NzX2NsaWVudF9jb3VudCA+IENMSUVOVF9N QVgpCi0JCXN2Y19ycGNfZ3NzX2ZvcmdldF9jbGllbnQoVEFJTFFfTEFTVCgmc3ZjX3JwY19nc3Nf Y2xpZW50cywKLQkJCSAgICBzdmNfcnBjX2dzc19jbGllbnRfbGlzdCkpOworCXN4X3hsb2NrKCZz dmNfcnBjX2dzc19sb2NrKTsKKwljbGllbnQgPSBUQUlMUV9MQVNUKCZzdmNfcnBjX2dzc19jbGll bnRzLCBzdmNfcnBjX2dzc19jbGllbnRfbGlzdCk7CisJd2hpbGUgKHN2Y19ycGNfZ3NzX2NsaWVu dF9jb3VudCA+IENMSUVOVF9NQVggJiYgY2xpZW50ICE9IE5VTEwpIHsKKwkJc3hfeHVubG9jaygm c3ZjX3JwY19nc3NfbG9jayk7CisJCXN2Y19ycGNfZ3NzX2ZvcmdldF9jbGllbnQoY2xpZW50KTsK KwkJc3hfeGxvY2soJnN2Y19ycGNfZ3NzX2xvY2spOworCQljbGllbnQgPSBUQUlMUV9MQVNUKCZz dmNfcnBjX2dzc19jbGllbnRzLAorCQkgICAgc3ZjX3JwY19nc3NfY2xpZW50X2xpc3QpOworCX0K IAlUQUlMUV9GT1JFQUNIX1NBRkUoY2xpZW50LCAmc3ZjX3JwY19nc3NfY2xpZW50cywgY2xfYWxs bGluaywgbmNsaWVudCkgewogCQlpZiAoY2xpZW50LT5jbF9zdGF0ZSA9PSBDTElFTlRfU1RBTEUK IAkJICAgIHx8IG5vdyA+IGNsaWVudC0+Y2xfZXhwaXJhdGlvbikgeworCQkJc3hfeHVubG9jaygm c3ZjX3JwY19nc3NfbG9jayk7CiAJCQlycGNfZ3NzX2xvZ19kZWJ1ZygiZXhwaXJpbmcgY2xpZW50 ICVwIiwgY2xpZW50KTsKIAkJCXN2Y19ycGNfZ3NzX2ZvcmdldF9jbGllbnQoY2xpZW50KTsKKwkJ CXN4X3hsb2NrKCZzdmNfcnBjX2dzc19sb2NrKTsKIAkJfQogCX0KKwlzeF94dW5sb2NrKCZzdmNf cnBjX2dzc19sb2NrKTsKIH0KIAogI2lmZGVmIERFQlVHCg== ------=_Part_1940251_322460862.1316825664744--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1498466253.1940252.1316825664747.JavaMail.root>