From owner-freebsd-current@FreeBSD.ORG Thu Oct 23 00:03:13 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4C535BA5; Thu, 23 Oct 2014 00:03:13 +0000 (UTC) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id A841887B; Thu, 23 Oct 2014 00:03:12 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ArsEAJpESFSDaFve/2dsb2JhbABcg2JYBIMCyVMKhnlUAoEhAX2EAgEBAQMBAQEBICsgCwUWGAICDRkCIwYBCSYOBwQBCBEDBIgMAwkIDbEUjg8NhjgBAQEHAQEBAQEdgSyMdYFeBgEBAQUVNAeCNkESgUIFlkuEDnODQjyNYoJdhAGEFCEvB4EACBcigQMBAQE X-IronPort-AV: E=Sophos;i="5.04,772,1406606400"; d="scan'208";a="161664397" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 22 Oct 2014 20:03:04 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 6A776B4039; Wed, 22 Oct 2014 20:03:04 -0400 (EDT) Date: Wed, 22 Oct 2014 20:03:04 -0400 (EDT) From: Rick Macklem To: araujo@FreeBSD.org Message-ID: <1786165726.5540137.1414022584423.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: kernel page fault with nfs MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.209] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: freebsd-current , "Tobias C. Berner" , Allan Jude X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Oct 2014 00:03:13 -0000 Tobias C. Berner wrote: > Tobias, > > Thank you very much, it really helps to simulate the problem. > I'm gonna try as soon as possible and I will keep you informed. > > Best Regards, > > 2014-10-21 17:10 GMT+08:00 Tobias C. Berner : > > > Hi Marcelo > > > > > > > > The following ist the current fstab-line which seems to run > > smoothly: > > > > odo.firefly:/storage/multimedia /multimedia nfs > > readahead=4,soft,intr,rw,tcp,wsize=32768,rsize=32768,late 0 0 > > I've just committed r273486 to head, which rounds rsize, wsize down to the power of 2 less than or equal to the value. I don't think anyone needs support for non-power of 2 rsize, wsize values, so this seems to be a reasonable resolution, given that the actual bug/fix is not known. rick > > > > > > nfsstat -m: > > > > odo.firefly:/storage/multimedia on /multimedia > > > > > > nfsv3,tcp,resvport,soft,intr,cto,lockd,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=32768,wsize=32768,readdirsize=32768,readahead=4,wcommitsize=2798255,timeout=120,retrans=2 > > > > > > > > > > > > > > > > > > > > Now the bad line (no different appart from the typo) > > > > odo.firefly:/storage/multimedia /multimedia nfs > > readahead=4,soft,intr,rw,tcp,wsize=32767,rsize=32767,late 0 0 > > > > which leads to the page-faults. > > > > And as you said wsize/rsize gets rounded down to the multiple of > > 512: > > > > odo.firefly:/storage/multimedia on /multimedia > > > > > > nfsv3,tcp,resvport,soft,intr,cto,lockd,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=32256,wsize=32256,readdirsize=32256,readahead=4,wcommitsize=2798255,timeout=120,retrans=2 > > > > > > > > > > > > I can easily reproduce the pagefault by letting for example > > multimedia/gpodder write to the nfs. > > > > > > > > > > > > > > > > hope this helps, > > > > > > > > mfg Tobias > > > > > > > > > > > > > > > > > > > > On Tuesday 21 October 2014 15.45:24 Marcelo Araujo wrote: > > > > > Hello Tobias, > > > > > > > > > > That sounds good, at least you don't have any crash so far. > > > > > I agree with you, seems a bug, I'm gonna take a look on that. > > > > > > > > > > Could you share with me your testbed or how you can reproduce the > > > issue? > > > > > > > > > > Best Regards, > > > > > > > > > > 2014-10-21 15:36 GMT+08:00 T.C.Berner : > > > > > > The system now has an uptime of >24h using NFS heavily. > > > > > > > > > > > > So wsize/rsize=2^15-1 seems to have been the problem.... which > > > > is imho > > a > > > > > > bug therefore. > > > > > > > > > > > > > > > > > > mfg Tobias > > > > > > > > > > > > 2014-10-21 5:11 GMT+02:00 Marcelo Araujo > > > > : > > > > > >> Hello Tibias, > > > > > >> > > > > > >> Any news? > > > > > >> > > > > > >> > > > > > >> Best Regards, > > > > > >> > > > > > >> 2014-10-20 20:55 GMT+08:00 Rick Macklem > > > >> : > > > > > >>> Tobias C. Berner wrote: > > > > > >>> > Now that I posted it, 32767 should of course be 2^15=32768. > > > >>> > Let me > > > > > >>> > recheck if it still > > > > > >>> > hangs with the correct value. > > > > > >>> > > > > > > >>> > On Monday 20 October 2014 09.15:39 Tobias C. Berner wrote: > > > > > >>> > > Hi Marcelo > > > > > >>> > > > > > > > >>> > > Yes, I'm using readahead: > > > > > >>> > > The mountoptions are > > > > > >>> > > "readahead=4,soft,intr,rw,tcp,wsize=32767,rsize=32767,late" > > > > > >>> > > > > > >>> If you type "nfsstat -m", you will see what is actually > > > >>> getting used. > > > > > >>> (I suspect the above rsize/wsize got clipped to 32256 or > > > >>> something > > like > > > > > >>> > > > > > >>> that. I think it clips it to a multiple of 512.) > > > > > >>> > > > > > >>> If rsize/wsize are not a power of 2, there are issues, > > > >>> although I've > > > > > >>> never > > > > > >>> been able to see why it is broken. Maybe it should clip it to > > > >>> the > > power > > > > > >>> of > > > > > >>> 2 below the value, since it causes unexplained problems > > > >>> otherwise. > > > > > >>> > > > > > >>> rick > > > > > >>> > > > > > >>> > > mfg Tobias > > > > > >>> > > > > > > > >>> > > On Monday 20 October 2014 10.41:30 Marcelo Araujo wrote: > > > > > >>> > > > Hello Tobias, > > > > > >>> > > > > > > > > >>> > > > Could you show how you are mount the NFS share? > > > > > >>> > > > Are you using 'readahead' option? > > > > > >>> > > > > > > > > >>> > > > Best Regards, > > > > > >>> > > > > > > > > >>> > > > 2014-10-19 17:40 GMT+08:00 Tobias C. Berner < > > tcberner@gmail.com>: > > > > > >>> > > > > both are at 1100038. > > > > > >>> > > > > > > > > > >>> > > > > On Sunday 19 October 2014 11.12:36 Marcelo Araujo > > > >>> > > > > wrote: > > > > > >>> > > > > > It is still strange, could you do what Allan said > > > >>> > > > > > and send > > us > > > > > >>> > > > > > the > > > > > >>> > > > > > result > > > > > >>> > > > > > > > > > >>> > > > > in > > > > > >>> > > > > > > > > > >>> > > > > > case you are not sure you have world and kernel in > > > >>> > > > > > the same > > > > > >>> > > > > > revision! > > > > > >>> > > > > > > > > > > >>> > > > > > On Oct 19, 2014 6:48 AM, "Tobias C. Berner" > > > > > >>> > > > > > > > > > > >>> > > > > > wrote: > > > > > >>> > > > > > > Hi > > > > > >>> > > > > > > > > > > > >>> > > > > > > World ist from october 16, installed world and > > > >>> > > > > > > kernel > > then. > > > > > >>> > > > > > > > > > > > >>> > > > > > > Kernel was later rebuilt with debug-options. > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > Is the following more sensible? > > > > > >>> > > > > > > >>> > ################################################## > > > > > >>> > > > > > > >>> > > > > > > # kgdb NOXON/kernel.debug vmcore.1 > > > > > >>> > > > > > > > > > > > >>> > > > > > > Fatal trap 12: page fault while in kernel mode > > > > > >>> > > > > > > > > > > > >>> > > > > > > cpuid = 5; apic id = 05 > > > > > >>> > > > > > > > > > > > >>> > > > > > > fault virtual address = 0xfffffe07d1744000 > > > > > >>> > > > > > > > > > > > >>> > > > > > > fault code = supervisor write data, page not > > > >>> > > > > > > present > > > > > >>> > > > > > > > > > > > >>> > > > > > > instruction pointer = 0x20:0xffffffff80d4d58a > > > > > >>> > > > > > > > > > > > >>> > > > > > > stack pointer = 0x28:0xfffffe086057f240 > > > > > >>> > > > > > > > > > > > >>> > > > > > > frame pointer = 0x28:0xfffffe086057f2f0 > > > > > >>> > > > > > > > > > > > >>> > > > > > > code segment = base 0x0, limit 0xfffff, type 0x1b > > > > > >>> > > > > > > > > > > > >>> > > > > > > = DPL 0, pres 1, long 1, def32 0, gran 1 > > > > > >>> > > > > > > > > > > > >>> > > > > > > processor eflags = interrupt enabled, resume, > > > >>> > > > > > > IOPL = 0 > > > > > >>> > > > > > > > > > > > >>> > > > > > > current process = 6524 (python2.7) > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > (kgdb) bt > > > > > >>> > > > > > > > > > > > >>> > > > > > > #0 doadump (textdump=1) at pcpu.h:219 > > > > > >>> > > > > > > > > > > > >>> > > > > > > #1 0xffffffff80926b6d in kern_reboot (howto=260) > > > >>> > > > > > > at > > > > > >>> > > > > > > /usr/src/sys/kern/kern_shutdown.c:447 > > > > > >>> > > > > > > > > > > > >>> > > > > > > #2 0xffffffff809270c0 in panic (fmt= > > >>> > > > > > > optimized > > out>) > > > > > >>> > > > > > > at > > > > > >>> > > > > > > /usr/src/sys/kern/kern_shutdown.c:746 > > > > > >>> > > > > > > > > > > > >>> > > > > > > #3 0xffffffff8035f167 in db_panic (addr= > > >>> > > > > > > optimized > > > > > >>> > > > > > > out>, > > > > > >>> > > > > > > have_addr=2, count=0, modif=0x0) at > > > > > >>> > > > > > > /usr/src/sys/ddb/db_command.c:473 > > > > > >>> > > > > > > > > > > > >>> > > > > > > #4 0xffffffff8035ed7d in db_command > > > >>> > > > > > > (cmd_table=0x0) at > > > > > >>> > > > > > > /usr/src/sys/ddb/db_command.c:440 > > > > > >>> > > > > > > > > > > > >>> > > > > > > #5 0xffffffff8035eaf4 in db_command_loop () at > > > > > >>> > > > > > > /usr/src/sys/ddb/db_command.c:493 > > > > > >>> > > > > > > > > > > > >>> > > > > > > #6 0xffffffff80361600 in db_trap (type= > > >>> > > > > > > optimized > > > > > >>> > > > > > > out>, > > > > > >>> > > > > > > code=0) > > > > > >>> > > > > > > > > > >>> > > > > at > > > > > >>> > > > > > > > > > >>> > > > > > > /usr/src/sys/ddb/db_main.c:251 > > > > > >>> > > > > > > > > > > > >>> > > > > > > #7 0xffffffff80966f01 in kdb_trap (type=12, > > > >>> > > > > > > code=0, > > > > > >>> > > > > > > tf= > > > > >>> > > > > > > optimized > > > > > >>> > > > > > > out>) at /usr/src/sys/kern/subr_kdb.c:654 > > > > > >>> > > > > > > > > > > > >>> > > > > > > #8 0xffffffff80d4fa7c in trap_fatal > > > > > >>> > > > > > > (frame=0xfffffe086057f190, > > > > > >>> > > > > > > > > > >>> > > > > eva= > > > > >>> > > > > > > > > > >>> > > > > > > optimized out>) at > > > >>> > > > > > > /usr/src/sys/amd64/amd64/trap.c:861 > > > > > >>> > > > > > > > > > > > >>> > > > > > > #9 0xffffffff80d4fe0c in trap_pfault > > > > > >>> > > > > > > (frame=0xfffffe086057f190, > > > > > >>> > > > > > > usermode=) at > > > > > >>> > > > > > > /usr/src/sys/amd64/amd64/trap.c:677 > > > > > >>> > > > > > > > > > > > >>> > > > > > > #10 0xffffffff80d4f42e in trap > > > >>> > > > > > > (frame=0xfffffe086057f190) > > > > > >>> > > > > > > at > > > > > >>> > > > > > > /usr/src/sys/amd64/amd64/trap.c:426 > > > > > >>> > > > > > > > > > > > >>> > > > > > > #11 0xffffffff80d33972 in calltrap () at > > > > > >>> > > > > > > /usr/src/sys/amd64/amd64/exception.S:231 > > > > > >>> > > > > > > > > > > > >>> > > > > > > #12 0xffffffff80d4d58a in bzero () at > > > > > >>> > > > > > > /usr/src/sys/amd64/amd64/support.S:53 > > > > > >>> > > > > > > > > > > > >>> > > > > > > #13 0xffffffff80830463 in ncl_doio > > (vp=0xfffff801e7f99938, > > > > > >>> > > > > > > bp=0xfffffe07c5a168e8, cr=, > > td= > > > > >>> > > > > > > optimized > > > > > >>> > > > > > > > > > >>> > > > > out>, > > > > > >>> > > > > > > > > > >>> > > > > > > called_from_strategy=) > > > > > >>> > > > > > > > > > > > >>> > > > > > > at /usr/src/sys/fs/nfsclient/nfs_clbio.c:1648 > > > > > >>> > > > > > > >>> > _______________________________________________ > > > > > >>> > freebsd-current@freebsd.org mailing list > > > > > >>> > http://lists.freebsd.org/mailman/listinfo/freebsd-current > > > > > >>> > To unsubscribe, send any mail to > > > > > >>> > "freebsd-current-unsubscribe@freebsd.org" > > > > > >> > > > > > >> -- > > > > > >> > > > > > >> -- > > > > > >> Marcelo Araujo (__)araujo@FreeBSD.org > > > > > >> \\\'',)http://www.FreeBSD.org \/ \ ^ > > > >> Power > > > > > >> To Server. .\. /_) > > > > > > > > > > -- > > -- > Marcelo Araujo (__)araujo@FreeBSD.org > \\\'',)http://www.FreeBSD.org \/ \ ^ > Power To Server. .\. /_) > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to > "freebsd-current-unsubscribe@freebsd.org" >