Date: Mon, 14 Nov 2016 10:35:20 +0100 From: Henri Hennebert <hlh@restart.be> To: Andriy Gapon <avg@FreeBSD.org>, freebsd-stable@FreeBSD.org Cc: Konstantin Belousov <kib@FreeBSD.org> Subject: Re: Freebsd 11.0 RELEASE - ZFS deadlock Message-ID: <26512d69-94c2-92da-e3ea-50aebf17e3a0@restart.be> In-Reply-To: <af4e0c2b-00f8-bbaa-bcb7-d97062a393b8@FreeBSD.org> References: <0c223160-b76f-c635-bb15-4a068ba7efe7@restart.be> <aaf2df40-b0df-2141-9ed8-5b947d8d5a33@FreeBSD.org> <43c9d4d4-1995-5626-d70a-f92a5b456629@FreeBSD.org> <a14d508d-351f-71f4-c7cc-ac73dbcde357@restart.be> <9d1f9a76-5a8d-6eca-9a50-907d55099847@FreeBSD.org> <6bc95dce-31e1-3013-bfe3-7c2dd80f9d1e@restart.be> <e4878992-a362-3f12-e743-8efa1347cabf@FreeBSD.org> <23a66749-f138-1f1a-afae-c775f906ff37@restart.be> <8e7547ef-87f7-7fab-6f45-221e8cea1989@FreeBSD.org> <6d991cea-b420-531e-12cc-001e4aeed66b@restart.be> <67f2e8bd-bff0-f808-7557-7dabe5cad78c@FreeBSD.org> <1cb09c54-5f0e-2259-a41a-fefe76b4fe8b@restart.be> <d25c8035-b710-5de9-ebe3-7990b2d0e3b1@FreeBSD.org> <9f20020b-e2f1-862b-c3fc-dc6ff94e301e@restart.be> <c1b7aa94-1f1d-7edd-8764-adb72fdc053c@FreeBSD.org> <599c5a5b-aa08-2030-34f3-23ff19d09a9b@restart.be> <32686283-948a-6faf-7ded-ed8fcd23affb@FreeBSD.org> <cf0fc1e3-b621-074e-1351-4dd89d980ddd@restart.be> <af4e0c2b-00f8-bbaa-bcb7-d97062a393b8@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 11/14/2016 10:07, Andriy Gapon wrote: > On 13/11/2016 15:28, Henri Hennebert wrote: >> On 11/13/2016 11:06, Andriy Gapon wrote: >>> On 12/11/2016 14:40, Henri Hennebert wrote: > [snip] > > Could you please show 'info local' in frame 14? > I expected that 'nd' variable would be defined there and it may contain some > useful information. > No luck there: (kgdb) fr 14 #14 0xffffffff80636838 in kern_statat (td=0xfffff80009ba0500, flag=<value optimized out>, fd=-100, path=0x0, pathseg=<value optimized out>, sbp=<value optimized out>, hook=0x800e2a388) at /usr/src/sys/kern/vfs_syscalls.c:2160 2160 if ((error = namei(&nd)) != 0) (kgdb) info local rights = <value optimized out> nd = <value optimized out> error = <value optimized out> sb = <value optimized out> (kgdb) >> I also try to get information from the execve of the other treads: >> >> for tid 101250: >> (kgdb) fr 10 >> #10 0xffffffff80508ccc in sys_execve (td=0xfffff800b6429000, >> uap=0xfffffe010184fb80) at /usr/src/sys/kern/kern_exec.c:218 >> 218 error = kern_execve(td, &args, NULL); >> (kgdb) print *uap >> $4 = {fname_l_ = 0xfffffe010184fb80 "`\220\217\002\b", fname = 0x8028f9060 >> <Address 0x8028f9060 out of bounds>, >> fname_r_ = 0xfffffe010184fb88 "`¶ÿÿÿ\177", argv_l_ = 0xfffffe010184fb88 >> "`¶ÿÿÿ\177", argv = 0x7fffffffb660, >> argv_r_ = 0xfffffe010184fb90 "\bÜÿÿÿ\177", envv_l_ = 0xfffffe010184fb90 >> "\bÜÿÿÿ\177", envv = 0x7fffffffdc08, >> envv_r_ = 0xfffffe010184fb98 ""} >> (kgdb) >> >> for tid 101243: >> >> (kgdb) f 15 >> #15 0xffffffff80508ccc in sys_execve (td=0xfffff800b642b500, >> uap=0xfffffe010182cb80) at /usr/src/sys/kern/kern_exec.c:218 >> 218 error = kern_execve(td, &args, NULL); >> (kgdb) print *uap >> $5 = {fname_l_ = 0xfffffe010182cb80 "ÀÏ\205\002\b", fname = 0x80285cfc0 <Address >> 0x80285cfc0 out of bounds>, >> fname_r_ = 0xfffffe010182cb88 "`¶ÿÿÿ\177", argv_l_ = 0xfffffe010182cb88 >> "`¶ÿÿÿ\177", argv = 0x7fffffffb660, >> argv_r_ = 0xfffffe010182cb90 "\bÜÿÿÿ\177", envv_l_ = 0xfffffe010182cb90 >> "\bÜÿÿÿ\177", envv = 0x7fffffffdc08, >> envv_r_ = 0xfffffe010182cb98 ""} >> (kgdb) > > I think that you see garbage in those structures because they contain pointers > to userland data. > > Hmm, I've just noticed another interesting thread: > Thread 668 (Thread 101245): > #0 sched_switch (td=0xfffff800b642aa00, newtd=0xfffff8000285f000, flags=<value > optimized out>) at /usr/src/sys/kern/sched_ule.c:1973 > #1 0xffffffff80561ae2 in mi_switch (flags=<value optimized out>, newtd=0x0) at > /usr/src/sys/kern/kern_synch.c:455 > #2 0xffffffff805ae8da in sleepq_wait (wchan=0x0, pri=0) at > /usr/src/sys/kern/subr_sleepqueue.c:646 > #3 0xffffffff805614b1 in _sleep (ident=<value optimized out>, lock=<value > optimized out>, priority=<value optimized out>, wmesg=0xffffffff809c51bc > "vmpfw", sbt=0, pr=<value optimized out>, flags=<value optimized out>) at > /usr/src/sys/kern/kern_synch.c:229 > #4 0xffffffff8089d1c1 in vm_page_busy_sleep (m=0xfffff800df68cd40, wmesg=<value > optimized out>) at /usr/src/sys/vm/vm_page.c:753 > #5 0xffffffff8089dd4d in vm_page_sleep_if_busy (m=0xfffff800df68cd40, > msg=0xffffffff809c51bc "vmpfw") at /usr/src/sys/vm/vm_page.c:1086 > #6 0xffffffff80886be9 in vm_fault_hold (map=<value optimized out>, vaddr=<value > optimized out>, fault_type=4 '\004', fault_flags=0, m_hold=0x0) at > /usr/src/sys/vm/vm_fault.c:495 > #7 0xffffffff80885448 in vm_fault (map=0xfffff80011d66000, vaddr=<value > optimized out>, fault_type=4 '\004', fault_flags=<value optimized out>) at > /usr/src/sys/vm/vm_fault.c:273 > #8 0xffffffff808d3c49 in trap_pfault (frame=0xfffffe0101836c00, usermode=1) at > /usr/src/sys/amd64/amd64/trap.c:741 > #9 0xffffffff808d3386 in trap (frame=0xfffffe0101836c00) at > /usr/src/sys/amd64/amd64/trap.c:333 > #10 0xffffffff808b7af1 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236 This tread is another program from the news system: 668 Thread 101245 (PID=49124: innfeed) sched_switch (td=0xfffff800b642aa00, newtd=0xfffff8000285f000, flags=<value optimized out>) at /usr/src/sys/kern/sched_ule.c:1973 > > I strongly suspect that this is thread that we were looking for. > I think that it has the vnode lock in the shared mode while trying to fault in a > page. > > Could you please check that by going to frame 6 and printing 'fs' and '*fs.vp'? > It'd be interesting to understand why this thread is waiting here. > So, please also print '*fs.m' and '*fs.object'. No luck :-( (kgdb) fr 6 #6 0xffffffff80886be9 in vm_fault_hold (map=<value optimized out>, vaddr=<value optimized out>, fault_type=4 '\004', fault_flags=0, m_hold=0x0) at /usr/src/sys/vm/vm_fault.c:495 495 vm_page_sleep_if_busy(fs.m, "vmpfw"); (kgdb) print fs Cannot access memory at address 0xffff00001fa0 (kgdb) Henri
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?26512d69-94c2-92da-e3ea-50aebf17e3a0>