Date: Wed, 03 Aug 2011 12:32:42 +0300 From: Andriy Gapon <avg@FreeBSD.org> To: maestro something <maestro82@gmail.com> Cc: freebsd-stable@FreeBSD.org Subject: Re: dtrace ustack kernel panic Message-ID: <4E3915BA.2010801@FreeBSD.org> In-Reply-To: <4E3909ED.1000807@FreeBSD.org> References: <CAJ_JOqvEmXBTBABhUcJ66=bh9%2B8S%2BC9v30hXxVZiCXuEpGPJ1A@mail.gmail.com> <CAJ_JOqszViwLi6TeQxAxeX2Mte5eBPsGJpjQPVOQs2BOwAq9JQ@mail.gmail.com> <4E33B7CF.90200@FreeBSD.org> <CAJ_JOqt4VdgJm3NnB1KUf1RFuk75nu6-Rh=Bqb53h5TAEzB0%2BA@mail.gmail.com> <4E344D15.1040508@FreeBSD.org> <CAJ_JOqvCDE1Zcv4jz14rtZu5_NmqfY04tz91uy76WnaW2trO7Q@mail.gmail.com> <CAJ_JOqu2SR2n1Cm__3x3rfcJpYY=ihnoE9oyuKeNmKGuUGgnGA@mail.gmail.com> <20110730192646.GC17489@deviant.kiev.zoral.com.ua> <CAJ_JOqtofEjt-jiZBiLKW--fY0sdiHvF36baS3Qxc8ubmQMazg@mail.gmail.com> <CAJ_JOqtsrOr7Uq0Jo-Mb8kKnucjUarJxKrgzNEbMxTcGP7OCAw@mail.gmail.com> <CAJ_JOqsWNDeh_2SW5cz1PDewALJoANhuW81d4TjxWSK1w=nAjw@mail.gmail.com> <4E386636.2000507@FreeBSD.org> <4E386741.5030801@FreeBSD.org> <CAJ_JOqvmDMe5m1=cpoS95qJfCqX7EwuDqYGjLv4kbBPKCF=r2Q@mail.gmail.com> <CAN6yY1tC4-zAb1TGDCfLAyNhkN_RZeuhsj=C9M=LtyHMKOPDkg@mail.gmail.com> <CAJ_JOqtoGLwa4JMGFYxCcdCOo0-1n8eE-YFZW1N_Z%2BFpwqumaA@mail.gmail.com> <4E3909ED.1000807@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
on 03/08/2011 11:42 Andriy Gapon said the following:
> on 03/08/2011 03:47 maestro something said the following:
>> now it seems that the ustack() action on i386 and FreeBSD-9.0BETA1 almost works.
>>
>> Almost because dtrace still exits (this time without an error message) The
>> process for which the ustack should be genereated is terminated
>> Here are the specifics:
>>
>> fb90i386# dtrace -n 'syscall::accept:return / execname == "nc" / { ustack();}'
>> dtrace: description 'syscall::accept:return ' matched 1 probe
>> CPU ID FUNCTION:NAME
>> 0 46457 accept:return
>> libc.so.7`__sys_accept+0x7
>> 0x3
>> 0xed96e824
>>
>>
>> This alsmost looks like a stack trace I'd say
>> However, dtrace quits once I connect to nc
>>
>> nc quits with the following error message
>>
>> fb90i386# nc -vl 4444
>> nc: Polling Error: Interrupted system call
>
> This could be a combination of several factors.
> First, for some reason nc gets EINTR in one of its system calls (poll(2)).
> Second, nc treats EINTR as a serious error and exits.
Looks like nc gets EINTR when dtrace does ptrace(PT_CONTINUE).
So this is the culprit for your test case.
> Third, this is what happens on the dtrace side:
Recorded dtrace behavior seems to be correct.
> 30495 100974 dtrace CALL write(0x1,0x8035c2000,0x1a)
> 30495 100974 dtrace GIO fd 1 wrote 26 bytes
> "libc.so.7`__sys_accept+0xc"
> 30495 100974 dtrace RET write 26/0x1a
> 30495 100974 dtrace CALL write(0x1,0x8035c2000,0x1)
> 30495 100974 dtrace GIO fd 1 wrote 1 byte
> "
> "
> 30495 100974 dtrace RET write 1
> 30495 100974 dtrace CALL thr_kill(0x1c129,SIGUSR1)
> 30495 100974 dtrace RET thr_kill 0
> 30495 100974 dtrace CALL ptrace(PT_IO,0x771e,0x7fffffffbb10,0)
> 30495 100974 dtrace RET ptrace 0
> 30495 100974 dtrace CALL ptrace(PT_IO,0x771e,0x7fffffffbb10,0)
> 30495 114985 dtrace RET _umtx_op -1 errno 4 Interrupted system call
> 30495 100974 dtrace RET ptrace 0
> 30495 114985 dtrace PSIG SIGUSR1 caught handler=0x80085d610 mask=0x5ffefedf
> code=0x10007
> 30495 100974 dtrace CALL _umtx_op(0x802007668,0x15,0x1,0,0)
> 30495 114985 dtrace CALL sigprocmask(SIG_SETMASK,0x7fffffbfd9fc,0)
> 30495 100974 dtrace RET _umtx_op 0
> 30495 114985 dtrace RET sigprocmask 0
> 30495 100974 dtrace CALL _umtx_op(0x800a71778,0xf,0,0,0)
> 30495 114985 dtrace CALL sigreturn(0x7fffffbfd630)
> 30495 114985 dtrace RET sigreturn JUSTRETURN
> 30495 114985 dtrace CALL ptrace(PT_CONTINUE,0x771e,0x1,0)
> 30495 114985 dtrace RET ptrace 0
> 30495 114985 dtrace CALL ptrace(PT_IO,0x771e,0x7fffffbfdee0,0)
> 30495 114985 dtrace RET ptrace -1 errno 16 Device busy
> 30495 114985 dtrace CALL _umtx_op(0x80200d668,0x15,0x1,0,0)
> 30495 114985 dtrace RET _umtx_op 0
> 30495 114985 dtrace CALL madvise(0x803c52000,0x11000,MADV_FREE)
> 30495 114985 dtrace RET madvise 0
> 30495 114985 dtrace CALL madvise(0x803c1d000,0x14000,MADV_FREE)
> 30495 114985 dtrace RET madvise 0
> 30495 114985 dtrace CALL madvise(0x803c07000,0x1000,MADV_FREE)
> 30495 114985 dtrace RET madvise 0
> 30495 114985 dtrace CALL madvise(0x8037fc000,0x2000,MADV_FREE)
> 30495 114985 dtrace RET madvise 0
> 30495 114985 dtrace CALL madvise(0x8037f2000,0x8000,MADV_FREE)
> 30495 114985 dtrace RET madvise 0
> 30495 114985 dtrace CALL madvise(0x8037f0000,0x1000,MADV_FREE)
> 30495 100974 dtrace RET _umtx_op 0
> 30495 114985 dtrace RET madvise 0
> 30495 100974 dtrace CALL ptrace(PT_DETACH,0x771e,0,0)
> 30495 114985 dtrace CALL madvise(0x8037e0000,0x9000,MADV_FREE)
> 30495 100974 dtrace RET ptrace -1 errno 3 No such process
>
> Looks like something fishy happens when that SIGUSR1 is generated.
> I wish a dtrace-for-userland expert and/or a ptrace(2) expert could help us here.
>
--
Andriy Gapon
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4E3915BA.2010801>