From owner-freebsd-current Sun Jun 30 20: 0:28 2002 Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2727937B400 for ; Sun, 30 Jun 2002 20:00:20 -0700 (PDT) Received: from sccrmhc02.attbi.com (sccrmhc02.attbi.com [204.127.202.62]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3F78643E0A for ; Sun, 30 Jun 2002 20:00:19 -0700 (PDT) (envelope-from julian@elischer.org) Received: from InterJet.elischer.org ([12.232.206.8]) by sccrmhc02.attbi.com (InterMail vM.4.01.03.27 201-229-121-127-20010626) with ESMTP id <20020701030018.VQYB6023.sccrmhc02.attbi.com@InterJet.elischer.org>; Mon, 1 Jul 2002 03:00:18 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id TAA88029; Sun, 30 Jun 2002 19:47:41 -0700 (PDT) Date: Sun, 30 Jun 2002 19:47:40 -0700 (PDT) From: Julian Elischer To: David Xu Cc: "\"Matthew Dillon\"" , sgk@troutmask.apl.washington.edu, wa1ter@hotmail.com, freebsd-current@FreeBSD.ORG Subject: Re: KSE / interrupt panic (patch) In-Reply-To: <002801c220a4$4637c2a0$ef01a8c0@davidwnt> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-current@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG thanks.. I was aware of he race but it looks as if I miscaldulated it somehow.. I will look at the source again and confirm your patch. As you say, msleep() is of the same form and has the same problems. On Mon, 1 Jul 2002, David Xu wrote: > Now let me describe where the race is: > > Thread A: | Thread B: > cv_timedwait() | softclock() > | cv_timedwait_end() > | > mtx_lock_spin(&sched_lock); | mtx_lock_spin(&sched_lock); /* suppose blocked!!! */ > ... | > callout_stop(&td->td_slpcallout) | > td->td_flags |= TDF_TIMEOUT; | > td->td_proc->p_stats->p_ru.ru_nivcsw++; | > // problem, does not set thread state to TDS_SLP!!! | > mi_switch(); | > // run at once again | > ... | > mtx_unlock_spin(&sched_lock); | mtx_lock_spin() returned !!! > thread is still running | if (td->td_flags & TDF_TIMEOUT) { > ... | td->td_flags &= ~TDF_TIMEOUT; > some place call mi_switch() and now on runqueue | setrunqueue(td); // crash > | } > > ---------------------------------------------------------- > here is the patch: > --- /sys/kern/kern_condvar.c.old Mon Jul 1 09:06:01 2002 > +++ /sys/kern/kern_condvar.c Mon Jul 1 09:32:50 2002 > @@ -396,6 +396,7 @@ > * between msleep and endtsleep. > */ > td->td_flags |= TDF_TIMEOUT; > + td->td_state = TDS_SLP; > td->td_proc->p_stats->p_ru.ru_nivcsw++; > mi_switch(); > } > @@ -472,6 +473,7 @@ > * between msleep and endtsleep. > */ > td->td_flags |= TDF_TIMEOUT; > + td->td_state = TDS_SLP; > td->td_proc->p_stats->p_ru.ru_nivcsw++; > mi_switch(); > } > ------------------------------------------------------------------- > > bug is because cv_timedwait() detects timeout callout is running, > but it does not correctly wait callout to complete, so panic. > BTW, the bug seems also exists in msleep() and endtsleep(), please > fix it! > > -David Xu > > > --- David Xu wrote: > > setrunqueue() call can be simply removed from cv_timedwait_end(), because > > there > > is a race in softclock() and callout_stop(), when cv_timedwait_end() losts a > > race, it means that that thread is already running(wokenup by another > > thread), > > when you setrunqueue() it, of course it will panic. > > in cv_timedwait_end(), sentence "if (td->td_flags & TDF_TIMEOUT) {...}" > > is to check this race condition. > > > > -David Xu > > > > ----- Original Message ----- > > From: "Matthew Dillon" > > To: "Julian Elischer" > > Cc: "Steve Kargl" ; "walt" > > ; > > Sent: Monday, July 01, 2002 4:43 AM > > Subject: KSE / interrupt panic > > > > > > > Got another one. Different panic, same place. > > > > > > panic: setrunqueue: bad thread state > > > cpuid = 0; lapic.id = 01000000 > > > Debugger("panic") > > > Stopped at Debugger+0x46: xchgl %ebx,in_Debugger.0 > > > db> trace > > > Debugger(c02ec2ba) at Debugger+0x46 > > > panic(c02ec8a9,c6461d80,c6461d80,c6461d80,c01afa30) at panic+0xd6 > > > setrunqueue(c6461d80) at setrunqueue+0x1dd > > > cv_timedwait_end(c6461d80) at cv_timedwait_end+0x36 > > > softclock(0) at softclock+0x159 > > > ithread_loop(c229c700,df3eed48,c22aec00,c01b9c6c,0) at ithread_loop+0x12c > > > fork_exit(c01b9c6c,c229c700,df3eed48) at fork_exit+0xa8 > > > fork_trampoline() at fork_trampoline+0x37 > > > db> gdb > > > ... > > > > > > #0 Debugger (msg=0xc02ec2ba "panic") > > > at /FreeBSD/FreeBSD-current/src/sys/i386/i386/db_interface.c:324 > > > #1 0xc01c878a in panic (fmt=0xc02ec8a9 "setrunqueue: bad thread state") > > > at /FreeBSD/FreeBSD-current/src/sys/kern/kern_shutdown.c:482 > > > #2 0xc01cc6cd in setrunqueue (td=0xc6461d80) > > > at /FreeBSD/FreeBSD-current/src/sys/kern/kern_switch.c:396 > > > #3 0xc01afa66 in cv_timedwait_end (arg=0xc6461d80) > > > at /FreeBSD/FreeBSD-current/src/sys/kern/kern_condvar.c:608 > > > #4 0xc01d22c9 in softclock (dummy=0x0) > > > at /FreeBSD/FreeBSD-current/src/sys/kern/kern_timeout.c:187 > > > #5 0xc01b9d98 in ithread_loop (arg=0xc229c700) > > > at /FreeBSD/FreeBSD-current/src/sys/kern/kern_intr.c:535 > > > #6 0xc01b923c in fork_exit (callout=0xc01b9c6c , > > > arg=0xc229c700, frame=0xdf3eed48) > > > at /FreeBSD/FreeBSD-current/src/sys/kern/kern_fork.c:863 > > > > > > I'm not sure why the panic was 'bad thread state' when gdb seems to > > > show it being stuck on 'unexpected ke present'. Maybe it was an > > > optimization and gdb is confused. The panic is definitely > > > 'bad thread state'. > > > > > > (gdb) print td->td_state > > > $2 = TDS_RUNQ > > > > > > setrunqueue() is being called on a thread which is already on the run > > > queue. > > > > > > -Matt > > > > > > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > > > with "unsubscribe freebsd-current" in the body of the message > > > > __________________________________________________ > > Do You Yahoo!? > > Yahoo! - Official partner of 2002 FIFA World Cup > > http://fifaworldcup.yahoo.com > > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-current" in the body of the message