Date: Thu, 29 Sep 2011 23:03:03 +0800 From: Adrian Chadd <adrian@freebsd.org> To: Andrew Duane <aduane@juniper.net> Cc: "freebsd-mips@freebsd.org" <freebsd-mips@freebsd.org> Subject: Re: eventtimer issue on mips: temporary workaround Message-ID: <CAJ-Vmok9X2XFbq=ZOee%2BXzuf6-=RYze%2BroM19HSQZYabm95nEw@mail.gmail.com> In-Reply-To: <AC6674AB7BC78549BB231821ABF7A9AEB8086A4A84@EMBX01-WF.jnpr.net> References: <CAJ-Vmo=qONOffCTgusWtbwuo43zKYyXDqqu5YEaL-MDQSbt-mQ@mail.gmail.com> <CAJ-Vmo=i6-3PNTPbP5xCftNU0w1OmMhZSysgaSRzDqgwLU6prQ@mail.gmail.com> <CA%2B7sy7DpEEhZ7WGoT-p9FCgvGBAeBHnyGVXmcUtHs%2BTt6tsTng@mail.gmail.com> <AC6674AB7BC78549BB231821ABF7A9AEB8086A4A84@EMBX01-WF.jnpr.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On 29 September 2011 22:12, Andrew Duane <aduane@juniper.net> wrote:
> I've been running JC's patch on my Octeon blade for a while, and it does =
fix some serious problems. It's been very stable.
>
> As to why "wait" wouldn't just return if an interrupt is asserted, I don'=
t know. The MIPS manual is (deliberately?) vague on these details, but any =
software engineer can explain why that's needed to write correct software. =
Maybe the silicon engineers didn't listen?
That's not the problem. I'm sorry if this isn't news to you - I've had
to explain what's going on to a few people, so I figure I may as well
just brain dump it one last time. :)
The problem is this:
* in the past, cpu_idle() would just call wait;. There'd be no
interrupts asserted and no critical section.
* any interrupt - clock (say, 1000hz) or other device interrupt would
interrupt that.
* if an interrupt happened -just- before wait, it'd force that
particular interrupt handler to be run. That may add something to the
run queue, so you don't want to have an interrupt schedule a task
(netisr, or a fast interrupt handler schedule a taskqueue, etc, etc)
* .. then you'd still have wait called. There's now a task on the
runqueue, but since you're now in wait; you won't break out until the
next interrupt occurs.
So the i386/amd64 code does this:
intr_disable();
if (sched_runnable()) {
intr_enable();
} else {
[atomic enable interrupts and wait]
}
So this way:
* if something schedules an interrupt _just before_ the wait call in
the idle thread, it'd be deferred due to intr_disable, then the atomic
sti;hlt would reenable interrupts and enter halt. The pending
interrupt would then immediately break that halt.
* If something didn't schedule an interrupt _just before_ the wait
call, then sti;hlt would enter wait, and the next interrupt would
trigger it.
The question here is what happens if there's no "atomic enable
interrupts and wait". An interrupt would come in between the "enable
interrupt" and "wait" call. That flips over to the interrupt handler
(netisr, fast, etc.) That may schedule a task on the run queue. Then
it flips back to the idle task and the idle task runs halt/wait/etc.
That scheduled task doesn't get run until the next interrupt breaks
the wait instruction.
If there's just a halt/wait call, with none of the above interrupt
disable/enable, then you still have the race - some interrupt may come
in between the last point where the scheduler does its thing (ie,
schedules something other than the idle task) and the wait/halt
instruction.
So with preemption, my understanding is the idle task can be
preempted. So as long as the interrupt occurs before the wait/halt
instruction, the scheduler could preempt the low priority idle task
with the higher priority "any other" task, and (i -think-) the race
condition becomes less of a problem.
Now this is made worse by the eventtimer changes. Because now, there's
a critical section being maintained across the wait call. So for
i386/amd64:
* interrupt occurs first, changes the runqueue, and if you don't have
preemption enabled it will just push something onto the runqueue.
* then in cpu_idle(), it calls critical_enter()
* .. then it does the eventtimer stuff
* then it does the above intr disable / sched_running dance - finds
something on the runqueue, and skips running wait
* .. then it eventually calls critical_exit(), which can be a point
where preemption occurs.
* .. and if no preemption, it exits cpu_idle() and runs the next task
All nice and fine. But then, this can happen:
* in cpu_idle(), it calls critical_enter()
* .. then you get an interrupt, which pushes something onto the runqueue
* then it runs the eventtimer stuff
* then it does the intr disable / sched running dance - finds
something on the runqueue, skips running wait.
* .. calls critical_exit(), where preemption occurs.
* .. and if no preemption, it exits cpu_idle() and runs the next task
Again, all nice and fine. but then with mips, this happens:
* in cpu_idle(), it calls critical_enter()
* .. interrupt - runqueue changes
* .. then it calls wait - which means that your runqueue won't be
handled until the next interrupt comes along, breaking the wait call
So if you add a non-atomic "enable interrupts and wait":
* cpu_idle() - calls critical_enter()
* disable interrupts for schedule check
* interrupt occurs - but they're disabled, so nothing happens yet
* nothing on runqueue, so it goes to call "enable interrupts, wait"
* interrupt happens - pushes something onto the runqueue
* then wait call happens - and won't wake up until the next interrupt.
Does this all make sense?
Adrian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-Vmok9X2XFbq=ZOee%2BXzuf6-=RYze%2BroM19HSQZYabm95nEw>