From owner-freebsd-mips@FreeBSD.ORG Tue Sep 27 03:36:27 2011 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4C261106564A; Tue, 27 Sep 2011 03:36:27 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id C160A8FC14; Tue, 27 Sep 2011 03:36:26 +0000 (UTC) Received: by ywp17 with SMTP id 17so6241606ywp.13 for ; Mon, 26 Sep 2011 20:36:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=e3vu+KCQnPyddaZ7enWH5N+Bt+1gajRj+42qCY/Lpm4=; b=aGaMZ8vExCqJOE1DJyYPem3QJA9nCekOPU5Oko34krvb8wS2Zr/O4S5+vIhrqkDv/+ dq2kVtL1CHELxBYnNWgspHZVrwX7B+FQy0+UfyRanD6bHdFzUyu++R5mGXRqqGh3pyA0 Cck1X/alQ4mp7exdobSdezkdngGPX3JUvxVi0= MIME-Version: 1.0 Received: by 10.236.176.65 with SMTP id a41mr44945415yhm.72.1317094586115; Mon, 26 Sep 2011 20:36:26 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.236.111.42 with HTTP; Mon, 26 Sep 2011 20:36:26 -0700 (PDT) In-Reply-To: References: <201109261053.30410.jhb@freebsd.org> <201109261305.57602.jhb@freebsd.org> Date: Tue, 27 Sep 2011 11:36:26 +0800 X-Google-Sender-Auth: MQwqO8SZOYGGyXDIcSp_xVj1Kuo Message-ID: From: Adrian Chadd To: John Baldwin , attilio@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-current@freebsd.org, freebsd-mips@freebsd.org Subject: Re: ath / 802.11n performance issues and timer code X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Sep 2011 03:36:27 -0000 .. and as a follow up (and cc'ing attillo and freebsd-mips, in case it's relevant to other platforms and there's a MIPS specific thing to fix): * 2128: mi_switch to idle * 2129: kern_clocksource.c:762 - ie, cpu_idleclock() has been called * 2130: the ath interrupt comes in * 2134: it's skipped for now as the idle thread is in a critical section * 2136: kern_clocksource.c:266 - ie, getnextcpuevent(), inside cpu_idleclock(). What I bet is happening is this race between the critical section + cpu_idleclock() and the ath0 interrupt: * idle gets scheduled * critical_enter() is called in the mips cpu_idle() routine * the ath interrupt comes in here and gets handled, but since we're in a critical section, it won't preempt things * the cpu_idleclock() code completes without releasing the preemption, and the only thing that wakes up from that wait is the next interrupt (clock, arge0, etc.) Adrian From owner-freebsd-mips@FreeBSD.ORG Tue Sep 27 13:51:09 2011 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 99EBF106566B; Tue, 27 Sep 2011 13:51:09 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 6F6B18FC16; Tue, 27 Sep 2011 13:51:09 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 0CABB46B58; Tue, 27 Sep 2011 09:51:09 -0400 (EDT) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 91B2B8A02F; Tue, 27 Sep 2011 09:51:08 -0400 (EDT) From: John Baldwin To: Adrian Chadd Date: Tue, 27 Sep 2011 09:51:07 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110617; KDE/4.5.5; amd64; ; ) References: In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201109270951.07839.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Tue, 27 Sep 2011 09:51:08 -0400 (EDT) Cc: attilio@freebsd.org, freebsd-current@freebsd.org, freebsd-mips@freebsd.org Subject: Re: ath / 802.11n performance issues and timer code X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Sep 2011 13:51:09 -0000 On Monday, September 26, 2011 11:36:26 pm Adrian Chadd wrote: > .. and as a follow up (and cc'ing attillo and freebsd-mips, in case > it's relevant to other platforms and there's a MIPS specific thing to > fix): > > * 2128: mi_switch to idle > * 2129: kern_clocksource.c:762 - ie, cpu_idleclock() has been called > * 2130: the ath interrupt comes in > * 2134: it's skipped for now as the idle thread is in a critical section > * 2136: kern_clocksource.c:266 - ie, getnextcpuevent(), inside cpu_idleclock(). > > What I bet is happening is this race between the critical section + > cpu_idleclock() and the ath0 interrupt: > > * idle gets scheduled > * critical_enter() is called in the mips cpu_idle() routine > * the ath interrupt comes in here and gets handled, but since we're in > a critical section, it won't preempt things > * the cpu_idleclock() code completes without releasing the preemption, > and the only thing that wakes up from that wait is the next interrupt > (clock, arge0, etc.) I think this is a mips-specific bug, though it may be well to audit all the cpu_idle() implementations. On x86 the idle hooks all check sched_runnable() with interrupts disabled and then atomically re-enable interrupts and sleep only if that is false, e.g.: static void cpu_idle_hlt(int busy) { int *state; state = (int *)PCPU_PTR(monitorbuf); *state = STATE_SLEEPING; /* * We must absolutely guarentee that hlt is the next instruction * after sti or we introduce a timing window. */ disable_intr(); if (sched_runnable()) enable_intr(); else __asm __volatile("sti; hlt"); *state = STATE_RUNNING; } I don't know if it is possible to do the same thing with the mips "wait" instruction. -- John Baldwin From owner-freebsd-mips@FreeBSD.ORG Tue Sep 27 14:24:06 2011 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BE443106566B; Tue, 27 Sep 2011 14:24:06 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 2AC7A8FC17; Tue, 27 Sep 2011 14:24:05 +0000 (UTC) Received: by wyj26 with SMTP id 26so5613054wyj.13 for ; Tue, 27 Sep 2011 07:24:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=7us3orChbTInro0z5oNt9JjWWKpmVqIxtyHSD0A9vKQ=; b=UlsnGWT9AweJaDH1FU2vcgxyx0jV8i18496ngvTHOmy++u3C2ZHJBecJbMVQd4Vm0C iWJkT8nH7deOK6k90oaHJ9GByqMpcfOP8wpON3XcB3wu9VveAfu3cIRoK2oMeQvzrThX fyI1lgFlFPwOspYXDPSEwPIG4fZLx3hg6z2rI= MIME-Version: 1.0 Received: by 10.216.159.133 with SMTP id s5mr8989170wek.27.1317131873667; Tue, 27 Sep 2011 06:57:53 -0700 (PDT) Sender: asmrookie@gmail.com Received: by 10.216.179.78 with HTTP; Tue, 27 Sep 2011 06:57:53 -0700 (PDT) In-Reply-To: <201109270951.07839.jhb@freebsd.org> References: <201109270951.07839.jhb@freebsd.org> Date: Tue, 27 Sep 2011 15:57:53 +0200 X-Google-Sender-Auth: tejor1ZkVAcCdRA-1xoK-OaPEc0 Message-ID: From: Attilio Rao To: John Baldwin Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-current@freebsd.org, freebsd-mips@freebsd.org Subject: Re: ath / 802.11n performance issues and timer code X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Sep 2011 14:24:06 -0000 2011/9/27 John Baldwin : > On Monday, September 26, 2011 11:36:26 pm Adrian Chadd wrote: >> .. and as a follow up (and cc'ing attillo and freebsd-mips, in case >> it's relevant to other platforms and there's a MIPS specific thing to >> fix): >> >> * 2128: mi_switch to idle >> * 2129: kern_clocksource.c:762 - ie, cpu_idleclock() has been called >> * 2130: the ath interrupt comes in >> * 2134: it's skipped for now as the idle thread is in a critical section >> * 2136: kern_clocksource.c:266 - ie, getnextcpuevent(), inside > cpu_idleclock(). >> >> What I bet is happening is this race between the critical section + >> cpu_idleclock() and the ath0 interrupt: >> >> * idle gets scheduled >> * critical_enter() is called in the mips cpu_idle() routine >> * the ath interrupt comes in here and gets handled, but since we're in >> a critical section, it won't preempt things >> * the cpu_idleclock() code completes without releasing the preemption, >> and the only thing that wakes up from that wait is the next interrupt >> (clock, arge0, etc.) > > I think this is a mips-specific bug, though it may be well to audit all t= he > cpu_idle() implementations. =C2=A0On x86 the idle hooks all check sched_r= unnable() > with interrupts disabled and then atomically re-enable interrupts and sle= ep > only if that is false, e.g.: > > static void > cpu_idle_hlt(int busy) > { > =C2=A0 =C2=A0 =C2=A0 =C2=A0int *state; > > =C2=A0 =C2=A0 =C2=A0 =C2=A0state =3D (int *)PCPU_PTR(monitorbuf); > =C2=A0 =C2=A0 =C2=A0 =C2=A0*state =3D STATE_SLEEPING; > =C2=A0 =C2=A0 =C2=A0 =C2=A0/* > =C2=A0 =C2=A0 =C2=A0 =C2=A0 * We must absolutely guarentee that hlt is th= e next instruction > =C2=A0 =C2=A0 =C2=A0 =C2=A0 * after sti or we introduce a timing window. > =C2=A0 =C2=A0 =C2=A0 =C2=A0 */ > =C2=A0 =C2=A0 =C2=A0 =C2=A0disable_intr(); > =C2=A0 =C2=A0 =C2=A0 =C2=A0if (sched_runnable()) > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0enable_intr(); > =C2=A0 =C2=A0 =C2=A0 =C2=A0else > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0__asm __volatile("= sti; hlt"); > =C2=A0 =C2=A0 =C2=A0 =C2=A0*state =3D STATE_RUNNING; > } > > I don't know if it is possible to do the same thing with the mips "wait" > instruction. After thinking about it I think this check is unnecessary. sti, infact, doesn't enable interrupts before hlt (it just sets IF=3D1) but interrupts can fire only after hlt, thus I don't think a preemption or interrupts firing there can be possible. This patch: http://www.freebsd.org/~attilio/stihlt.patch removes the check and also replaces simple 'hlt' instruction insertion in C code with the already defined halt(). I still have to go through Adrian's e-mails so I'm not sure if the logic you post is going to help him or not, but this is my thinking on the x86 implementation (specifically). Comments? Attilio --=20 Peace can only be achieved by understanding - A. Einstein From owner-freebsd-mips@FreeBSD.ORG Wed Sep 28 14:44:18 2011 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 134381065673 for ; Wed, 28 Sep 2011 14:44:18 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-yx0-f182.google.com (mail-yx0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id CB0818FC0C for ; Wed, 28 Sep 2011 14:44:17 +0000 (UTC) Received: by yxk36 with SMTP id 36so8125130yxk.13 for ; Wed, 28 Sep 2011 07:44:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:content-type; bh=i7QYe2eM/BihUFteKBHIanadOO/n7X2Ijxs+W2OENfM=; b=ddiKiKsjEJxd+GmNKYhvkqvEpB+Esw7hk/5iSSLV5LaY15ofupFGUQ13JHwaetuU+h IYeW+d5OJb+ioCpnAmK2OQM1Vn9sSp9FX6/LQEmPARaWQLR3J6gUtQ/hbPZp9eSDbpYn bgTvEkLF0PLV+zszBrXVF3TGjtp7WdGv+lpLU= MIME-Version: 1.0 Received: by 10.236.75.227 with SMTP id z63mr56186904yhd.55.1317221057024; Wed, 28 Sep 2011 07:44:17 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.236.111.42 with HTTP; Wed, 28 Sep 2011 07:44:13 -0700 (PDT) Date: Wed, 28 Sep 2011 22:44:13 +0800 X-Google-Sender-Auth: nnUkKRjr9nYokpVVIqRU-M5ITEQ Message-ID: From: Adrian Chadd To: freebsd-mips@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: eventtimer issue on mips: temporary workaround X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Sep 2011 14:44:18 -0000 Hi all, I've found that there's an issue with how mav@ shoehorned in event timer handling to mips. Basically, if a non-fast interrupt comes in after the critical_enter() call in cpu_idle() but before the wait instruction, it won't interrupt wait and it won't be serviced until the next interrupt ends the wait. For ath, since it's a netisr, this means the netisr won't be run until the next interrupt fires. Here's my temporary, not-quite-correct workaround. ray@ pointed out this from Linux: http://lxr.free-electrons.com/source/arch/mips/kernel/cpu-probe.c?a=sh .. which indicates that they're calling wait with interrupts masked. This apparently doesn't stop it breaking wait, it merely stops it from flipping to the interrupt handler. There are also a number of MIPS platform specific workarounds to implement idle. I'd really appreciate some feedback on this and hopefully a correct solution. :) Adrian Index: mips/machdep.c =================================================================== --- mips/machdep.c (revision 225610) +++ mips/machdep.c (working copy) @@ -497,7 +497,16 @@ critical_enter(); cpu_idleclock(); } - __asm __volatile ("wait"); + + intr_disable(); + if (sched_runnable()) { + intr_enable(); + } else { + /* XXX this isn't atomic! */ + intr_enable(); + __asm __volatile ("wait"); + } + if (!busy) { cpu_activeclock(); critical_exit(); From owner-freebsd-mips@FreeBSD.ORG Wed Sep 28 16:36:37 2011 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E44C61065673 for ; Wed, 28 Sep 2011 16:36:37 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-yx0-f182.google.com (mail-yx0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id A4B948FC14 for ; Wed, 28 Sep 2011 16:36:37 +0000 (UTC) Received: by yxk36 with SMTP id 36so8270715yxk.13 for ; Wed, 28 Sep 2011 09:36:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; bh=lI27WY5bRLzJNejjHz0kIdNGpfrY6pqQtQEscVBcusI=; b=iTRA6iflWo+dBxy1SdlFeFbbM5MrQ4MF+m0UQJTm5X04vy//KxNKSr5c5aRKsOn0Ov f94/6tHJYaEDZTGMs4r4+pX48T2pDV0cvtVnFy1X3k7ElHpn3cdgrPZNwlUYYj51hJv4 zZ6UZePcc2emw/ONyKmJSjHbD1sTc1PgvT9bM= MIME-Version: 1.0 Received: by 10.236.201.165 with SMTP id b25mr6928511yho.72.1317227796766; Wed, 28 Sep 2011 09:36:36 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.236.111.42 with HTTP; Wed, 28 Sep 2011 09:36:36 -0700 (PDT) In-Reply-To: References: Date: Thu, 29 Sep 2011 00:36:36 +0800 X-Google-Sender-Auth: 4NtWMGHS-lzEwGXfKgEEQCjnBcY Message-ID: From: Adrian Chadd To: freebsd-mips@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: Re: eventtimer issue on mips: temporary workaround X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Sep 2011 16:36:38 -0000 .. so, the patch is totallyw rong. intr_disable() needs to be moved before critical_enter() or it doesn't achieve anything. But the race is still there, between intr_enable() and "wait". The only way to eliminate this race is to completely eliminate all the code in cpu_idle(). Would someone clued in the implementation of wait please step up and help? :) Adrian From owner-freebsd-mips@FreeBSD.ORG Thu Sep 29 03:51:04 2011 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 220DF106564A for ; Thu, 29 Sep 2011 03:51:04 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id C981D8FC0A for ; Thu, 29 Sep 2011 03:51:03 +0000 (UTC) Received: by ywp17 with SMTP id 17so208922ywp.13 for ; Wed, 28 Sep 2011 20:51:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; bh=y6jJG4ICNT6F4uH85VRJtNbdXLV2vwy8Qd4u4F+1+iI=; b=N1A5mJLUYz4B3tCCiq/FBPyUvzxPwjSnlAPBmjtpMyAOZQO4Y3NHElBt3M+t5bKx+D i+kB4W3FTC1kO/8qUHyVJofSwzmpXFN+RX/AXTYOwR5FiVe5DrI5tNy4nsA96/HtlSd/ jrQu2LJUiRwVN1S9DJQi6yd8isqd7NwJ5Z0wI= MIME-Version: 1.0 Received: by 10.236.201.165 with SMTP id b25mr10746492yho.72.1317268263079; Wed, 28 Sep 2011 20:51:03 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.236.111.42 with HTTP; Wed, 28 Sep 2011 20:51:03 -0700 (PDT) In-Reply-To: References: Date: Thu, 29 Sep 2011 11:51:03 +0800 X-Google-Sender-Auth: 02KHpheHDWYNLPOnbZWfx3eNr28 Message-ID: From: Adrian Chadd To: freebsd-mips@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: Re: eventtimer issue on mips: temporary workaround X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Sep 2011 03:51:04 -0000 .. and since I'd rather this not be ignored, I'm going to #if 0 out the contents of cpu_idle() in say, 4 days. I'll add a comment to -HEAD explaining why; then I'll request this be MFCed for 9.0. It's unfortunate, but I'd rather see MIPS working in 9.0 over trying to save power/heat. Adrian From owner-freebsd-mips@FreeBSD.ORG Thu Sep 29 06:21:22 2011 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2A1E9106566B; Thu, 29 Sep 2011 06:21:22 +0000 (UTC) (envelope-from c.jayachandran@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 89B4B8FC20; Thu, 29 Sep 2011 06:21:21 +0000 (UTC) Received: by wwe3 with SMTP id 3so408576wwe.31 for ; Wed, 28 Sep 2011 23:21:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=+QdIwXE5tlMEE+fJxbxZ2KFBvpm7FeorQUcsL7f13oM=; b=LP8mev/nxMmM1JDgm1wf0LXSfi37APlcYjMnJScT7NYn2nfL2zLtxtxMJBqVS4l9Mh yQhDSaYh6tlCo2iepc9s02ZAzjxtlxyak1mOGlyBb26mt9cTs0qOHkiqE1PAoPX2G3sw on8nH+3KGrEHFV6HJ5OBlS81hdoPKAefH2414= MIME-Version: 1.0 Received: by 10.216.131.225 with SMTP id m75mr882931wei.1.1317275963529; Wed, 28 Sep 2011 22:59:23 -0700 (PDT) Received: by 10.216.154.5 with HTTP; Wed, 28 Sep 2011 22:59:23 -0700 (PDT) In-Reply-To: References: Date: Thu, 29 Sep 2011 11:29:23 +0530 Message-ID: From: "Jayachandran C." To: Adrian Chadd Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-mips@freebsd.org Subject: Re: eventtimer issue on mips: temporary workaround X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Sep 2011 06:21:22 -0000 On Wed, Sep 28, 2011 at 8:14 PM, Adrian Chadd wrote: > Hi all, > > I've found that there's an issue with how mav@ shoehorned in event > timer handling to mips. > > Basically, if a non-fast interrupt comes in after the critical_enter() > call in cpu_idle() but before the wait instruction, it won't interrupt > wait and it won't be serviced until the next interrupt ends the wait. > > For ath, since it's a netisr, this means the netisr won't be run until > the next interrupt fires. > > Here's my temporary, not-quite-correct workaround. ray@ pointed out > this from Linux: > > http://lxr.free-electrons.com/source/arch/mips/kernel/cpu-probe.c?a=3Dsh > > .. which indicates that they're calling wait with interrupts masked. > This apparently doesn't stop it breaking wait, it merely stops it from > flipping to the interrupt handler. Linux has many versions of wait, there is the r4k_wait in assembly which keeps the interrupts enabled http://lxr.free-electrons.com/source/arch/mips/kernel/genex.S#L131 The MIPS architecture manual says it is implementation-dependent whether the non-enabled interrupt will make wait to continue. > There are also a number of MIPS platform specific workarounds to implemen= t idle. > > I'd really appreciate some feedback on this and hopefully a correct solut= ion. :) > > > > Adrian > > Index: mips/machdep.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- mips/machdep.c =A0 =A0 =A0(revision 225610) > +++ mips/machdep.c =A0 =A0 =A0(working copy) > @@ -497,7 +497,16 @@ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0critical_enter(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0cpu_idleclock(); > =A0 =A0 =A0 =A0} > - =A0 =A0 =A0 __asm __volatile ("wait"); > + > + =A0 =A0 =A0 intr_disable(); > + =A0 =A0 =A0 if (sched_runnable()) { > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 intr_enable(); > + =A0 =A0 =A0 } else { > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* XXX this isn't atomic! */ > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 intr_enable(); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 __asm __volatile ("wait"); > + =A0 =A0 =A0 } > + > =A0 =A0 =A0 =A0if (!busy) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0cpu_activeclock(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0critical_exit(); JC. From owner-freebsd-mips@FreeBSD.ORG Thu Sep 29 06:41:49 2011 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5E067106566C; Thu, 29 Sep 2011 06:41:49 +0000 (UTC) (envelope-from c.jayachandran@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id BCE948FC13; Thu, 29 Sep 2011 06:41:48 +0000 (UTC) Received: by wyj26 with SMTP id 26so99954wyj.13 for ; Wed, 28 Sep 2011 23:41:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=PUHJC2nZ1YecBNuXL7WcdITpUncNMuCqV308DYFCVOw=; b=mQaUqhvHXGLoYp1viwY/IbQhxGLj3dXCDibIIRSnAzfXsDvsOsd1nd8VegfdWCqiZb DkzU0KM8o/8OOvQAmP/7+LeM1YAdAe7LDw2QLZve8OYuIEJoPuLSygooSdcg0yZYpO+e OTherCqWZJsgKb8Ma9mKdf8SU0Z0ohG5sK90g= MIME-Version: 1.0 Received: by 10.216.220.131 with SMTP id o3mr9871811wep.11.1317278507517; Wed, 28 Sep 2011 23:41:47 -0700 (PDT) Received: by 10.216.154.5 with HTTP; Wed, 28 Sep 2011 23:41:47 -0700 (PDT) In-Reply-To: References: Date: Thu, 29 Sep 2011 12:11:47 +0530 Message-ID: From: "Jayachandran C." To: Adrian Chadd Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-mips@freebsd.org Subject: Re: eventtimer issue on mips: temporary workaround X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Sep 2011 06:41:49 -0000 On Wed, Sep 28, 2011 at 10:06 PM, Adrian Chadd wrote: > .. so, the patch is totallyw rong. > > intr_disable() needs to be moved before critical_enter() or it doesn't > achieve anything. I'm not able to figure out why... > But the race is still there, between intr_enable() and "wait". > The only way to eliminate this race is to completely eliminate all the > code in cpu_idle(). the amd implementation seems to be using the STI instruction to enable interrupts - but I'm not able to see how to avoid this race condition on platforms which does not have a similar instruction. > Would someone clued in the implementation of wait please step up and help? :) What if go back to the earlier version of cpu_idle which does not have critical_enter() and cpu_idleclock() for now, or does this also have issues? I had also seen issues on XLR which went away when I took out 'ET_FLAGS_ONESHOT' from the mips clock event timer . That is another possible workaround. JC. From owner-freebsd-mips@FreeBSD.ORG Thu Sep 29 07:19:51 2011 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 25D57106564A for ; Thu, 29 Sep 2011 07:19:51 +0000 (UTC) (envelope-from juli@clockworksquid.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id BA1768FC12 for ; Thu, 29 Sep 2011 07:19:50 +0000 (UTC) Received: by wyj26 with SMTP id 26so138550wyj.13 for ; Thu, 29 Sep 2011 00:19:49 -0700 (PDT) Received: by 10.227.179.76 with SMTP id bp12mr3306090wbb.82.1317278982062; Wed, 28 Sep 2011 23:49:42 -0700 (PDT) MIME-Version: 1.0 Sender: juli@clockworksquid.com Received: by 10.227.58.139 with HTTP; Wed, 28 Sep 2011 23:49:22 -0700 (PDT) In-Reply-To: References: From: Juli Mallett Date: Wed, 28 Sep 2011 23:49:22 -0700 X-Google-Sender-Auth: PocXnke5HSDmt0D2ydAJ8-tahRA Message-ID: To: "Jayachandran C." Content-Type: text/plain; charset=UTF-8 Cc: freebsd-mips@freebsd.org Subject: Re: eventtimer issue on mips: temporary workaround X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Sep 2011 07:19:51 -0000 On Wed, Sep 28, 2011 at 23:41, Jayachandran C. wrote: > the amd implementation seems to be using the STI instruction to enable > interrupts - but I'm not able to see how to avoid this race condition > on platforms which does not have a similar instruction. If I'm understanding right, that's much of the thrust of the problem. I mean, we can check whether a non-masked interrupt was asserted before doing the wait, but there's still a window between that and the wait. It sounds like we need a conditional wait instruction, which is to say it sounds like the code does need a rethink for MIPS (although why "wait" wouldn't return immediately if there's a non-masked interrupt asserted is beyond me.) But I'm still somewhat confused by the original post, so may be missing the mark on that. From owner-freebsd-mips@FreeBSD.ORG Thu Sep 29 07:21:44 2011 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 399FA106564A; Thu, 29 Sep 2011 07:21:44 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id A64A78FC13; Thu, 29 Sep 2011 07:21:43 +0000 (UTC) Received: by gyf2 with SMTP id 2so340637gyf.13 for ; Thu, 29 Sep 2011 00:21:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=BGS3FddUWM0q6zRJzlQGumQuFipwFRQqzwyrjJ6PROQ=; b=cs0pbkbCxW1z06eXKfctuYE0JbzfbYlYFutPIDPeYTVwFTnQiAbEh41xMb/eeh9K/X 1Qhn2Vi3z9CuvEtdcEAamH7mYiXhkrN9yo7ymoVX2xLACh1D97DRaul9pYK1Sh9BjbN3 WnKVfCnSJVw+7CGTGrismRqIkLPUHLEExNXhk= MIME-Version: 1.0 Received: by 10.236.129.165 with SMTP id h25mr4755366yhi.38.1317280903092; Thu, 29 Sep 2011 00:21:43 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.236.111.42 with HTTP; Thu, 29 Sep 2011 00:21:43 -0700 (PDT) In-Reply-To: References: Date: Thu, 29 Sep 2011 15:21:43 +0800 X-Google-Sender-Auth: ZlN3R6mAujq2dyL_vmRdZ5fp1lw Message-ID: From: Adrian Chadd To: Juli Mallett Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-mips@freebsd.org Subject: Re: eventtimer issue on mips: temporary workaround X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Sep 2011 07:21:44 -0000 On 29 September 2011 14:49, Juli Mallett wrote: > If I'm understanding right, that's much of the thrust of the problem. > I mean, we can check whether a non-masked interrupt was asserted > before doing the wait, but there's still a window between that and the > wait. =A0It sounds like we need a conditional wait instruction, which is Yes. That's absolutely right. > to say it sounds like the code does need a rethink for MIPS (although > why "wait" wouldn't return immediately if there's a non-masked > interrupt asserted is beyond me.) =A0But I'm still somewhat confused by > the original post, so may be missing the mark on that. No, I think you have it. Adrian From owner-freebsd-mips@FreeBSD.ORG Thu Sep 29 14:27:59 2011 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 994C0106566B; Thu, 29 Sep 2011 14:27:59 +0000 (UTC) (envelope-from aduane@juniper.net) Received: from exprod7og125.obsmtp.com (exprod7og125.obsmtp.com [64.18.2.28]) by mx1.freebsd.org (Postfix) with ESMTP id 180E68FC0A; Thu, 29 Sep 2011 14:27:58 +0000 (UTC) Received: from P-EMHUB03-HQ.jnpr.net ([66.129.224.36]) (using TLSv1) by exprod7ob125.postini.com ([64.18.6.12]) with SMTP ID DSNKToSAbrQ/NpMF6PX79F7Z5m2D1vKo17L/@postini.com; Thu, 29 Sep 2011 07:27:59 PDT Received: from p-emfe02-wf.jnpr.net (172.28.145.25) by P-EMHUB03-HQ.jnpr.net (172.24.192.37) with Microsoft SMTP Server (TLS) id 8.3.83.0; Thu, 29 Sep 2011 07:12:48 -0700 Received: from EMBX01-WF.jnpr.net ([fe80::8002:d3e7:4146:af5f]) by p-emfe02-wf.jnpr.net ([fe80::c126:c633:d2dc:8090%11]) with mapi; Thu, 29 Sep 2011 10:12:47 -0400 From: Andrew Duane To: Jayachandran C. , Adrian Chadd Date: Thu, 29 Sep 2011 10:12:46 -0400 Thread-Topic: eventtimer issue on mips: temporary workaround Thread-Index: Acx+cuSpRZVi5xNTS/apx6gBZwXIvAAPomSQ Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "freebsd-mips@freebsd.org" Subject: RE: eventtimer issue on mips: temporary workaround X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Sep 2011 14:27:59 -0000 I've been running JC's patch on my Octeon blade for a while, and it does fi= x some serious problems. It's been very stable. As to why "wait" wouldn't just return if an interrupt is asserted, I don't = know. The MIPS manual is (deliberately?) vague on these details, but any so= ftware engineer can explain why that's needed to write correct software. Ma= ybe the silicon engineers didn't listen? =A0................................... Andrew Duane Juniper Networks o=A0=A0=A0+1 978 589 0551 m=A0 +1 603-770-7088 aduane@juniper.net =A0 > -----Original Message----- > From: owner-freebsd-mips@freebsd.org [mailto:owner-freebsd- > mips@freebsd.org] On Behalf Of Jayachandran C. > Sent: Wednesday, September 28, 2011 11:42 PM > To: Adrian Chadd > Cc: freebsd-mips@freebsd.org > Subject: Re: eventtimer issue on mips: temporary workaround >=20 > On Wed, Sep 28, 2011 at 10:06 PM, Adrian Chadd > wrote: > > .. so, the patch is totallyw rong. > > > > intr_disable() needs to be moved before critical_enter() or it > doesn't > > achieve anything. >=20 > I'm not able to figure out why... >=20 > > But the race is still there, between intr_enable() and "wait". > > The only way to eliminate this race is to completely eliminate all > the > > code in cpu_idle(). >=20 > the amd implementation seems to be using the STI instruction to enable > interrupts - but I'm not able to see how to avoid this race condition > on platforms which does not have a similar instruction. >=20 > > Would someone clued in the implementation of wait please step up and > help? :) >=20 > What if go back to the earlier version of cpu_idle which does not have > critical_enter() and cpu_idleclock() for now, or does this also have > issues? >=20 > I had also seen issues on XLR which went away when I took out > 'ET_FLAGS_ONESHOT' from the mips clock event timer . That is another > possible workaround. >=20 > JC. > _______________________________________________ > freebsd-mips@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-mips > To unsubscribe, send any mail to "freebsd-mips-unsubscribe@freebsd.org" From owner-freebsd-mips@FreeBSD.ORG Thu Sep 29 15:03:04 2011 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CB407106566B for ; Thu, 29 Sep 2011 15:03:04 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-yi0-f54.google.com (mail-yi0-f54.google.com [209.85.218.54]) by mx1.freebsd.org (Postfix) with ESMTP id 8467B8FC08 for ; Thu, 29 Sep 2011 15:03:04 +0000 (UTC) Received: by yia13 with SMTP id 13so788411yia.13 for ; Thu, 29 Sep 2011 08:03:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=xIcYl5JAdkIBPBqgF3mFc+TGiqIjikAN+NBo0bCyVzk=; b=ggWsVE09W3OWr2+9duitIogEaE+9BJg1EqFbrT/78SXQyXSG+qD0DSQToNGVvh9nX9 J51DQDNx5A/FE/tUaqVlosFuC/0VSux5Jn5/ONgvUGIPuVouMKcFxTDbL97cw/HtPPW2 l/2mdvVNEJu1paJHlZQmKqLIgyy+8TY3+qmOs= MIME-Version: 1.0 Received: by 10.236.79.72 with SMTP id h48mr66740150yhe.4.1317308583281; Thu, 29 Sep 2011 08:03:03 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.236.111.42 with HTTP; Thu, 29 Sep 2011 08:03:03 -0700 (PDT) In-Reply-To: References: Date: Thu, 29 Sep 2011 23:03:03 +0800 X-Google-Sender-Auth: hYc5BhL4xpP6dTT5WcP1CZsNDGw Message-ID: From: Adrian Chadd To: Andrew Duane Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-mips@freebsd.org" Subject: Re: eventtimer issue on mips: temporary workaround X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Sep 2011 15:03:04 -0000 On 29 September 2011 22:12, Andrew Duane wrote: > I've been running JC's patch on my Octeon blade for a while, and it does = fix some serious problems. It's been very stable. > > As to why "wait" wouldn't just return if an interrupt is asserted, I don'= t know. The MIPS manual is (deliberately?) vague on these details, but any = software engineer can explain why that's needed to write correct software. = Maybe the silicon engineers didn't listen? That's not the problem. I'm sorry if this isn't news to you - I've had to explain what's going on to a few people, so I figure I may as well just brain dump it one last time. :) The problem is this: * in the past, cpu_idle() would just call wait;. There'd be no interrupts asserted and no critical section. * any interrupt - clock (say, 1000hz) or other device interrupt would interrupt that. * if an interrupt happened -just- before wait, it'd force that particular interrupt handler to be run. That may add something to the run queue, so you don't want to have an interrupt schedule a task (netisr, or a fast interrupt handler schedule a taskqueue, etc, etc) * .. then you'd still have wait called. There's now a task on the runqueue, but since you're now in wait; you won't break out until the next interrupt occurs. So the i386/amd64 code does this: intr_disable(); if (sched_runnable()) { intr_enable(); } else { [atomic enable interrupts and wait] } So this way: * if something schedules an interrupt _just before_ the wait call in the idle thread, it'd be deferred due to intr_disable, then the atomic sti;hlt would reenable interrupts and enter halt. The pending interrupt would then immediately break that halt. * If something didn't schedule an interrupt _just before_ the wait call, then sti;hlt would enter wait, and the next interrupt would trigger it. The question here is what happens if there's no "atomic enable interrupts and wait". An interrupt would come in between the "enable interrupt" and "wait" call. That flips over to the interrupt handler (netisr, fast, etc.) That may schedule a task on the run queue. Then it flips back to the idle task and the idle task runs halt/wait/etc. That scheduled task doesn't get run until the next interrupt breaks the wait instruction. If there's just a halt/wait call, with none of the above interrupt disable/enable, then you still have the race - some interrupt may come in between the last point where the scheduler does its thing (ie, schedules something other than the idle task) and the wait/halt instruction. So with preemption, my understanding is the idle task can be preempted. So as long as the interrupt occurs before the wait/halt instruction, the scheduler could preempt the low priority idle task with the higher priority "any other" task, and (i -think-) the race condition becomes less of a problem. Now this is made worse by the eventtimer changes. Because now, there's a critical section being maintained across the wait call. So for i386/amd64: * interrupt occurs first, changes the runqueue, and if you don't have preemption enabled it will just push something onto the runqueue. * then in cpu_idle(), it calls critical_enter() * .. then it does the eventtimer stuff * then it does the above intr disable / sched_running dance - finds something on the runqueue, and skips running wait * .. then it eventually calls critical_exit(), which can be a point where preemption occurs. * .. and if no preemption, it exits cpu_idle() and runs the next task All nice and fine. But then, this can happen: * in cpu_idle(), it calls critical_enter() * .. then you get an interrupt, which pushes something onto the runqueue * then it runs the eventtimer stuff * then it does the intr disable / sched running dance - finds something on the runqueue, skips running wait. * .. calls critical_exit(), where preemption occurs. * .. and if no preemption, it exits cpu_idle() and runs the next task Again, all nice and fine. but then with mips, this happens: * in cpu_idle(), it calls critical_enter() * .. interrupt - runqueue changes * .. then it calls wait - which means that your runqueue won't be handled until the next interrupt comes along, breaking the wait call So if you add a non-atomic "enable interrupts and wait": * cpu_idle() - calls critical_enter() * disable interrupts for schedule check * interrupt occurs - but they're disabled, so nothing happens yet * nothing on runqueue, so it goes to call "enable interrupts, wait" * interrupt happens - pushes something onto the runqueue * then wait call happens - and won't wake up until the next interrupt. Does this all make sense? Adrian From owner-freebsd-mips@FreeBSD.ORG Sat Oct 1 10:31:32 2011 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 687BC1065672; Sat, 1 Oct 2011 10:31:32 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 00C768FC15; Sat, 1 Oct 2011 10:31:31 +0000 (UTC) Received: by vws11 with SMTP id 11so2675436vws.13 for ; Sat, 01 Oct 2011 03:31:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=VbSz/w81QGNd8MG1XbnpwnTswcINv1J7uBJsu7YNrhc=; b=l8RIeuVbrp+eoBV+7ZEvvZqX+UXcLi52r6bX6uN6g7UCNGuxxR+OqGKwMtvwQvoN8D 6gUAzJvCGv1Tp3t/HWikFpqmzxIIHpe5rPqdrKpWIfNl3BBF4qQyKfj+om543j/uJ171 85Z3bCEWXICznpe5TeDqoyJskXhyfiwHPy4bk= MIME-Version: 1.0 Received: by 10.52.68.203 with SMTP id y11mr2961731vdt.462.1317465091197; Sat, 01 Oct 2011 03:31:31 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.52.161.138 with HTTP; Sat, 1 Oct 2011 03:31:31 -0700 (PDT) In-Reply-To: References: <201110010556.p915uQH6003016@svn.freebsd.org> Date: Sat, 1 Oct 2011 18:31:31 +0800 X-Google-Sender-Auth: T1MhMM8VbmAXm0To1qPywVsXlNo Message-ID: From: Adrian Chadd To: "Jayachandran C." Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: freebsd-mips@freebsd.org Subject: Re: svn commit: r225892 - head/sys/mips/mips X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Oct 2011 10:31:32 -0000 Just FYI, the MIPS24K software users manual, section 9.2.1: 9.2.1 Wait IE Ignore A feature is included in the core that simplifies the task of using the WAIT instruction in the idle loop of an OS. The WAIT instruction is typically in block of code where the OS first checks to see if there is any pending work and if there is not, it will execute the WAIT as shown below. ... There is a tricky race condition present in this code. If an interrupt arrives between the pending check and the WAIT instruction, the service routine will return and execute the WAIT and go to sleep. However, the interrupt may have been enabling some pending work to be done in the =91bottom-half=92 processing. If the core goes back to sleep, this pending work will not be done until the next interrupt arrives. The OS can check to see if the interrupt was signalled in this window and adjust the EPC value to before the pending check, but this involves a fair amount of work. The Wait IE Ignore feature enables a simpler solution for the race con- dition. With this feature, a WAIT condition will be terminated by an active interrupt signal, even if that signal is pre- vented from causing an interrupt by StatusIE being clear or TCStatusIXMT being set. This allows interrupts to be disabled in this section of code while still allowing the WAIT to complete. That's likely a good solution for my platform, if it actually is implemented for my core. Would you mind seeing if XLR has something similar? I'll code this up tonight/tomorrow and see if I can coax my mips24k cores to work correctly. Adrian From owner-freebsd-mips@FreeBSD.ORG Sat Oct 1 12:08:32 2011 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 38AD31065670; Sat, 1 Oct 2011 12:08:32 +0000 (UTC) (envelope-from c.jayachandran@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 662108FC0C; Sat, 1 Oct 2011 12:08:31 +0000 (UTC) Received: by wyj26 with SMTP id 26so2523841wyj.13 for ; Sat, 01 Oct 2011 05:08:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=ePXSWMQDHmwYlc7/MH1AxYwtW3smAIiaxFWnRyQKO58=; b=bDSbkyzRB6Gp7C7Uxy2FBGZiCAUtPUz3iqAh7pyChjRJBPJzZTlz0TqM68VU5+JSI+ U7dVAV42S18tOB+zbILSb07sDg35nB693V7v6sD6C2RhzIdQ2Ahyxm2YH0GLyfC/YGTT +nRoJk5ZRSUP6xf3iW5Tgmdn81HPqMyRWSIwc= MIME-Version: 1.0 Received: by 10.216.220.131 with SMTP id o3mr13174859wep.11.1317470910417; Sat, 01 Oct 2011 05:08:30 -0700 (PDT) Sender: c.jayachandran@gmail.com Received: by 10.216.154.5 with HTTP; Sat, 1 Oct 2011 05:08:30 -0700 (PDT) In-Reply-To: References: <201110010556.p915uQH6003016@svn.freebsd.org> Date: Sat, 1 Oct 2011 17:38:30 +0530 X-Google-Sender-Auth: AlAnHcdzo8XFAY5_JpctiPghLws Message-ID: From: "Jayachandran C." To: Adrian Chadd Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: Alexander Motin , freebsd-mips@freebsd.org Subject: Re: svn commit: r225892 - head/sys/mips/mips X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Oct 2011 12:08:32 -0000 Hi Adrain, On Sat, Oct 1, 2011 at 4:01 PM, Adrian Chadd wrote: > Just FYI, the MIPS24K software users manual, section 9.2.1: > > > 9.2.1 Wait IE Ignore > A feature is included in the core that simplifies the task of using > the WAIT instruction in the idle loop of an OS. The WAIT instruction > is typically in block of code where the OS first checks to see if > there is any pending work and if there is not, it will execute the > WAIT as shown below. > ... > > There is a tricky race condition present in this code. If an interrupt > arrives between the pending check and the WAIT instruction, the > service routine will return and execute the WAIT and go to sleep. > However, the interrupt may have been enabling some pending work to be > done in the =91bottom-half=92 processing. If the core goes back to sleep, > this pending work will not be done until the next interrupt arrives. > > The OS can check to see if the interrupt was signalled in this window > and adjust the EPC value to before the pending check, but this > involves a fair amount of work. Linux code seems to have this solution, i.e in the interrupt handler, compare the EPC with the r4_wait function's address before the 'wait' and then fixup EPC if needed. > The Wait IE Ignore feature enables a > simpler solution for the race con- dition. With this feature, a WAIT > condition will be terminated by an active interrupt signal, even if > that signal is pre- vented from causing an interrupt by StatusIE being > clear or TCStatusIXMT being set. This allows interrupts to be disabled > in this section of code while still allowing the WAIT to complete. Thanks for finding this. > That's likely a good solution for my platform, if it actually is > implemented for my core. Would you mind seeing if XLR has something > similar? I'll code this up tonight/tomorrow and see if I can coax my > mips24k cores to work correctly. Looks like I will have to implement the EPC fixup solution for XLR. But in the end I think probably we will have two implementations, one for the processors which can safely wait with interrupts off without the scheduling latency, and the other for platforms that need the EPC fixup. JC. From owner-freebsd-mips@FreeBSD.ORG Sat Oct 1 16:53:21 2011 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2F1B2106566B; Sat, 1 Oct 2011 16:53:21 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id B6E658FC0A; Sat, 1 Oct 2011 16:53:20 +0000 (UTC) Received: by ywp17 with SMTP id 17so2986773ywp.13 for ; Sat, 01 Oct 2011 09:53:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=QZvPQ//JLjGCDJj8M+aJV9rHx63MZKL8zjenVx9rekc=; b=u6AnShXPKEy/H9+8fRdZEo5hOyNmKQH8wwfxoSyEuIh+XuVFR0uUhZsp4grR6NPXpw oE6012k/NEhYMTKy9nyLGL68ewBSFadceQE/bw8rtBmA3GtpAXtS9OFW/IlYCXYwqERn UBYivjd4wYImiJytaC2e5IQJQGstGkzaJGiIo= MIME-Version: 1.0 Received: by 10.236.129.242 with SMTP id h78mr78655722yhi.89.1317487999990; Sat, 01 Oct 2011 09:53:19 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.236.111.42 with HTTP; Sat, 1 Oct 2011 09:53:19 -0700 (PDT) In-Reply-To: References: <201110010556.p915uQH6003016@svn.freebsd.org> Date: Sun, 2 Oct 2011 00:53:19 +0800 X-Google-Sender-Auth: M1VnzSM1c3X9xYEBYHjUxbp2RIE Message-ID: From: Adrian Chadd To: "Jayachandran C." Content-Type: text/plain; charset=ISO-8859-1 Cc: Alexander Motin , freebsd-mips@freebsd.org Subject: Re: svn commit: r225892 - head/sys/mips/mips X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Oct 2011 16:53:21 -0000 Well, the fallback solution can be "no eventtimer, just call wait; and document the shortcomings." I'll see about implementing the workaround outlined in the MIPS24K software users manual from the MIPS website. Hopefully it fixes my problem. Is there an XLR specific workaround more suitable to your workload? What's the Linux codebase do? Adrian