Date: Wed, 23 Aug 2017 22:26:36 +0300 From: Andriy Gapon <avg@FreeBSD.org> To: Don Lewis <truckman@FreeBSD.org>, freebsd-arch@FreeBSD.org Subject: Re: ULE steal_idle questions Message-ID: <d9dae0c1-e718-13fe-b6b5-87160c71784e@FreeBSD.org> In-Reply-To: <201708231504.v7NF4nYe035934@gw.catspoiler.org> References: <201708231504.v7NF4nYe035934@gw.catspoiler.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 23/08/2017 18:04, Don Lewis wrote:
> I've been looking at the steal_idle code in tdq_idled() and found some
> things that puzzle me.
>
> Consider a machine with three CPUs:
> A, which is idle
> B, which is busy running a thread
> C, which is busy running a thread and has another thread in queue
> It would seem to make sense that the tdq_load values for these three
> CPUs would be 0, 1, and 2 respectively in order to select the best CPU
> to run a new thread.
>
> If so, then why do we pass thresh=1 to sched_highest() in the code that
> implements steal_idle? That value is used to set cs_limit which is used
> in this comparison in cpu_search:
> if (match & CPU_SEARCH_HIGHEST)
> if (tdq->tdq_load >= hgroup.cs_limit &&
> That would seem to make CPU B a candidate for stealing a thread from.
> Ignoring CPU C for the moment, that shouldn't happen if the thread is
> running, but even if it was possible, it would just make CPU B go idle,
> which isn't terribly helpful in terms of load balancing and would just
> thrash the caches. The same comparison is repeated in tdq_idled() after
> a candidate CPU has been chosen:
> if (steal->tdq_load < thresh || steal->tdq_transferable == 0) {
> tdq_unlock_pair(tdq, steal);
> continue;
> }
>
> It looks to me like there is an off-by-one error here, and there is a
> similar problem in the code that implements kern.sched.balance.
I agree with your analysis. I had the same questions as well.
I think that the tdq_transferable check is what saves the code from
running into any problems. But it indeed would make sense for the code
to understand that tdq_load includes a currently running, never
transferable thread as well.
> The reason I ask is that I've been debugging random segfaults and other
> strange errors on my Ryzen machine and the problems mostly go away if I
> either disable kern.sched.steal_idle and kern_sched.balance, or if I
> leave kern_sched.steal_idle enabled and hack the code to change the
> value of thresh from 1 to 2. See
> <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221029>; for the gory
> details. I don't know if my CPU has what AMD calls the "performance
> marginality issue".
I have been following your experiments and it's interesting that
"massaging" the CPU in certain ways makes it a bit happier. But
certainly the fault is with the CPU as the code is trouble-free on many
different architectures including x86, and various processors from both
Intel and AMD [with earlier CPU families].
--
Andriy Gapon
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?d9dae0c1-e718-13fe-b6b5-87160c71784e>