Date: Tue, 19 Mar 2019 22:10:21 +0100 From: =?UTF-8?Q?Mi=C5=82osz_Kaniewski?= <m.kaniewski@fudosecurity.com> To: freebsd-hackers@freebsd.org Subject: sched_pickcpu() and a pinned thread. Message-ID: <CABQUGZzxrnS%2B9xkrbufc9cx1jyG%2B1V553AurYGeSV-WtJX0A2w@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hi, I have an application which exchanges packets between two NIC ports. Most of a CPU time is therefore consumed by the userspace process and two IRQ threads. All of them are pinned to the same CPU using cpu_setaffinity() function so I expect that the scheduler usage should be minimal even on a high network load when a lot of interrupts are generated. I recently find out that my application spends a lot of time in cpu_search_lowest() function. A backtrace shown me that cpu_search_lowest() is called by sched_pickcpu(). I tried to analyze this second function but I don't know what exactly should I expect from it if my thread is pinned to a specific CPU. I assume that the function should get to a point where ts->ts_cpu is returned. However it doesn't do it and sched_pickcpu() always ends looking for the lowest busy CPU using cpu_search_lowest(). In result it finds the CPU that the thread is pinned to but it is very time consuming process. Therefore I suspect that sched_pickcpu() should return before that happens. For interrupt threads I see that there is a special case at the beginning of sched_pickcpu(): /* * Prefer to run interrupt threads on the processors that generate * the interrupt. */ pri =3D td->td_priority; if (td->td_priority <=3D PRI_MAX_ITHD && THREAD_CAN_SCHED(td, self) && curthread->td_intr_nesting_level && ts->ts_cpu !=3D self) { SCHED_STAT_INC(pickcpu_intrbind); ts->ts_cpu =3D self; if (TDQ_CPU(self)->tdq_lowpri > pri) { SCHED_STAT_INC(pickcpu_affinity); return (ts->ts_cpu); } } However IRQ threads from the NIC doesn't fall into this case because ts->ts_cpu =3D=3D self (these variables are equal to the CPU to which my threads are pinned). Is this check required? For non-interrupt threads there are some other possible options where ts->ts_cpu is returned however I am not sure which should handle a pinned thread. I would be grateful if someone would give me some clue how should I understand the sched_pickcpu() function and when it should return in case of a pinned thread. Thanks Mi=C5=82osz Kaniewski
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CABQUGZzxrnS%2B9xkrbufc9cx1jyG%2B1V553AurYGeSV-WtJX0A2w>