Date: Thu, 5 Apr 2012 06:56:45 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Sushanth Rai <sushanth_rai@yahoo.com> Cc: freebsd-hackers@freebsd.org Subject: Re: Startvation of realtime piority threads Message-ID: <20120405035645.GO2358@deviant.kiev.zoral.com.ua> In-Reply-To: <1333590846.58474.YahooMailClassic@web180011.mail.gq1.yahoo.com> References: <1333590846.58474.YahooMailClassic@web180011.mail.gq1.yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--KKfHGHXZBgaZT5Zm Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Apr 04, 2012 at 06:54:06PM -0700, Sushanth Rai wrote: > I have a multithreaded user space program that basically runs at realtime= priority. Synchronization between threads are done using spinlock. When ru= nning this program on a SMP system under heavy memory pressure I see that t= hread holding the spinlock is starved out of cpu. The cpus are effectively = consumed by other threads that are spinning for lock to become available.= =20 >=20 > After instrumenting the kernel a little bit what I found was that under m= emory pressure, when the user thread holding the spinlock traps into the ke= rnel due to page fault, that thread sleeps until the free pages are availab= le. The thread sleeps PUSER priority (within vm_waitpfault()). When it is r= eady to run, it is queued at PUSER priority even thought it's base priority= is realtime. The other siblings threads that are spinning at realtime prio= rity to acquire the spinlock starves the owner of spinlock.=20 >=20 > I was wondering if the sleep in vm_waitpfault() should be a MAX(td_user_p= ri, PUSER) instead of just PUSER. I'm running on 7.2 and it looks like this= logic is the same in the trunk. It just so happen that your program stumbles upon a single sleep point in the kernel. If for whatever reason the thread in kernel is put off CPU due to failure to acquire any resource without priority propagation, you would get the same effect. Only blockable primitives do priority propagation, that are mutexes and rwlocks, AFAIR. In other words, any sx/lockmgr/sleep points are vulnerable to the same issue. Speaking of exactly your problem, did you considered wiring the memory of your realtime process ? This is a common practice, used e.g. by ntpd. --KKfHGHXZBgaZT5Zm Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAk99F/0ACgkQC3+MBN1Mb4gWIACghu8Uy8q/hfORgXAb3d/gkJyJ xyMAmwYeXIBi4xxiod19/r/LXuvpqz/2 =8MEs -----END PGP SIGNATURE----- --KKfHGHXZBgaZT5Zm--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120405035645.GO2358>