Date: Sat, 18 Aug 2007 14:43:52 +0200 From: Pawel Jakub Dawidek <pjd@FreeBSD.org> To: freebsd-arch@FreeBSD.org Subject: Re: Lockless uidinfo. Message-ID: <20070818124352.GB6498@garage.freebsd.pl> In-Reply-To: <20070818120056.GA6498@garage.freebsd.pl> References: <20070818120056.GA6498@garage.freebsd.pl>
next in thread | previous in thread | raw e-mail | index | archive | help
--U+BazGySraz5kW0T Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Aug 18, 2007 at 02:00:56PM +0200, Pawel Jakub Dawidek wrote: > Hi. >=20 > The patch below remove per-uidinfo locks: >=20 > http://people.freebsd.org/~pjd/patches/uidinfo_lockless.patch >=20 > With the patch uidinfo is handled using atomics only and no locks > (except for the global hash lock, which is not really important, as it's > not used in the fast paths). >=20 > I needed to change ui_sbsize from rlim_t (64bit) to long, because we > don't have 64bit atomics on all archs, and because sbsize represents > size in bytes, it can't go beyond 32bit on 32bit archs (PAE might be a > bit of a problem). >=20 > I changed maxval argument in chgproccnt() from int to rlim_t, as this is > what is passed to the function. >=20 > In simple ping-pong test on unix domain socket, uidinfo lock was highly > contented: >=20 > max total wait_total count avg wait_avg cnt_hold cn= t_lock name > 1508 3242859 96267052 1467476 2 65 374553 102= 4743 /usr/src/sys/kern/kern_resource.c:1339 (sleep mutex:sleep mtxpool) >=20 > The ping-pong program you can find here: >=20 > http://people.freebsd.org/~pjd/misc/unixpingpong.c >=20 > At the end we reduced uidinfo structure size by 8 bytes and gain no > measurable performance improvements:) > Yes, I wasn't able to measure anything interesting, unfortunately, but I > still believe the patch should be committed, as I'm sure there are > workloads that will see improvements - note that uidinfo structure is > per-uid, so if there are thousands of processes running with the same > uid, they all need to fight for this one lock. > Not only contention is important, but also the fact that number of > atomic operations in chgsbsize() and chgproccnt() functions was reduced > from 2 to 1. >=20 > Ok, while writting this e-mail I came up with a better benchmark: >=20 > http://people.freebsd.org/~pjd/misc/unixpingpong2.c >=20 > This one doesn't do ping-pong between two processes, but within one > proccess only. This way we eliminate cost of context switches. I was > running 8 such processes (I tested it on a 8way machine) and here are > the results: >=20 > x ./uidinfo1.txt > + ./uidinfo2.txt > +------------------------------------------------------------------------= ------+ > |x = +| > |xx = ++| > |xx = ++| > |A| = |A| > +------------------------------------------------------------------------= ------+ > N Min Max Median Avg Stdd= ev > x 5 402742 416417 406216 408269.2 5388.34= 85 > + 5 1561566 1575987 1568964 1569767 5853.13= 99 > Difference at 95.0% confidence > 1.1615e+06 +/- 8204.54 > 284.493% +/- 2.00959% > (Student's t, pooled s =3D 5625.55) >=20 > As you can see this was just a matter of good benchmark - here we can see > 284% performance improvement, yay:) Kris asked me to verify just in case that the unit cost is not higher and here are the results from running only one ping-pong process (so no contention): x ./uidinfo_up1.txt + ./uidinfo_up2.txt +--------------------------------------------------------------------------= ----+ |x x x x x + + + + = +| | |________A_M_______| |___________A_M__________= | | +--------------------------------------------------------------------------= ----+ N Min Max Median Avg Stddev x 5 415023 420059 418405 418016.6 1878.5011 + 5 423731 430984 428084 427735.8 2623.4883 Difference at 95.0% confidence 9719.2 +/- 3327.59 2.32508% +/- 0.796043% (Student's t, pooled s =3D 2281.61) We can see 2.3% of improvement most likely because of atomic operations reduction. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --U+BazGySraz5kW0T Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFGxumIForvXbEpPzQRAoRAAKDUmAIgqAc6kx4KvXs+snHuAiqWegCg7z6C VtWabfYPU5PAeGxjX4Tg8cA= =rtmv -----END PGP SIGNATURE----- --U+BazGySraz5kW0T--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070818124352.GB6498>