Date: Tue, 19 Feb 2013 20:51:29 +0200 From: Konstantin Belousov <kostikbel@gmail.com> To: Svatopluk Kraus <onwahe@gmail.com> Cc: freebsd-current@freebsd.org Subject: Re: [patch] i386 pmap sysmaps_pcpu[] atomic access Message-ID: <20130219185129.GC2598@kib.kiev.ua> In-Reply-To: <CAFHCsPXQHuR8a2JjChmPoFc9YQWPT4NW-3hEs=Beo6FeAsTLGQ@mail.gmail.com> References: <CAFHCsPUVTM9jfrnzY72YsPszLWkg-UaJcycTR4xXcS%2BfPzS1Vg@mail.gmail.com> <20130218150809.GG2598@kib.kiev.ua> <CAFHCsPVbkwj7fhqax5D5kk89VZgAjW9gT8uJunjevav2eTUbNQ@mail.gmail.com> <20130218170957.GJ2598@kib.kiev.ua> <CAFHCsPXE-0-_SsYFYGdFnUDO6Hp9PErmq1kf3js87f2GZuTs6w@mail.gmail.com> <20130218203630.GO2598@kib.kiev.ua> <CAFHCsPXQHuR8a2JjChmPoFc9YQWPT4NW-3hEs=Beo6FeAsTLGQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--6dGMKdYe2Ft9UtxE Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Feb 18, 2013 at 11:18:16PM +0100, Svatopluk Kraus wrote: > On Mon, Feb 18, 2013 at 9:36 PM, Konstantin Belousov > <kostikbel@gmail.com> wrote: > Well, I'm taking a part on porting FreeBSD to ARM11mpcore. UP case was > simple. SMP case is more complex and rather new for me. Recently, I > was solving a problem with PCPU stuff. For example, PCPU_GET is > implemented by one instruction on i386 arch. So, a need of atomicity > with respect to interrupts can be overlooked. On load-store archs, the > implementation which works in SMP case is not so simple. And what > works in UP case (single PCPU), not works in SMP case. Believe me, > mysterious and sporadic 'mutex not owned' assertions and others ones > caused by curthreads mess, it takes a while ... Note that PCPU_GET() is not needed to be atomic. The reason is that the code which uses its result would not be atomic (single-instruction or whatever, = see below). Thus, either the preemption should not matter since the action with the per-cpu data is advisory, as is in the case of i386 pmap you noted. or some external measures should be applied in advance to the containing region (which you proposed, by bracing with sched_pin()). Also, note that it is not interrupts but preemption which is concern. >=20 > After this, I took a look at how PCPU stuff is used in whole kernel > and found out mentioned here i386 pmap issue. So, my concern is > following: >=20 > 1. to verify my newly gained ideas how per CPU data should be used, > 2. to decide how to change my implementation of pmap on ARM11mpcore, > as it's based on i386 pmap, > 3. to make FreeBSD code better. >=20 > In the meanwhile, it looks that using data dedicated to one CPU on > another one is OK. However, I can't agree. At least, without comments, > it is misleading for anyone new in FreeBSD and makes code misty. >=20 > > Both threads in your description make useful progress, and computation > > proceeds correctly. >=20 > I thought, there is only one thread in my example. One thread running > on CPU1, but holding sysmaps dedicated to CPU0 instead of holding > sysmaps dedicated to CPU1. So, any thread running on CPU0 must wait > because the thread running on CPU1 is a thief. Futhermore, the idea > that a thread on CPU1 should hold data for CPU1 is not valid. So, > either some comment is missing in the code that it's OK or the using > of PCPU_GET(cpuid) is unneeded and some kind of free sysmaps list can > be used and it will serve better. What is wrong on almost all architectures except x86 is PCPU_ADD(), which is usually supposed by the MD code to be atomic in regard to _both_ interrupts and preemption. I said almost all, but in fact I am not aware of any architecture except x86 where it is right. And, RISCs with the load-link and store-conditional (ll/sc) primitives are well capable of doing the PCPU_ADD() correctly. You could look at the projects/counters branch, where single-instruction increment is used on x86 for dynamic per-cpu counters, and critical_enter() for other architectures. I might do some work for other arches, but the counters are correct but slower that it could be, on !x86. --6dGMKdYe2Ft9UtxE Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJRI8mxAAoJEJDCuSvBvK1B5koQAKa2FXuEJYXOp77FYXQqME1r b73a6IqZFz68RHwKVe66CLMqCWt3Ej+r8EoqQKbl9N3DbvEWWkNrJthSAEYZqT6q jcfMoJTI4qNhTPtuuRnY4mJYyFNAe74/aabIRSEJyzEGEPekJ2M7+K06drRPtD5Y PkqKavZwtPOgAgRDnEuMZcztNjsQbWFkAP15tpTfMyhFeDhpLZy+FFJ1Rus8aFKK lwqWXZK7+yBdNULi5XwgN3lb+Qu6e2thaDEh2psRKsfpwvrPc35jYlBftXQjYfUb rR5qDO6LgHxOEIpunAEo6CQsfH4LK69XhMeQgBELMRlNbkWcfyMr0+ussymvFm9X s6goFnlX9ajbEq/XkrlDDmkR5aKCVr8l6Y2j7o3zT6FESccBXMF4mLLo3KFnv0nM rAaonT+RIWzwXw6Y+zrY/rAjJ0pam6vxo5sazd3L9EEaiCfpCNmRNbmFr0k3ONP8 4PHPxVBrL3GCLG6uLeiWpTfw09S5UligzutcIf9C37QUPBAzTQNS52z9sgyR+F/E L78hp9TeWs89OHJKwKksYzgvT3c1huH4H97uKhv5U0jwt7cb+cmU+DCMNXT2CrCF qYttncg6WGpKn6wcrthZ5nTZBehw06ma1WFN5hjRJFNyGrLh5hNKSoUvHUOQFFjD wS47WRblh+H98bLPHECY =cqAF -----END PGP SIGNATURE----- --6dGMKdYe2Ft9UtxE--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130219185129.GC2598>