Date: Fri, 3 May 2019 11:09:03 -0700 From: Mark Millard <marklmi@yahoo.com> To: Justin Hibbits <chmeeedalf@gmail.com>, FreeBSD PowerPC ML <freebsd-ppc@freebsd.org> Subject: Re: 970MP PowerMac G5s: What printf's show about cpu_mp_unleash hangups on the test variant of head -r347003 (found cpu_mp_unleash counterexample) Message-ID: <4D659851-8731-4116-A6B6-33A75E9F0B76@yahoo.com> In-Reply-To: <EF44A358-CC6D-4244-A911-6D4DACFF4B21@yahoo.com> References: <EF44A358-CC6D-4244-A911-6D4DACFF4B21@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[Exeriment with instead disabling the code in each of
the nop_prio_ routines instead of commenting out specific
calls. But mostly: more test runs. It does not support
what I thought yesterday's cpu_mp_unlead suggested.]
On 2019-May-2, at 22:23, Mark Millard <marklmi at yahoo.com> wrote:
> [Note: I still have your requested loop change, my
> isync additions, and my libc string compare code
> change in what I'm working with for head -r347003 .]
>=20
> I started using printf to help identify more about what
> code managed to execute vs what code did not for
> hang-ups.
>=20
> This note is just about cpu_mp_unleash observations and
> experiments related to what printf's showed.
>=20
> I did:
>=20
> static void
> cpu_mp_unleash(void *dummy)
> {
> . . . (omitted as all earlier printf's printed) . . .
> printf("cpu_mp_unleash: before DELAY\n");
> /* Let the APs get into the scheduler */
> DELAY(10000);
> printf("cpu_mp_unleash: after DELAY\n");
>=20
> }
>=20
> What I saw was only the first of the twoDEALY printf's
> shown above was printing when cpu_mp_unleash hung up,
> such a hangup being the normal case when vt_upgrade
> did not hang-up first.
>=20
> So I looked at /mnt/usr/src/sys/powerpc/powerpc/clock.c
> and its DELAY routine and came up with only one thing
> that looked like a useful experiment. Note what I
> then commented out:
>=20
> # svnlite diff /mnt/usr/src/sys/powerpc/powerpc/clock.c
> Index: /mnt/usr/src/sys/powerpc/powerpc/clock.c
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> --- /mnt/usr/src/sys/powerpc/powerpc/clock.c (revision 347003)
> +++ /mnt/usr/src/sys/powerpc/powerpc/clock.c (working copy)
> @@ -309,10 +309,10 @@
> TSENTER();
> tb =3D mftb();
> ttb =3D tb + howmany((uint64_t)n * 1000000, ps_per_tick);
> - nop_prio_vlow();
> + //nop_prio_vlow();
> while (tb < ttb)
> tb =3D mftb();
> - nop_prio_medium();
> + //nop_prio_medium();
> TSEXIT();
> }
>=20
> After this change I've not (yet?) seen another cpu_mp_unleash
> hangup in my test context.
>=20
> Even if not documented to do so, it appears to me that
> ori Rx,Rx,Rx code that is behind the nop_prio_vlow() does
> something specific on the 970MP's in the 2-socket/2-core-each
> G5 PowerMac11,2's --and what it does interferes with making
> progress in DELAY, in at least that specific use of it and/or
> any others on the ap's during cpu_mp_unleash.
>=20
> Of course, this testing process is of a probabilistic context
> and I do not have hundreds or more of examples of any specific
> condition at this point. But, so far, the change in behavior
> seems clear: I went from always-hanging-up-so-far to
> always-booting-so-far (when vt_upgrade did not prevent the
> test in each context).
So I uncommented the 2 calls commented out the contents of
the nop_prio_ routines, summarized here via:
# egrep -r 'or.*(31,31,31|1,1,1|6,6,6|2,2,2|5,5,5|3,3,3)' =
/mnt/usr/src/sys/powerpc/ | more =
=
/mnt/usr/src/sys/powerpc/include/cpufunc.h: //__asm __volatile("or =
31,31,31");
/mnt/usr/src/sys/powerpc/include/cpufunc.h: //__asm __volatile("or =
1,1,1");
/mnt/usr/src/sys/powerpc/include/cpufunc.h: //__asm __volatile("or =
6,6,6");
/mnt/usr/src/sys/powerpc/include/cpufunc.h: //__asm __volatile("or =
2,2,2");
/mnt/usr/src/sys/powerpc/include/cpufunc.h: //__asm __volatile("or =
5,5,5");
/mnt/usr/src/sys/powerpc/include/cpufunc.h: //__asm __volatile("or =
3,3,3");
Then I've continued running tests based on the rebuild. The
results so far are . . .
vt_upgrade behavior is unchanged, still frequently hanging-up
in the same place.
fhanew_init behavior is unchanged, still frequently hanging-up
in the same place (when vt_upgrade does not hang-up).
It will take some time to get a number of examples where the
prior 2 hang-ups do not occur, but for cpu_mp_unlead I have
seen a "3 CPUs woken" notice followed by only the
"before DELAY" message, no "after DELAY" one. I confirmed
with objdump that DELAY did not have the ori Rx,Rx,Rx
instructions.
So, despite how things looked yesterday with no cpu_mp_unlead
hang-ups after using the code with the 2 calls commented out,
ori 31,31,31 use is not involved in the latest example, matching
your expectations.
(And this illustrates why I try to accumulate a fair number of
examples over time: the hang-ups are probabilistic and small
example counts can be misleading.)
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4D659851-8731-4116-A6B6-33A75E9F0B76>