From owner-freebsd-ppc@freebsd.org Fri May 3 18:09:14 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 11B0F15973C0 for ; Fri, 3 May 2019 18:09:14 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic316-54.consmr.mail.gq1.yahoo.com (sonic316-54.consmr.mail.gq1.yahoo.com [98.137.69.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0378B83557 for ; Fri, 3 May 2019 18:09:12 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: fs9m7s4VM1lJTcsgrF0Jzexeuz7kdpOucD_eavsAlfzlXv6JDos1vKrSdTxECSq wZ_VflsaFwUbnQ8Rceao.PNr8GN6D5GDcUehsW5GpbTMYv14rSiJGWvl5..8baw4yA3RMlMv9wPv jfeCU1oXqRlBPjaKYUrNIsgq2n.ySk93fWL9GeCPimBlcsJZDtcZ3XBrb7RtsezgJGyIyFcK7rZA EIX7TmoDwbYBb2roR1hMGMF2OanWnfV6WOuRNm3dXDQg4fwhNCOF13vsNu5lvP1akUl9BrL3m4ue kkm5dGSsw3UZ_3RN8B9ECltCMs8gKf_TApD8kS8KothNz8w2sbvDL7lXEks.H8MhnrrhTaoMLAk_ He3Uv16n3849wyCMVwDo6PrqAZWTLP90Cl1JfFeHt.WGU3HFnEPKXpnWMcB854Byz9oVjKFpob6Q oJvZiug4Ys0f4IAoRmF9tduX3KezrFKLMkbv2uYS170wfC9k_qkDa79ctNzb8.MffQbR8Y2ix0tm 9.sLXKWk9iAsulRXuMgXgF1foOY4KGBcjkLnrCQqIXmIWSO_VoR1tPHHeVdAlEwkW4ye1AJSurD5 EXkUtO.Mkhv1pG2XL9pj6QWqoVHuJvKNz3UAk3tOcBpsa7vt8.iudx6aKqOfJeqRjHcLCEHEP8Wd Y77E2wtYd.R7FeiTXNk5PYMm3ofu9fVOqOXxoG5ZrCprwS42eAXnp.S8dkK5d.KucprQabr9WXyY w6nQFjh0fYTdr6.8oIxjQPNVOVmMb96G.EASJWXYA6S0QqWdSxcVKi0x0vJrti009YNknjuY6hZw Yl_apJDV8EqRX_K8oXTF1TGNqUSjeDDQT7JE.zkUi5Eu0W6N3pirjPA1GIiOH6vG_QdH5gzgF3yq v_ZH_zJM5xkUhg1R5yNs9LOX2PeVYPB6aMB0nbuxXcEqnu5vFdQ.3Ah6rgK2pS9oA1SYuALkBjeM vG8G8yITRXvEI1l3npF1GLUdW7uVTJJrQKSjjSnzQ7pujFDfYj0ynifcm6BldOnq6KEuDMIQIQ_v FVpe5ysJocNRUOgz.8M84e5oB5rNvMqCuWFgvgJ2wj19DFlbXiu2Dg_jNp_Pet4echBXm8GVB3te uBsjezY1OOzD6YI3DdYumpq3cSKSoRkBeSMledPxp_1LXVqrxGKGDVzxPmGmYukcmUgwDiQoCLnz jGw-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic316.consmr.mail.gq1.yahoo.com with HTTP; Fri, 3 May 2019 18:09:06 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.103]) ([76.115.7.162]) by smtp411.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 9bea4b3cac9e558ca30720c42fcf7b3a; Fri, 03 May 2019 18:09:03 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: 970MP PowerMac G5s: What printf's show about cpu_mp_unleash hangups on the test variant of head -r347003 (found cpu_mp_unleash counterexample) Date: Fri, 3 May 2019 11:09:03 -0700 References: To: Justin Hibbits , FreeBSD PowerPC ML In-Reply-To: Message-Id: <4D659851-8731-4116-A6B6-33A75E9F0B76@yahoo.com> X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 0378B83557 X-Spamd-Bar: ++ X-Spamd-Result: default: False [2.20 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(0.97)[0.970,0]; NEURAL_HAM_LONG(-0.05)[-0.045,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(1.14)[ip: (4.16), ipnet: 98.137.64.0/21(0.88), asn: 36647(0.70), country: US(-0.06)]; NEURAL_SPAM_MEDIUM(0.65)[0.654,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[30.69.137.98.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 May 2019 18:09:14 -0000 [Exeriment with instead disabling the code in each of the nop_prio_ routines instead of commenting out specific calls. But mostly: more test runs. It does not support what I thought yesterday's cpu_mp_unlead suggested.] On 2019-May-2, at 22:23, Mark Millard wrote: > [Note: I still have your requested loop change, my > isync additions, and my libc string compare code > change in what I'm working with for head -r347003 .] >=20 > I started using printf to help identify more about what > code managed to execute vs what code did not for > hang-ups. >=20 > This note is just about cpu_mp_unleash observations and > experiments related to what printf's showed. >=20 > I did: >=20 > static void > cpu_mp_unleash(void *dummy) > { > . . . (omitted as all earlier printf's printed) . . . > printf("cpu_mp_unleash: before DELAY\n"); > /* Let the APs get into the scheduler */ > DELAY(10000); > printf("cpu_mp_unleash: after DELAY\n"); >=20 > } >=20 > What I saw was only the first of the twoDEALY printf's > shown above was printing when cpu_mp_unleash hung up, > such a hangup being the normal case when vt_upgrade > did not hang-up first. >=20 > So I looked at /mnt/usr/src/sys/powerpc/powerpc/clock.c > and its DELAY routine and came up with only one thing > that looked like a useful experiment. Note what I > then commented out: >=20 > # svnlite diff /mnt/usr/src/sys/powerpc/powerpc/clock.c > Index: /mnt/usr/src/sys/powerpc/powerpc/clock.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- /mnt/usr/src/sys/powerpc/powerpc/clock.c (revision 347003) > +++ /mnt/usr/src/sys/powerpc/powerpc/clock.c (working copy) > @@ -309,10 +309,10 @@ > TSENTER(); > tb =3D mftb(); > ttb =3D tb + howmany((uint64_t)n * 1000000, ps_per_tick); > - nop_prio_vlow(); > + //nop_prio_vlow(); > while (tb < ttb) > tb =3D mftb(); > - nop_prio_medium(); > + //nop_prio_medium(); > TSEXIT(); > } >=20 > After this change I've not (yet?) seen another cpu_mp_unleash > hangup in my test context. >=20 > Even if not documented to do so, it appears to me that > ori Rx,Rx,Rx code that is behind the nop_prio_vlow() does > something specific on the 970MP's in the 2-socket/2-core-each > G5 PowerMac11,2's --and what it does interferes with making > progress in DELAY, in at least that specific use of it and/or > any others on the ap's during cpu_mp_unleash. >=20 > Of course, this testing process is of a probabilistic context > and I do not have hundreds or more of examples of any specific > condition at this point. But, so far, the change in behavior > seems clear: I went from always-hanging-up-so-far to > always-booting-so-far (when vt_upgrade did not prevent the > test in each context). So I uncommented the 2 calls commented out the contents of the nop_prio_ routines, summarized here via: # egrep -r 'or.*(31,31,31|1,1,1|6,6,6|2,2,2|5,5,5|3,3,3)' = /mnt/usr/src/sys/powerpc/ | more = = /mnt/usr/src/sys/powerpc/include/cpufunc.h: //__asm __volatile("or = 31,31,31"); /mnt/usr/src/sys/powerpc/include/cpufunc.h: //__asm __volatile("or = 1,1,1"); /mnt/usr/src/sys/powerpc/include/cpufunc.h: //__asm __volatile("or = 6,6,6"); /mnt/usr/src/sys/powerpc/include/cpufunc.h: //__asm __volatile("or = 2,2,2"); /mnt/usr/src/sys/powerpc/include/cpufunc.h: //__asm __volatile("or = 5,5,5"); /mnt/usr/src/sys/powerpc/include/cpufunc.h: //__asm __volatile("or = 3,3,3"); Then I've continued running tests based on the rebuild. The results so far are . . . vt_upgrade behavior is unchanged, still frequently hanging-up in the same place. fhanew_init behavior is unchanged, still frequently hanging-up in the same place (when vt_upgrade does not hang-up). It will take some time to get a number of examples where the prior 2 hang-ups do not occur, but for cpu_mp_unlead I have seen a "3 CPUs woken" notice followed by only the "before DELAY" message, no "after DELAY" one. I confirmed with objdump that DELAY did not have the ori Rx,Rx,Rx instructions. So, despite how things looked yesterday with no cpu_mp_unlead hang-ups after using the code with the 2 calls commented out, ori 31,31,31 use is not involved in the latest example, matching your expectations. (And this illustrates why I try to accumulate a fair number of examples over time: the hang-ups are probabilistic and small example counts can be misleading.) =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)