From owner-freebsd-ppc@freebsd.org Wed Apr 17 02:07:10 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8105B1583422 for ; Wed, 17 Apr 2019 02:07:10 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic308-3.consmr.mail.bf2.yahoo.com (sonic308-3.consmr.mail.bf2.yahoo.com [74.6.130.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4F8B48A386 for ; Wed, 17 Apr 2019 02:07:09 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: gfyCQ4kVM1nnCax45zVhTD6TkH6on3_GFlDYDqrOYRH8cuw5B1WbpVakcBwnlZU Kkl8nXQ2E8oYfyTuq986TaiIyrs92Ss5XAA4u9pnlkgpBWgKmAHJzo66uJ3mfwG6PHVnePUUaPK0 PofbocINqiJCO8bLDkX3HOi9ibVfStWS8FDs5M9gcXbCsfnmMCimrATzQwIc0ituHEWHxhoLW34S _EwmNVmDdSyZdaj1ZOcq223fe660._s_OmIbNSdBLC3dT6FN0QvM6SMxUiWnP7b21fPjFzUW2QfY Yx6oY4WmlEN0SuVJuMISq5i4hrpOM7P3mjEDluMyqhvxcTXwbEEv9crowHzaBf0FmRy5f5.EWfMg xTlyp1mzUT47IosKezbXZQUK.yWafN8adTq6WVb_X13mmstCkFNESwmkZOuQzlWflcv43jY41ct0 eeKHrc2_rZKkM6NbSwmj.S1SPmmO7vf6OlVW8O0JC5vQPWat48qNiQkhRoWjiYCern3CBlCyfk9t NEHKqbV0uddDN0HnMW5GS6M6ukcH7EIU0gHWu7CG9pkztGPmctRrsSS9M.zg46vMO1R9L6abyJ5e TsLmmaGEa4YuBjGXtU0nfEi0mOmYW4Y.ZDe3Z638KQINkjJc6Vs_Wbbr23lAfuBmyhAPy9XCe2VC 4StOGQGFqRu0Ipf3RD4kqtgfc_xA8IQAHzCqf7gBKrxVvVffRAQBCZRr9TRJszUfNrgq20B8TetG 6OYuAWaNYuiJHcbMGAW6.uc5mBGlKFbnQfPUuLzdrzpOhNLmZ1tBNZEAafDO8r6u9XCzci2cTUHC A2k9QK_29T5uJe.SMvNBaC8n5BV.qex3geySHq7c5DWjBxREESpLDTa08xx7KjKIfRIv5fG4Whr5 fK3FsT5toNhwLFmpmK.rCVRoQn.9qWSj7JC26dkiIXdA6L_eoTpCrRjlWchGmi2QHU.cyl01v96u .k3jEXJtbPAknG3vywaGJlsB4nplg9uC_lOop8KJXLgfE01LpM112ZmdarXW7fWF7tbYXjXuBMX6 sLUVsu0nwsBY9dC4lsWXedR60JJgQboOQz2xX3c2f6IrizOPBGhMCH3.eXQiSeW374fBpOlLnumC t4hiFMqtq_seA3mHP_DlToEud2A-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic308.consmr.mail.bf2.yahoo.com with HTTP; Wed, 17 Apr 2019 02:07:03 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.103]) ([76.115.7.162]) by smtp432.mail.bf1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 02be651f3c2f2bcf7b4c536d12630005; Wed, 17 Apr 2019 02:07:02 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: head -r344018 powerpc64 variant on Powermac G5 (2 sockets, 2 cores each): [*buffer arena] shows up more . . .? [mpc85xx_smp_timebase_sync problem too] From: Mark Millard In-Reply-To: <306BC23A-3261-41BE-AB0B-4264E864DAA5@yahoo.com> Date: Tue, 16 Apr 2019 19:06:59 -0700 Cc: freebsd-ppc Content-Transfer-Encoding: quoted-printable Message-Id: <0CC10B05-F0FD-483C-911D-20F5AC2DB427@yahoo.com> References: <20190306151914.44ea831c@titan.knownspace> <8668AAF7-9E6A-4278-9D1B-2ECDBD3804AA@yahoo.com> <99AD89F8-0F90-48BE-A060-DA12FD7129E6@yahoo.com> <20190306223611.75c8a87e@titan.knownspace> <984188AE-93A9-4321-99EC-522C2E0CE9A9@yahoo.com> <20190416164210.26827a6d@titan.knownspace> <8141B858-08B0-4B88-8439-747023A98822@yahoo.com> <306BC23A-3261-41BE-AB0B-4264E864DAA5@yahoo.com> To: Justin Hibbits X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 4F8B48A386 X-Spamd-Bar: ++ X-Spamd-Result: default: False [2.92 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:26101, ipnet:74.6.128.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; NEURAL_SPAM_SHORT(0.80)[0.805,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(1.43)[ip: (4.34), ipnet: 74.6.128.0/21(1.59), asn: 26101(1.27), country: US(-0.06)]; NEURAL_SPAM_MEDIUM(0.53)[0.526,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.67)[0.672,0]; RCVD_IN_DNSWL_NONE(0.00)[42.130.6.74.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2019 02:07:10 -0000 [Experiments produced an unexpected result, from what I can tell. I note them.] On 2019-Apr-16, at 17:20, Mark Millard wrote: > [Looks a little odder in the details than I initially thought: the > 0 in platform_smp_timebase_sync(timebase, 0) does not mean the bsp > instead of some other ap. I've still not found a reasonable way > to not have the sleep-gets-stuck issue for usefdt mode.] >=20 > On 2019-Apr-16, at 15:36, Mark Millard wrote: >=20 >> [Unfortunately cpufreq_drv_set leads to additional >> platform_smp_timebase_sync(timebase, 0) use.] >>=20 >> On 2019-Apr-16, at 14:42, Justin Hibbits = wrote: >>=20 >>> On Tue, 16 Apr 2019 14:26:39 -0700 >>> Mark Millard wrote: >>>=20 >>>> [Looks to me like the use and content of mpc85xx_smp_timebase_sync >>>> have the same type of problems I noted for the proposed powermac >>>> patch.] >>>>=20 >>> ... >>>>>=20 >>>>> As mentioned, I had only compiled it. Your examination of the = code >>>>> path demonstrates that the patch is insufficient, and would hang = at >>>>> unleash anyway. The sleep/wake logic probably needs to be updated >>>>> anyway. It was written for a G4 powerbook primarily for the >>>>> PMU-based cpufreq driver, so some bits might need to be moved >>>>> around. Orthogonal to this issue, though. =20 >>>>=20 >>>> It appears to me that the overall sequence: >>>>=20 >>>> platform_smp_timebase_sync(0, 1); // called from >>>> cpudep_ap_setup . . . >>>> // The following are from in machdep_ap_bootstrap . . . >>>> while (ap_letgo =3D=3D 0) >>>> __asm __volatile("or 31,31,31"); >>>> __asm __volatile("or 6,6,6"); >>>> . . . >>>> platform_smp_timebase_sync(ap_timebase, 1); >>>>=20 >>>> calls mpc85xx_smp_timebase_sync twice per ap and that the >>>> 2nd call has tb_ready && mp_ncpus<=3Dcpu_done for each such >>>> ap, leading to a lack of coordination of the activity that >>>> 2nd time for each ap: >>>>=20 >>>> static void >>>> mpc85xx_smp_timebase_sync(platform_t plat, u_long tb, int ap) >>>> { >>>> static volatile bool tb_ready; >>>> static volatile int cpu_done; >>>>=20 >>>> if (ap) { >>>> /* APs. Hold off until we get a stable timebase. */ >>>> while (!tb_ready) >>>> atomic_thread_fence_seq_cst(); >>>> mttb(tb); >>>> atomic_add_int(&cpu_done, 1); >>>> while (cpu_done < mp_ncpus) >>>> atomic_thread_fence_seq_cst(); >>>> } else { >>>> /* BSP */ >>>> freeze_timebase(rcpm_dev, true); >>>> tb_ready =3D true; >>>> mttb(tb); >>>> atomic_add_int(&cpu_done, 1); >>>> while (cpu_done < mp_ncpus) >>>> atomic_thread_fence_seq_cst(); >>>> freeze_timebase(rcpm_dev, false); >>>> } >>>> } >>>>=20 >>>=20 >>> Not for mpc85xx. This is call is only in the AIM cpudep_ap_setup, = and >>> really shouldn't be there anyway. >>=20 >> Sorry for having looked in the wrong source. >>=20 >>> The original code to just set the >>> timebase was there to set it to 0 just as a semi-sane value until = the >>> core got to a stable state. The original code was literally = "mttb(0)" >>> for G4, and a check for hypervisor with mttb(0). Had that been >>> sufficient, the platform_smp_timebase_sync() would not be a problem = to >>> do the real sync as I had mentioned in my patch before. >>>=20 >>> The powermac patch that I had provided was derived from this >>> well-working mpc85xx patch. However, I had neglected to check that = the >>> sync wasn't used elsewhere as well. If the cpudep_ap_setup() use is >>> removed, then this should work fine. Not with your patch, just the original code, I tried removing the platform_smp_timebase_sync(0, 1) in cpudep_ap_setup . The consequence was that the variability in time across cpus got much worse (wider range of times). The scale on which my hack had been calibrated was not nearly sufficient to prevent there being lots of stuck-sleeping. (I did not try to see if recalibrating the range it spans would make things work again.) I experimented with powerpd vs. lack of it under these conditions, and its cpufreq activity seem to sometimes limit the variability, rather than making it worse. (It was not sufficient to make things reasonable.) I've put back the platform_smp_timebase_sync(0, 1) in cpudep_ap_setup for now. With that, my hack and its old caliberation again makes things work without stuck-sleeping problems. It almost makes me wonder if the mttb(0) that it causes is needed to reach a usable tb status for setting other values later. That might partially explain the comment: /* The following is needed for restoring from sleep. */ platform_smp_timebase_sync(0, 1); It reads like it was observed at some point to be important to the operation. >> Unfortunately there is use from cpufreq_drv_set getting to = cpu_sleep's >> platform_smp_timebase_sync use as well: >>=20 >> /usr/src/sys/powerpc/powerpc/mp_machdep.c: = platform_smp_timebase_sync(ap_timebase, 1); >> /usr/src/sys/powerpc/powerpc/mp_machdep.c: = platform_smp_timebase_sync(ap_timebase, 0); >> /usr/src/sys/powerpc/aim/aim_machdep.c: = platform_smp_timebase_sync(timebase, 0); >>=20 >> that last is in cpu_sleep. Tracing towards what ultimately uses = cpu_sleep . . . >>=20 >> void >> powermac_sleep(platform_t platform) >> { >>=20 >> *(unsigned long *)0x80 =3D 0x100; >> cpu_sleep(); >> } >>=20 >> is used as platform_sleep. >>=20 >> pu_mu_set_speed calls platform_sleep and is called by pmufreq_set. >> pmufreq_set is used as cpufreq_drv_set. >>=20 >> At least on the PowerMac11,2, this seems to be in use for >> powerd if powerpd is started. >>=20 >> It also looks like such use does not try to make the other aps >> track the change, just ap=3D=3D0 . >>=20 >> That may explain much of the variability across CPUs for usefdt mode! >>=20 >> It looks to me like the ap=3D=3D0 case in the = platform_smp_timebase_sync >> use in cpu_sleep would not be well behaved under the patch, >> even for that one ap: >>=20 >> static void >> powermac_smp_timebase_sync(platform_t plat, u_long tb, int ap) >> { >> + static int cpus; >> + static int unleash; >> + >> + if (ap) { >> + atomic_add_int(&cpus, 1); >> + while (!atomic_load_acq_int(&unleash)) >> + ; >> + } else { >> + atomic_add_int(&cpus, 1); >> + while (atomic_load_int(&cpus) !=3D mp_ncpus) >> + ; >> + atomic_store_rel_int(&unleash, 1); >> + } >>=20 >> mttb(tb); >> } >>=20 >> The else would increment cpus beyond mp_cpus and the loop >> would be stuck. >=20 > cpu_sleep getting to powermac_smp_timebase_sync seems to be based > on cpu_reset_handler doing the longjmp in: >=20 > GET_CPUINFO(%r5) > ld %r3,(PC_RESTORE)(%r5) > cmpldi %cr0,%r3,0 > beq %cr0,2f > nop > li %r4,1 > bl CNAME(longjmp) > nop > 2: > #ifdef SMP > bl CNAME(machdep_ap_bootstrap) /* And away! */ > nop > #endif >=20 > The PC_RESTORE related test is tied to: >=20 > PCPU_SET(restore, &resetjb); >=20 > in cpu_sleep. The usage is: >=20 > if (setjmp(resetjb) =3D=3D 0) { > . . . > timebase =3D mftb(); > . . . > while (1) > mtmsr(msr); > } > platform_smp_timebase_sync(timebase, 0); > ... >=20 > So it seems there is a platform_smp_timebase_sync(timebase, 0) > for every cpu that gets cpu_reset_handler invoked, even the > cpu's that are not the bsp. >=20 > I had assumed the ", 0" meant a bsp context previously. >=20 > There is: >=20 > static u_quad_t timebase =3D 0; >=20 > So timebase is not preserved for each cpu if more than > one ends up in cpu_sleep. I'm not sure the timebase value > is synchronized across cpus. (It is not volatile either. > I'm not sure that the code generation would always > store and load the value from memory.) >=20 > Overall it looks like the ", 0" is (ambiguously) intending to > indicate that the platform_smp_timebase_sync activity should > be unsynchronized with other cpus. May be a distinct, alternate > argument value for that that is explicitly handled? >=20 >=20 >=20 > So I'm still not to a stage of having a reasonable workaround > for the "sleep gets stuck" problem for usefdt mode. (Just an > inappropriate avoidance-hack.) >=20 > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D233863 has > investigative patches for other usefdt tied issues, patches > that could be useful starting points for official updates. > But the hack that I've been using to avoid stuck-sleeps is > not appropriate --and so is not submitted. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)