From owner-freebsd-ppc@freebsd.org Sat Feb 16 20:07:29 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2065614F3695 for ; Sat, 16 Feb 2019 20:07:29 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic310-22.consmr.mail.ne1.yahoo.com (sonic310-22.consmr.mail.ne1.yahoo.com [66.163.186.203]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0A3136B425 for ; Sat, 16 Feb 2019 20:07:27 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: qQ3VsMQVM1nHB7K78Yn12ouHMqePWBUoGXTg7krfdWqwujjWp_z8zNXsVFfCGuY ZmAzgHwcxOCMfbQzHvR5BM9H6k0HDFEC.fAIEa4CV1weHnwExUZKvXjsYWWx8QVdaSMfCAccdnYI DPy_TbppL_JPqOS3bHCPh_I7friQsDdvmdZDgcSSFrZMmkKy1hpc6zKkdDDe4lCRgSZ6InD.9WCv mrCYUl7MWfn3kYwlEEQwFzVWdAnhFkC7dZGE6QU8Do8TxynujvVc6bkNPMc97AyDGeMdDNZvd2Np eyzkjc1ia9oI7EvfZqzXwnVy5W5H4GqYgQhgJ02wvpXLYB.oAJXbiNyopNQwoKyYXuE3WuO4IynY YGA2rBin0yVGxP4l_vTFA7tz8YSevFpOolU0x4GZOmUerMKDxXcWIJYbhEHnzgkj.PMusyTFFREy ktw6YYhZYx4sJZqq9NAIVuLSvFX.O11s.CbwgN4j9CjARYOQ8GuThman2bLXw8eAg3_d7dY.wKhG dv5vUbT_0mH1CRUdLlbpvj2rMmV7DR4056nMZQXsn9.usDirKo4bfN5ngq_rNBpIcPf1h186hKnS SFY215cpSDF0gnX9sG1bIVX8U44cE8cXnJfXh7UfJp1_4I_uBaWCmKfZglU0JISwywudLgE8Ffyy n2iLHalvllUTRTP4dx3tiSqcvafORSTA8s0oj13Sh1rwPz8cN31vYSx3oOeuhJMxLX9IDobj1lNh xcOkFrXuXy86inm5jV7igJnGj1OugzszrdlvkwExvZaHGviMHZvRH7MKYWTXYhuCzxaeCbbOjOnT xiEvfbVNkvQ4YE79YSEsCiAgNvftfEZ5dnBHi.LQy9Pp.Y3UVeUb5hnVzqc0kcZ3f8S15R77xMDb 34bjLCT0dMaWD2AwgYtbIO.kLxqSzywt79kkOl9eK1BmPPiIEz4SzTC_C..W1cxAqKkXKR3nvYC_ HTb.hABE7sdOfk3QPuIY3RzRXXlBWYMOJB2tyjUKKc_bpUj6kXLr7z6D1KBzKBoictw8VWgCQLiL GkN03BXVX5YJ26rqunmOb2Z4NGJZk4S4C_aavlqQb0xpVqlMeWwNp2R1WJw-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.ne1.yahoo.com with HTTP; Sat, 16 Feb 2019 20:07:20 +0000 Received: from c-67-170-167-181.hsd1.or.comcast.net (EHLO [192.168.1.113]) ([67.170.167.181]) by smtp432.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 00fc36668c17770762178c48f9a1a251; Sat, 16 Feb 2019 20:07:20 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: Re: Some evidence about the PowerMac G5 multiprocessor boot hang ups with the modern VM_MAX_KERNEL_ADDRESS value [found more staging info] From: Mark Millard In-Reply-To: Date: Sat, 16 Feb 2019 12:07:18 -0800 Cc: Mark Millard via freebsd-ppc Content-Transfer-Encoding: quoted-printable Message-Id: <518C5B96-75C4-4C24-BDEE-68A542242CA3@yahoo.com> References: <11680D15-D43D-4115-AF4F-5F6E4E0022C9@yahoo.com> <9FBCA729-CE80-44CD-8873-431853E55231@yahoo.com> <1F3411CF-3D28-43C0-BEF1-4672B5CC1543@yahoo.com> <20190215151710.35545a26@ralga.knownspace> <6445CE54-26AA-4E21-B17E-921D72D4081A@yahoo.com> <20190215160942.1b282f71@ralga.knownspace> <744610C7-90EB-42A0-8B08-AFA0F12E5994@yahoo.com> <20190215180421.61afcae3@ralga.knownspace> To: Justin Hibbits X-Mailer: Apple Mail (2.3445.102.3) X-Rspamd-Queue-Id: 0A3136B425 X-Spamd-Bar: ++ X-Spamd-Result: default: False [2.23 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36646, ipnet:66.163.184.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(0.66)[0.655,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(1.17)[ip: (3.93), ipnet: 66.163.184.0/21(1.11), asn: 36646(0.88), country: US(-0.07)]; NEURAL_SPAM_MEDIUM(0.87)[0.868,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.05)[0.048,0]; RCVD_IN_DNSWL_NONE(0.00)[203.186.163.66.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[203.186.163.66.rep.mailspike.net : 127.0.0.17] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Feb 2019 20:07:29 -0000 [I needed to allow more time after the 2 resets before having CPU 0 look at the memory. It was reporting older values instead of my added writes. The odd non-zero value was from before the activity of interest.] I start with the new result found, then give supporting material. I've now seen hangs with: *(unsigned long*)0xc0000000000000f0)=3D0x10 for CPU 3. So the following completed: void cpudep_ap_early_bootstrap(void) { #ifndef __powerpc64__ register_t reg; #endif switch (mfpvr() >> 16) { case IBM970: case IBM970FX: case IBM970MP: /* Restore HID4 and HID5, which are necessary for the = MMU */ #ifdef __powerpc64__ mtspr(SPR_HID4, bsp_state[2]); powerpc_sync(); isync(); mtspr(SPR_HID5, bsp_state[3]); powerpc_sync(); isync(); #else __asm __volatile("ld %0, 16(%2); sync; isync; \ mtspr %1, %0; sync; isync;" : "=3Dr"(reg) : "K"(SPR_HID4), "b"(bsp_state)); __asm __volatile("ld %0, 24(%2); sync; isync; \ mtspr %1, %0; sync; isync;" : "=3Dr"(reg) : "K"(SPR_HID5), "b"(bsp_state)); #endif powerpc_sync(); break; case IBMPOWER8: case IBMPOWER8E: case IBMPOWER9: #ifdef __powerpc64__ if (mfmsr() & PSL_HV) { isync(); /* * Direct interrupts to SRR instead of HSRR and * reset LPCR otherwise */ mtspr(SPR_LPID, 0); isync(); =20 mtspr(SPR_LPCR, lpcr); isync(); } #endif break; } __asm __volatile("mtsprg 0, %0" :: "r"(ap_pcpu)); powerpc_sync(); *(unsigned long*)0xc0000000000000f0 =3D 0x10; // HACK!!! powerpc_sync(); // HACK!!! } but the following (and later) did not complete: void pmap_cpu_bootstrap(int ap) { /* =20 * No KTR here because our console probably doesn't work yet */ =20 return (MMU_CPU_BOOTSTRAP(mmu_obj, ap)); =20 *(unsigned long*)0xc0000000000000f0 =3D 0x20; // HACK!!! powerpc_sync(); // HACK!!! } Background for reference relative to showing the intended values. . . I now have: *rstvec =3D 4; powerpc_sync(); (void)(*rstvec); powerpc_sync(); DELAY(1); *rstvec =3D 0; powerpc_sync(); (void)(*rstvec); powerpc_sync(); if (bootverbose) // HACK!!! printf("After reset 4&0 for CPU %d, hwref=3D%jx, = awake=3D%x, *(unsigned long*)0xc0000000000000e0=3D0x%jx, *(unsigned = long*)0xc0000000000000f0=3D0x%jx\n", pc->pc_cpuid, (uintmax_t)pc->pc_hwref, pc->pc_awake,(uintmax_t)*(unsigned = long*)0xc0000000000000e0,(uintmax_t)*(unsigned = long*)0xc0000000000000f0); timeout =3D 10000; while (!pc->pc_awake && timeout--) DELAY(100); if (bootverbose) // HACK!!! printf("After attempted wait for awake CPU %d, = hwref=3D%jx, awake=3D%x, *(unsigned long*)0xc0000000000000e0=3D0x%jx, = *(unsigned long*)0xc0000000000000f0=3D0x%jx\n", pc->pc_cpuid, (uintmax_t)pc->pc_hwref, pc->pc_awake,(uintmax_t)*(unsigned = long*)0xc0000000000000e0,(uintmax_t)*(unsigned = long*)0xc0000000000000f0); The 2nd printf shows the expected values but the first above shows old memory values. The memory at 0xc0000000000000e0 hold the get_pcpu() result. The memory at 0xc0000000000000f0 hold the 0x?? values that I'd reported = earlier although I've added a 0x5E but really use the value at = 0xc0000000000000e0 to check since 0x5e would be replaced by 0x51: struct pcpu* thepcpu =3D get_pcpu(); // HACK!!! *(struct pcpu**)0xc0000000000000e0 =3D thepcpu; // HACK!!! if ( thepcpu=3D=3D&__pcpu[1] // HACK!!! || thepcpu=3D=3D&__pcpu[2] || thepcpu=3D=3D&__pcpu[3] ) *(unsigned long*)0xc0000000000000f0 =3D 0x5F; // HACK!!! else *(unsigned long*)0xc0000000000000f0 =3D 0x5E; // HACK!!! powerpc_sync(); // HACK!!! PCPU_SET(awake, 1); __asm __volatile("msync; isync"); *(unsigned long*)0xc0000000000000f0 =3D 0x51; // HACK!!! powerpc_sync(); // HACK!!! while (ap_letgo =3D=3D 0) __asm __volatile("or 31,31,31"); __asm __volatile("or 6,6,6"); For reference, a successful boot now looks like: Adding CPU 0, hwref=3Dcd38, awake=3D1 Trying to mount root from ufs:/dev/ufs/FBSDG5L2rootfs [rw,noatime]... Waking up CPU 3 (dev=3Dc480) powermac_smp_start_cpu 's OF_getprop for CPU 3, hwref=3Dc480, awake=3D0: = res=3D4, reset=3D8c powermac_smp_start_cpu for CPU 3, hwref=3Dc480, awake=3D0: = rstvec_virtbase=3D0xe000000087fd2000 powermac_smp_start_cpu for CPU 3, hwref=3Dc480, awake=3D0: = rstvec=3D0xe000000087fd208c Before reset 4&0 for CPU 3, hwref=3Dc480, awake=3D0 After reset 4&0 for CPU 3, hwref=3Dc480, awake=3D0, *(unsigned = long*)0xc0000000000000e0=3D0x0, *(unsigned long*)0xc0000000000000f0=3D0x0 After attempted wait for awake CPU 3, hwref=3Dc480, awake=3D1, = *(unsigned long*)0xc0000000000000e0=3D0xc0000000016c6100, *(unsigned = long*)0xc0000000000000f0=3D0x51 cpu_mp_unleash attempting to wait for pc_awake: CPU 3, hwref=3Dc480, = awake=3D1 cpu_mp_unleash after platform_smp_start_cpu and waiting: CPU 3, = hwref=3Dc480, awake=3D1 Adding CPU 3, hwref=3Dc480, awake=3D1 Waking up CPU 2 (dev=3Dc768) powermac_smp_start_cpu 's OF_getprop for CPU 2, hwref=3Dc768, awake=3D0: = res=3D4, reset=3D8b powermac_smp_start_cpu for CPU 2, hwref=3Dc768, awake=3D0: = rstvec=3D0xe000000087fd208b Before reset 4&0 for CPU 2, hwref=3Dc768, awake=3D0 After reset 4&0 for CPU 2, hwref=3Dc768, awake=3D0, *(unsigned = long*)0xc0000000000000e0=3D0xc0000000016c6100, *(unsigned = long*)0xc0000000000000f0=3D0x51 After attempted wait for awake CPU 2, hwref=3Dc768, awake=3D1, = *(unsigned long*)0xc0000000000000e0=3D0xc0000000016c5100, *(unsigned = long*)0xc0000000000000f0=3D0x51 cpu_mp_unleash attempting to wait for pc_awake: CPU 2, hwref=3Dc768, = awake=3D1 cpu_mp_unleash after platform_smp_start_cpu and waiting: CPU 2, = hwref=3Dc768, awake=3D1 Adding CPU 2, hwref=3Dc768, awake=3D1 Waking up CPU 1 (dev=3Dca50) powermac_smp_start_cpu 's OF_getprop for CPU 1, hwref=3Dca50, awake=3D0: = res=3D4, reset=3D8a powermac_smp_start_cpu for CPU 1, hwref=3Dca50, awake=3D0: = rstvec=3D0xe000000087fd208a Before reset 4&0 for CPU 1, hwref=3Dca50, awake=3D0 After reset 4&0 for CPU 1, hwref=3Dca50, awake=3D0, *(unsigned = long*)0xc0000000000000e0=3D0xc0000000016c5100, *(unsigned = long*)0xc0000000000000f0=3D0x51 After attempted wait for awake CPU 1, hwref=3Dca50, awake=3D1, = *(unsigned long*)0xc0000000000000e0=3D0xc0000000016c4100, *(unsigned = long*)0xc0000000000000f0=3D0x51 cpu_mp_unleash attempting to wait for pc_awake: CPU 1, hwref=3Dca50, = awake=3D1 cpu_mp_unleash after platform_smp_start_cpu and waiting: CPU 1, = hwref=3Dca50, awake=3D1 Adding CPU 1, hwref=3Dca50, awake=3D1 machdep_ap_bootstrap before ap_boot_mtx lock: AP CPU #2 launched machdep_ap_bootstrap before ap_boot_mtx lock: AP CPU #1 launched machdep_ap_bootstrap before ap_boot_mtx lock: AP CPU #3 launched SMP: AP CPU #2 launched SMP: AP CPU #1 launched SMP: AP CPU #3 launched machdep_ap_bootstrap after smp_started!=3D0: AP CPU #2 launched machdep_ap_bootstrap after smp_started!=3D0: AP CPU #1 launched machdep_ap_bootstrap after smp_started!=3D0: AP CPU #3 launched =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)