From owner-freebsd-ppc@freebsd.org Thu May 2 10:45:55 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CCD9315902B1 for ; Thu, 2 May 2019 10:45:54 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-34.consmr.mail.ne1.yahoo.com (sonic317-34.consmr.mail.ne1.yahoo.com [66.163.184.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3DC0887B6B for ; Thu, 2 May 2019 10:45:53 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: ZvY95d0VM1k.P2R.WYxKPR26MiaQe6sxSGn.egRIIPu4v_olwWNLbh0Kg4ywLmy UYYL5IfK_o3SwOXXrDB164.s1sRlTjCGy56BoSXIJZX_UTafz5QniqNeO7UFRl4pGAEgzULzaoZa NuBrO9UyH5g5okHm7TyGm5C4Lo2KFbtbgEa3EI3gxhLsQ3aM6JdaBI9qpyCZjOeBy0RcYXaXmk4s F5n.Ii3dYy43YGm3iD6g6Q3qJyK_Zke1QvOzG2rkbLUnvtNTiNd8Svx2WkqzSUsU717DHaE2XSJV 7qSx6hUv5JKgJYu8KOS.5vORNTBRvy84Zhpc3PrwSjqVuk5QfSN5np5qkbCoXU2Q64b1ffTi2GC7 XE7.o_A6yJIeoMTCVqgO6sIAy2fhMQ.MnG.Bx50XqSQ62n7r3GlVP._590gT5i7w8U7MEb5P9fJU lL9HjY5Gl.ZATbYPbcFQ0pGsrVvDvhNiIaB2COLhg14ABWudiCoH605emXAHF3q9LrxDZyxyoNYJ Px1fxwO3YFMplz_v45ccppww2g9BybA1OU6sMuBQw9.1NwewM90MaSN7q0X1vjk_XqTo0R214ZIi pw7mwzNsVXNIzOYBKhttHnVz3qkzbZnxhbaKPWTusFYOIPpa6APE4wVlokwf5k1TIAE8yVJrK_q6 oHRFLciVH68orvNK4tvHwHpwuOsjPudV4Ja62QZSOXoCoSpT1YdTVExtRP_03DXXB.K7SQNN873C 9y0_Xq3.ltdoXe6dEFpwY.FDkyNxNxbYHtOTRwOftxe8pfTzKxqbZyDp0_KYissvYZXR7.EqGbbF aEpd3_xOF18UaxDraMYGvQMSMxoz18LxHHsxKXctDAunULSGHOK9cM6u6YRTI__eyyGvaK2i.CPP mez1Uibse_3lfiGeTNOsvnXaOgQ589jICpdfjqMR9QseKh9nlgdW_0GZ2XFR63sUE_NUsr_Y8b3h A6gOFW4Fe0saRuxDH0L43F93DRLNrVPqtnhX8JUKZitsuZwFNv9xfqqa6RRmwzNwkHzclbhIo254 cd25FNf_.DXpUejL2c_DY25Uw1VibOLryimBkB6Lpgwkmz3mqb6MgFvN4vLn8IjAPn_qWFBTBZbn AecS0Ph6hcC0XKc6G1lsZD_CSM1U.XtKdBm54MGhnj88x6IQPbNDPwj.e00guK9o_W5SkZldeLmx A3GzNsXHlsT2s Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.ne1.yahoo.com with HTTP; Thu, 2 May 2019 10:45:51 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.103]) ([76.115.7.162]) by smtp424.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 9423d2b4e1a8b798a6ec7750471bba68; Thu, 02 May 2019 10:45:49 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: How many segments does it take to span from VM_MIN_KERNEL_ADDRESS through VM_MAX_SAFE_KERNEL_ADDRESS? 128 in moea64_late_bootstrap From: Mark Millard In-Reply-To: Date: Thu, 2 May 2019 03:45:47 -0700 Cc: FreeBSD PowerPC ML Content-Transfer-Encoding: quoted-printable Message-Id: References: <3C69CF7C-7F33-4C79-92C0-3493A1294996@yahoo.com> <6159F4A6-9431-4B99-AA62-451B8DF08A6E@yahoo.com> <20190501094029.542c5f46@titan.knownspace> <212E50E5-7EB1-4381-A662-D5EACB1E5D46@yahoo.com> <20190501165403.7d8d1f8f@titan.knownspace> <1B8116F2-9749-4331-95BD-D528AA52A771@yahoo.com> To: Justin Hibbits X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 3DC0887B6B X-Spamd-Bar: + X-Spamd-Result: default: False [1.62 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36646, ipnet:66.163.184.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; NEURAL_SPAM_SHORT(0.67)[0.670,0]; NEURAL_HAM_LONG(-0.17)[-0.172,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(1.25)[ip: (3.81), ipnet: 66.163.184.0/21(1.38), asn: 36646(1.11), country: US(-0.06)]; NEURAL_SPAM_MEDIUM(0.38)[0.382,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[45.184.163.66.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[45.184.163.66.rep.mailspike.net : 127.0.0.17] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 May 2019 10:45:55 -0000 ["vt_upgrade(&vt_consdev). . ." hang-ups with the patch do happen for -r347003. The patch does not fix the overall hangs-up behavior, although it changes some detailed behavior that is associated. I've also avoided the panic issue by avoiding cmpb use. This does not fix the "mtx_lock of spin mutex WWV" but avoids it.] On 2019-May-1, at 23:21, Mark Millard wrote: > [Some results, mixed Im afraid.] >=20 > On 2019-May-1, at 17:22, Mark Millard wrote: >=20 >> On 2019-May-1, at 14:54, Justin Hibbits = wrote: >>=20 >>> On Wed, 1 May 2019 14:35:56 -0700 >>> Mark Millard wrote: >>>=20 >>>>>> What happens if you revert all your patches, =20 >>>>>=20 >>>>> Most of the patches in Bugzilla 233863 are not for this >>>>> issue at all and are not tied to starting the non-bsp >>>>> cpus. (The one for improving how close the Time Base >>>>> registers are is tied to starting these cpus.) Only the >>>>> aim/mp_cpudep.c and aim/slb.c changes seem relevant. >>>>>=20 >>>>> Are you worried about some form of interaction that means >>>>> I need to avoid patches for other issues? >>>>>=20 >>>>> Note: for now I'm staying at using head -r345758 as the >>>>> basis for my experiments. >>>>>=20 >>>>>> and change this loop to >>>>>> stop at n_slb? So something more akin to: >>>>>>=20 >>>>>> int i =3D 0; >>>>>>=20 >>>>>> for (va =3D virtual_avail; va < virtual_end && i < n_slb - >>>>>> 1; va +=3D SEGMENT_LENGTH, i++); >>>>>> ... >>>>>>=20 >>>>>> If it reliably boots with that, then that's fine. We can = prefault >>>>>> as much as we can and leave the rest for on-demand. =20 >>>>>=20 >>>>> I'm happy to experiment with this loop without my hack >>>>> for forcing the slb entry to exist in cpudep_ap_bootstrap. >>>>>=20 >>>>> But, it seems to presume that the pc_curpcb's will >>>>> all always point into the lower address range spanned >>>>> when cpudep_ap_bootstrap is executing on the cpu. >>>>> Does some known property limit the pc_curpcb-> >>>>> references to such? Only that would be sure to >>>>> avoid an slb-miss at that stage. Or is this just an >>>>> alternate hack or a means of getting evidence, not a >>>>> proposed solution? >>>>>=20 >>>>> (Again, I'm happy to disable my hack that forces the >>>>> slb entry and to try the loop suggested.) =20 >>> ... >>>> And the patch for the loop looks like: >>>>=20 >>>> virtual_end =3D VM_MAX_SAFE_KERNEL_ADDRESS;=20 >>>>=20 >>>> /* >>>> - * Map the entire KVA range into the SLB. We must not fault >>>> there. >>>> + * Map the lower-address part of the KVA range into the SLB. >>>> We must not fault there. */ >>>> #ifdef __powerpc64__ >>>> - for (va =3D virtual_avail; va < virtual_end; va +=3D >>>> SEGMENT_LENGTH) >>>> + i =3D 0; >>>> + for (va =3D virtual_avail; va < virtual_end && i>>> +=3D SEGMENT_LENGTH, i++) moea64_bootstrap_slb_prefault(va, 0); >>>> #endif >>>>=20 >>>=20 >>> Yep, that's the patch I was going for. >>>=20 >>>>=20 >>>> So I've built, installed, and have tested some: it did not go well >>>> overall. >>>>=20 >>>> Using: >>>>=20 >>>> OK set debug.verbose_sysinit=3D1 >>>>=20 >>>> to show better context about where the hangs occur, shows: >>>> (Typed from a screen picture.) >>>>=20 >>>> subsystem a800000 >>>> boot_run_interrupt_driven_config_hooks(0)... >>>> . . . (omitted) . . . >>>> done. >>>> vt_upgrade(&vt_consdev). . . >>>>=20 >>>> The "vt_upgrade(&vt_consdev). . ." never says done when booting >>>> hangs with the above changes. >>>>=20 >>>> Trying to boot a bunch of times did produce one >>>> completed boot, all 4 cpus working. Otherwise I'm >>>> using kernel.old to manage to complete a boot. >>>>=20 >>>> I'll note that "vt_upgrade(&vt_consdev). . ." is where >>>> Dennis Clarke reported for the hangups that he was >>>> seeing, without any of my patches being available back >>>> then: 2019-Feb-14. >>>=20 >>> Maybe try the commit that caused the problem back in July? r334498. >>>=20 >>=20 >> I'd already started down the path of getting materials from: >>=20 >> = https://artifact.ci.freebsd.org/snapshot/head/r347003/powerpc/powerpc64/ >>=20 >> and putting them on a separate SSD that I sometimes use for = artifact.ci >> or snapshot experiments. Also: checking out matching svn sources for >> -r347003 and then doing a buildworld buildkernel with a bootstrap gcc >> 4.2.1 compiler used. I'm verifying that I can build it before making >> the source changes for the kernel. The build is of a debug kernel >> (GENERIC64). >>=20 >> The test buildworld is still in process. >>=20 >> Let me know if this is insufficient for your purposes. I could revert >> to: >>=20 >> = https://artifact.ci.freebsd.org/snapshot/head/r334594/powerpc/powerpc64/ >>=20 >> (There is no head/r334498/ and the first after that with a >> powerpc64/ is head/r334594/ .) >>=20 >> For either head/r347003/ or head/r334594/ : >>=20 >> Use of artifact materials allows using officially built files for >> every file but some specific file(s) that I replace. It also allows >> comparison/contrast of the behavior of the official files vs. when >> adjusted ones are substituted. >>=20 >> Use of artifact-version materials also means that I know I'm using >> a vintage that actually built --and so I hope to avoid other problems >> getting in the way. >=20 > I present without-the-patch results before presenting > with-the-patch results. The end result is mixed, I'm > afraid. >=20 >=20 >=20 > As for the results without any patch, > just artifact materials . . . >=20 > Note: "Add debug.verbose_sysinit tunable for VERBOSE_SYSINIT" was > not checked-in until -r335458 . >=20 > Trying to boot without any updates or rebuilds, just artifact > materials shows variable stopping points: >=20 > (For debug.verbose_sysinit=3D1 :) > -r347003 stops sometimes at: vt_upgrade(&vt_consdev). . . > -r347003 stops sometimes at: cpu_mp_unleash(0). . . >=20 > -r334594 stops after: ada0 lines, VERBOSE_SYSINIT not built in >=20 >=20 >=20 > So I had to build my own -r334594 kernel to see verbose_sysinit > information about the stopping point. Again, no patch here, > I just copied over my build of the /boot/kernel/kernel file: >=20 > -r334594 stops sometimes at: vt_upgrade(&vt_consdev). . . > -r334594 stops sometimes at: cpu_mp_unleash(0). . . >=20 >=20 > Summary thus far: >=20 > I did not find any obvious difference in how often each stops > in either of the alternatives. >=20 > So I'm seeing if the proposed patch changes the behavior of > -r347003 . >=20 >=20 >=20 > Later test of patched -r347003 . . . >=20 > The patched kernel is based on: >=20 > # svnlite diff /mnt/usr/src/ | more > Index: /mnt/usr/src/sys/powerpc/aim/mmu_oea64.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- /mnt/usr/src/sys/powerpc/aim/mmu_oea64.c (revision 347003) > +++ /mnt/usr/src/sys/powerpc/aim/mmu_oea64.c (working copy) > @@ -959,7 +959,8 @@ > * Map the entire KVA range into the SLB. We must not fault = there. > */ > #ifdef __powerpc64__ > - for (va =3D virtual_avail; va < virtual_end; va +=3D = SEGMENT_LENGTH) > + i =3D 0; > + for (va =3D virtual_avail; va < virtual_end && i moea64_bootstrap_slb_prefault(va, 0); > #endif >=20 >=20 > So far with the patched code: >=20 > -r347003 has never stopped at: vt_upgrade(&vt_consdev). . . I have since had hang-ups at "vt_upgrade(&vt_consdev). . .". > -r347003 stops sometimes at: cpu_mp_unleash(0). . . [but differently!] > -r347003 panics at a particular point the rest of the time >=20 > The cpu_mp_unleash hangups report: > (typed from screen pictures) >=20 > subsystem f000000 > cpu_mp_unleash(0)... Launching APs 1 2 SMP: 4 CPUs found; 4 CPUs = usable; 3 CPUs woken >=20 > After that it is hung-up. >=20 >=20 > As for when that does not happen . . . >=20 > I do not even have /etc/fstab set up and so end up at the mountroot> > prompt. When I enter "ufs:/dev/daa0s3" I get a panic for: >=20 > panic: mtx_lock of spin mutex WWV @ = /mnt/usr/src/sys/powerpc/aim/mmu_oea64.c:2812 > (it is a debug-kernel build) >=20 > For reference, line 2812 is: PMAP_LOCK(pm); >=20 > panic is reached via an interesting(?) call chain, > showing the backtrace (typed from screen pictures): >=20 > .__mtx_lock_flags+0xd4 > .moea64_sync_icache+0x48 > .pmap_sync_icache+0x90 > .ppc_instr_emulate+0x1b4 > .trap+0x10fc > .powerpc_interrupt+0x2cc > user PGM trap by 0x810053bb4: srr1=3D0x900000000008d032 > r1=3D0x3ffffffffffffcc00 cr=3D0x20002024 xer=3D0 ctr=3D0x1 = r2=3D0x81007bdd0 frame=3D0xe000000070ca9810 >=20 > It was thread pid 28 tid 100097 >=20 > So far these details seem consistent. >=20 > But I will note that openfirmware use via ofwdump -ap > and the like causes system crashes going back to when > the direct map base was moved to high memory addresses > ( -r330610 and later ). This is one of the reasons I > want to avoid openfirmware and use the conversion to > fdt instead. (There is a -r330614 artifact to test > such crashes with --or use a later one that otherwise > boots.) I avoided the panics by adjusting src/lib/libc/powerpc64/string/strcmp.S to not use cmpb instructions. This does not fix the "mtx_lock of spin mutex WWV" but avoids it. So now there are two patches: # svnlite diff /mnt/usr/src/ Index: /mnt/usr/src/lib/libc/powerpc64/string/strcmp.S =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /mnt/usr/src/lib/libc/powerpc64/string/strcmp.S (revision = 347003) +++ /mnt/usr/src/lib/libc/powerpc64/string/strcmp.S (working copy) @@ -88,9 +88,16 @@ .Lstrcmp_compare_by_word: ld %r5,0(%r3) /* Load double words. */ ld %r6,0(%r4) - xor %r8,%r8,%r8 /* %r8 <- Zero. */ + lis %r8,32639 /* 0x7f7f */ + ori %r8,%r8,32639 /* 0x7f7f7f7f */ + rldimi %r8,%r8,32,0 /* 0x7f7f7f7f'7f7f7f7f */ xor %r0,%r5,%r6 /* Check if double words are different. = */ - cmpb %r7,%r5,%r8 /* Check if double words contain zero. = */ + /* Check for zero vs. not bytes: */ + and %r9,%r5,%r8 /* 0x00->0x00, 0x80->0x00, = other->ms-bit-in-byte=3D=3D0 */ + add %r9,%r9,%r8 /* ->0x7f, ->0x7f, = ->ms-bit-in-byte=3D=3D1 */ + nor %r7,%r9,%r5 /* ->0x80, ->0x00, = ->ms-bit-in-byte=3D=3D0 */ + andc %r7,%r7,%r8 /* ->0x80, ->0x00, ->0x00 = */ + /* sort of like cmpb %r7,%r5,%r8 for %r8 = being zero */ =20 /* * If double words are different or contain zero, @@ -104,7 +111,12 @@ ldu %r5,8(%r3) /* Load double words. */ ldu %r6,8(%r4) xor %r0,%r5,%r6 /* Check if double words are different. = */ - cmpb %r7,%r5,%r8 /* Check if double words contain zero. = */ + /* Check for zero vs. not bytes: */ + and %r9,%r5,%r8 /* 0x00->0x00, 0x80->0x00, = other->ms-bit-in-byte=3D=3D0 */ + add %r9,%r9,%r8 /* ->0x7f, ->0x7f, = ->ms-bit-in-byte=3D=3D1 */ + nor %r7,%r9,%r5 /* ->0x80, ->0x00, = ->ms-bit-in-byte=3D=3D0 */ + andc %r7,%r7,%r8 /* ->0x80, ->0x00, ->0x00 = */ + /* sort of like cmpb %r7,%r5,%r8 for %r8 = being zero */ =20 /* * If double words are different or contain zero, Index: /mnt/usr/src/sys/powerpc/aim/mmu_oea64.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /mnt/usr/src/sys/powerpc/aim/mmu_oea64.c (revision 347003) +++ /mnt/usr/src/sys/powerpc/aim/mmu_oea64.c (working copy) @@ -959,7 +959,8 @@ * Map the entire KVA range into the SLB. We must not fault = there. */ #ifdef __powerpc64__ - for (va =3D virtual_avail; va < virtual_end; va +=3D = SEGMENT_LENGTH) + i =3D 0; + for (va =3D virtual_avail; va < virtual_end && i