From owner-freebsd-ppc@freebsd.org Wed May 1 21:54:10 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F0F3B15A1E4C for ; Wed, 1 May 2019 21:54:09 +0000 (UTC) (envelope-from chmeeedalf@gmail.com) Received: from mail-it1-x141.google.com (mail-it1-x141.google.com [IPv6:2607:f8b0:4864:20::141]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E6EEB8F336 for ; Wed, 1 May 2019 21:54:08 +0000 (UTC) (envelope-from chmeeedalf@gmail.com) Received: by mail-it1-x141.google.com with SMTP id p18so4031412itm.1 for ; Wed, 01 May 2019 14:54:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FZZO5EtdKy7INewSJdSbFgA4W53XN13eazv0D4L3a1k=; b=Up53zamq38UeNbqq1w7RfsHgXK6GDoiAiwqowwY0LBjXKZjIw+ua7rOss+cMXxYCvN z63deE8i7yBcntMF7KhvzPJxkg1t+06iQvTFjnL+Vh3c8VC+38HxioppvjpJI/Qtarew o31BhFuYgJrOqNcov5fceUx2zcY+DrvvFkYbxIm8XhKXwEg4sETqQGvGTc0azrbApCii acdC9dXoh4AUqEzXA9n+Txj4iThpP2z8bwQVyL9eTqv1Q9y70q4syjV1wU6D6gnIoi54 cpbxcCXPNE+clL1F+TxwMM0iIkVgBrAznuUmAnFsSJlEcXpVIZUsx3Dhg1RTixzE4P8K kmPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FZZO5EtdKy7INewSJdSbFgA4W53XN13eazv0D4L3a1k=; b=dp/8tPdf5oD8iHqhv9pKQtTxpSUoLdpeeqY9k5oJinHOEKhHnOIp1d4djuv47GZvcE WYpEb9mIZUe94oygRv85Gg6enb8qeFaMbi+2icIbfoNuMQJgKrE5O8crWGG6qReQdLG0 oep7Hks1rTgQto8B9gHptpUw0xAoDwEONk5F5nedAxFJvDredM0j+Df4lASHBif5Uxez +CPsKc61QSMOxKx4fapYbK4LJty/VGLjalouhUh+p9Lm1h048h9TbPlRT7DJx98wuLb5 SnyRYGDAz6Kx62/M1GavvqT52d6r3LaPMaBMQcN4hZH4kFhvao/Pfxm0Xhc59S7MQ+86 lpFA== X-Gm-Message-State: APjAAAVLLavMQjiqoGW+t0EgpbJ/uYWB3+a5jD91iaSaTCqxkXJ+rNw6 gUO5e1J8x9r1OunI6J2Bsm1tihOt X-Google-Smtp-Source: APXvYqwTomqouBKDX+bm9Qht1Br6MSTKHcHqHFDW8+xAI8r0BUaVw8MQDgKRg2xiydrDWrrLDQeNqw== X-Received: by 2002:a05:660c:20e:: with SMTP id y14mr9159068itj.17.1556747647893; Wed, 01 May 2019 14:54:07 -0700 (PDT) Received: from titan.knownspace (173-25-245-129.client.mchsi.com. [173.25.245.129]) by smtp.gmail.com with ESMTPSA id y22sm13949382iod.65.2019.05.01.14.54.07 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 01 May 2019 14:54:07 -0700 (PDT) Date: Wed, 1 May 2019 16:54:03 -0500 From: Justin Hibbits To: Mark Millard Cc: FreeBSD PowerPC ML Subject: Re: How many segments does it take to span from VM_MIN_KERNEL_ADDRESS through VM_MAX_SAFE_KERNEL_ADDRESS? 128 in moea64_late_bootstrap Message-ID: <20190501165403.7d8d1f8f@titan.knownspace> In-Reply-To: References: <3C69CF7C-7F33-4C79-92C0-3493A1294996@yahoo.com> <6159F4A6-9431-4B99-AA62-451B8DF08A6E@yahoo.com> <20190501094029.542c5f46@titan.knownspace> <212E50E5-7EB1-4381-A662-D5EACB1E5D46@yahoo.com> X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; powerpc64-portbld-freebsd13.0) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: E6EEB8F336 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Up53zamq; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of chmeeedalf@gmail.com designates 2607:f8b0:4864:20::141 as permitted sender) smtp.mailfrom=chmeeedalf@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.17)[-0.171,0]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FREEMAIL_TO(0.00)[yahoo.com]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(-0.82)[ip: (1.40), ipnet: 2607:f8b0::/32(-3.19), asn: 15169(-2.25), country: US(-0.06)]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; SUBJECT_HAS_QUESTION(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-ppc@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[1.4.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 May 2019 21:54:10 -0000 On Wed, 1 May 2019 14:35:56 -0700 Mark Millard wrote: > >> What happens if you revert all your patches, > > > > Most of the patches in Bugzilla 233863 are not for this > > issue at all and are not tied to starting the non-bsp > > cpus. (The one for improving how close the Time Base > > registers are is tied to starting these cpus.) Only the > > aim/mp_cpudep.c and aim/slb.c changes seem relevant. > > > > Are you worried about some form of interaction that means > > I need to avoid patches for other issues? > > > > Note: for now I'm staying at using head -r345758 as the > > basis for my experiments. > > > >> and change this loop to > >> stop at n_slb? So something more akin to: > >> > >> int i = 0; > >> > >> for (va = virtual_avail; va < virtual_end && i < n_slb - > >> 1; va += SEGMENT_LENGTH, i++); > >> ... > >> > >> If it reliably boots with that, then that's fine. We can prefault > >> as much as we can and leave the rest for on-demand. > > > > I'm happy to experiment with this loop without my hack > > for forcing the slb entry to exist in cpudep_ap_bootstrap. > > > > But, it seems to presume that the pc_curpcb's will > > all always point into the lower address range spanned > > when cpudep_ap_bootstrap is executing on the cpu. > > Does some known property limit the pc_curpcb-> > > references to such? Only that would be sure to > > avoid an slb-miss at that stage. Or is this just an > > alternate hack or a means of getting evidence, not a > > proposed solution? > > > > (Again, I'm happy to disable my hack that forces the > > slb entry and to try the loop suggested.) ... > And the patch for the loop looks like: > > virtual_end = VM_MAX_SAFE_KERNEL_ADDRESS; > > /* > - * Map the entire KVA range into the SLB. We must not fault > there. > + * Map the lower-address part of the KVA range into the SLB. > We must not fault there. */ > #ifdef __powerpc64__ > - for (va = virtual_avail; va < virtual_end; va += > SEGMENT_LENGTH) > + i = 0; > + for (va = virtual_avail; va < virtual_end && i += SEGMENT_LENGTH, i++) moea64_bootstrap_slb_prefault(va, 0); > #endif > Yep, that's the patch I was going for. > > So I've built, installed, and have tested some: it did not go well > overall. > > Using: > > OK set debug.verbose_sysinit=1 > > to show better context about where the hangs occur, shows: > (Typed from a screen picture.) > > subsystem a800000 > boot_run_interrupt_driven_config_hooks(0)... > . . . (omitted) . . . > done. > vt_upgrade(&vt_consdev). . . > > The "vt_upgrade(&vt_consdev). . ." never says done when booting > hangs with the above changes. > > Trying to boot a bunch of times did produce one > completed boot, all 4 cpus working. Otherwise I'm > using kernel.old to manage to complete a boot. > > I'll note that "vt_upgrade(&vt_consdev). . ." is where > Dennis Clarke reported for the hangups that he was > seeing, without any of my patches being available back > then: 2019-Feb-14. Maybe try the commit that caused the problem back in July? r334498. > You wrote in another reply: > > > The idea with this is if you can test with stock -CURRENT (or > > post-VM_KERNEL_MAXADDR change), to eliminate any other variables. > > This is *only* for testing that it brings up the APs, not that > > they're properly synced. That will happen with other changes. > > This is a proposed solution. From my understanding, we typically > > allocate from low to high for KVA allocations, so keeping the low > > addresses in memory long enough to bring up the APs to sanity is > > the goal, so the commit would be along the lines of "Prefault as > > much of KVA as we can fit into the SLB". > > This will have the sleep-gets-stuck problem, likely normally happening > quickly after booting and logging in (presuming a boot). The resulting > boots for such are not always all that useful after various threads > hang up. As mentioned, that's a different problem to solve. If we can at least get the APs going, that's a big step up in the first place. - Justin