Date: Fri, 18 Mar 2016 12:37:24 -0700 From: "K. Macy" <kmacy@freebsd.org> To: John Baldwin <jhb@freebsd.org> Cc: "freebsd-arch@freebsd.org" <freebsd-arch@freebsd.org>, "arch@freebsd.org" <arch@freebsd.org> Subject: Re: Starting APs earlier during boot Message-ID: <CAHM0Q_P_DZg9XpuP0ZRj0nBFhPxUK1VZ8SuEuaB4r9wmEnqJ2Q@mail.gmail.com> In-Reply-To: <2980696.6AEyEjetGn@ralph.baldwin.cx> References: <1730061.8Ii36ORVKt@ralph.baldwin.cx> <2980696.6AEyEjetGn@ralph.baldwin.cx>
next in thread | previous in thread | raw e-mail | index | archive | help
So none of these changes have been committed yet? I'm hitting hangs in USB on boot with recent HEAD and without having investigating had thought this might be what exposed the problem. Thanks. -M On Friday, March 18, 2016, John Baldwin <jhb@freebsd.org> wrote: > On Tuesday, February 16, 2016 12:50:22 PM John Baldwin wrote: > > Currently the kernel bootstraps the non-boot processors fairly early in > the > > SI_SUB_CPU SYSINIT. The APs then spin waiting to be "released". We > currently > > release the APs as one of the last steps at SI_SUB_SMP. On the one hand > this > > removes much of the need for synchronization while SYSINITs are running > since > > SYSINITs basically assume they are single-threaded. However, it also > enforces > > some odd quirks. Several places that deal with per-CPU resources have to > > split initialization up so that the BSP init happens in one SYSINIT and > the > > initialization of the APs happens in a second SYSINIT at SI_SUB_SMP. > > > > Another issue that is becoming more prominent on x86 (and probably will > also > > affect other platforms if it isn't already) is that to support working > > interrupts for interrupt config hooks we bind all interrupts to the BSP > during > > boot and only distribute them among other CPUs near the end at > SI_SUB_SMP. > > This is especially problematic with drivers for modern hardware > allocating > > num(CPUs) interrupts (hoping to use one per CPU). On x86 we have aboug > 190 > > IDT vectors available for device interrupts, so in theory we should be > able to > > tolerate a lot of drivers doing this (e.g. 60 drivers could allocate 3 > > interrupts for every CPU and we should still be fine). However, if you > have, > > say, 32 cores in a system, then you can only handle about 5 drivers doing > > this before you run out of vectors on CPU 0. > > > > Longer term we would also like to eventually have most drivers attach in > the > > same environment during boot as during post-boot. Right now post-boot is > > quite different as all CPUs are running, interrupts work, etc. One of > the > > goals of multipass support for new-bus is to help us get there by probing > > enough hardware to get timers working and starting the scheduler before > > probing the rest of the devices. That goal isn't quite realized yet. > > > > However, we can run a slightly simpler version of our scheduler before > > timers are working. In fact, sleep/wakeup work just fine fairly early > (we > > allocate the necessary structures at SI_SUB_KMEM which is before the APs > > are even started). Once idle threads are created and ready we could in > > theory let the APs startup and run other threads. You just don't have > working > > timeouts. OTOH, you can sort of simulate timeouts if you modify the > scheduler > > to yield the CPU instead of blocking the thread for a sleep with a > timeout. > > The effect would be for threads that do sleeps with a timeout to fall > back to > > polling before timers are working. In practice, all of the early kernel > > threads use sleeps without timeouts when idle so this doesn't really > matter. > > After some more testing, I've simplified the early scheduler a bit. It no > longer tries to simulate timeouts by just keeping the thread runnable. > Instead, > a sleep with a timeout just panics. However, it does still permit sleeps > with > infinite sleeps. Some code that uses a timeout really wants a timeout > (note > that pause() has a hack to fallback to DELAY() internally if cold is true > for > this reason). Instead, my feeling is that any kthreads that need timeouts > to > work need to defer their startup until SI_SUB_KICK_SCHEDULER. > > > However, I'd like feedback on the general idea and if it is acceptable > I'd > > like to coordinate testing with other platforms so this can go into the > > tree. > > I don't think I've seen any objections? This does need more testing. I > will > update the patch to add a new EARLY_AP_STARTUP kernel option so this can be > committed (but not yet enabled) allowing for easier testing (and allowing > other platforms to catch up to x86). > > > The current changes are in the 'ap_startup' branch at > github/bsdjhb/freebsd. > > You can view them here: > > > > https://github.com/bsdjhb/freebsd/compare/master...bsdjhb:ap_startup > > -- > John Baldwin > _______________________________________________ > freebsd-arch@freebsd.org <javascript:;> mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org > <javascript:;>" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAHM0Q_P_DZg9XpuP0ZRj0nBFhPxUK1VZ8SuEuaB4r9wmEnqJ2Q>