Date: Fri, 13 Sep 1996 00:16:18 -0700 From: erich@uruk.org To: Steve Passe <smp@csn.net> Cc: terry@lambert.org, peter@spinner.dialix.com, rv@groa.uct.ac.za, freebsd-smp@freebsd.org Subject: Re: (long) Intel SMP info (was -> Re: Intel XXpress - some SMP benchmarks) Message-ID: <199609130716.AAA14688@uruk.org> In-Reply-To: Your message of "Thu, 12 Sep 1996 17:07:15 MDT." <199609122307.RAA20132@clem.systemsix.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Steve Passe <smp@csn.net> writes: > > Also realize that when a processor goes through INIT, the APIC id will be > > reset to the hardware id. This means if your OS is in the middle of doing > > pentium processor manual vol3, 19.3.1.11.3 'Initialization Reset (INIT)': > > "INIT is a software reset, and is delivered as a bus message. INIT has > the same effect on the Local APIC as the power-up Reset, except that the > APIC ID and the Arb ID registers are not affected." This is *different* for different parts. *don't* rely on it not resetting the APIC state! NOTE that the "Pentium" manual refers to the Pentium APIC. The 82489DX manual is necessary for other important information. > Everyone please bear with me here, this gets ugly, > but I think we are getting close to resolution. > > --- > > It's not a problem of the first STARTUP being "missed", the problem > > (which is only present on the Pentium integrated APIC) is as follows: > > When the Pentium is in the state waiting for a STARTUP, an INIT > > "in the state waiting for a STARTUP", is this a state reached by hardware > RESET? or perhaps I'm asking what this state is. Is it as if a 'hlt' > instruction had been executed? It is the state after an INIT, when the Pentium or Pentium Pro thinks it is not the "boot processor". The essential issue is that you *don't* *know* what state it is in!!! It might have been intercepted by the BIOS at boot time, it might be using an architecture where the "secondary processor" internal state bit isn't being set, etc... This allowed much more flexibility for them to built fault-tolerant hardware based on the APIC stuff built into the Pentium CPU (it was really designed to minimize hardware for the 2-CPU case). The Pentium Pro is again much more flexible along these lines. > > doesn't get processed, but it is latched. The STARTUP then allows the > > INIT to be executed, which effectively kills the STARTUP message (this is > > where the time-delays in the recommended INIT/STARTUP/STARTUP process > > come from, to allow the CPU to finish the INIT process), so you need > > another STARTUP message. > > so if I understand this we have: > > 2nd CPU is waiting for a STARTUP IPI > 1st CPU does INIT, > 1st CPU waits for INIT > to run > 2nd CPU latches INIT IPI, DOESN'T process it yet. > 1st CPU finishes wait > 1st CPU does STARTUP IPI > 2nd CPU catches STARTUP IPI, > starts to vector thru supplied STARTUP vector > 2nd CPU immediately catches INIT IPI, > aborting STARTUP IPI Up to here is correct. The real finish is: 1st CPU send second STARTUP IPI 2nd CPU responds as it is supposed to, and jumps to startup vector. That sequence is only valid for when the "secondary processor" flag is set. In many cases (such as with the XXPRESS), this flag isn't set, and an INIT is REALLY a RESET, and it doesn't halt in the weird state waiting for the STARTUP IPI. In theory, with the XXPRESS, all you have to do is set the warm-boot vector and send it an INIT, so you following comments should be valid: > INIT causes vector thru warm-boot to bootMP() > bootMP() gets 2nd CPU initialized and running Again, it's simpler to just use the working sequence in all instances. > to get the XXPRESS box working I HAD to preceed the INIT IPI with > a setup of the BIOS warm-start vector to the bootMP() code. Yes, the XXPRESS architecture doesn't use the "secondary process" flags at all (again, fault tolerance stuff... so you can yank any CPU and have it work). > pointing it at a 'hlt' instruction, then allowing the STARTUP IPI to > provide the vector to bootMP() fails. so it would appear that BOTH > of the STARTUP IPIs (both vectoring to bootMP()) fail and/or are > ignored by the XXPRESS. I'm sure (he says with confidence) that the INIT > IPI actually travels thru the vector as we replaced the 'hlt' instruction > with an NMI and we got spontanious reboot. See above. > > The problem with trying to just do an INIT, then if that doesn't work, a > > STARTUP and INIT, is that you're never sure what the state the BIOS > > puts CPUs in on a lot of the Pentium boxes (you'd also have to wait > > to see if it responded to each part of the sequence). I spent a lot of > > time finding out that the ENTIRE sequence was required on even machines > > that I thought didn't need it (the XXPress box). The Intel folks > > spent a lot of time generating a single guaranteed sequence that > > covered all the bases. > > I'm getting a headache... SO at this point I think I buy the double > INIT IPI followed by 2 STARTUP IPS, with appropriate timings in-between > as the cure for all hardware, in all the "brokenness" we are likely to find. -- set warm-boot vector -- single INIT IPI -- wait a bit -- single STARTUP IPI -- wait a bit -- single STARTUP IPI -- wait for CPU's response. > I'm just not clear on the appropriate thing to do with the warm-start vector. > I believe its needed, and that the XXPRESS for one needs it to point to > the actual 2nd CPU boot code. Can we expect to be able to vector to > the boot code on all hardware, or will we encounter boxes that run the > boot code via the INIT IPI warm-start vector, then re-run it via one > of the following STARTUP IPIs??????? Some use the warm-boot vector, and some the vector from the STARTUP IPIs. All Pentium Pro machines I've seen, for example, use the STARTUP IPI vector. > is there a definitive code sample that encompasses all this. in the > MP spec they have the pseudo code with 2 STARTUP IPIs, but no actual code. Pretty much. The Linux-SMP startup code, or the code in my bootloader GRUB, are pretty universal. I'm still tweaking the one I'm going to place on my web site as a "canonical" startup sequence. I found some definition issues that won't likely be seen in practice, but are technically wrong with the Linux and FreeBSD startup codes. > the pentium manual shows some real code with timings, but NO double > STARTUP sequence.... The pentium manual is out-of-date, and doesn't include what in the MP spec 1.4 is really an algorithm that is a workaround to account for all the problems that have been encountered. Use the 1.4 spec!!!! Don't even refer to the 1.1 spec, as some parts of it (for example the bus hierarchy stuff) is simply inconsistent. -- Erich Stefan Boleyn \_ E-mail (preferred): <erich@uruk.org> Mad Genius wanna-be, CyberMuffin \__ (finger me for other stats) Web: http://www.uruk.org/~erich/ Motto: "I'll live forever or die trying"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199609130716.AAA14688>