Date: Thu, 12 Sep 1996 14:13:44 -0700 From: erich@uruk.org To: Terry Lambert <terry@lambert.org>, smp@csn.net (Steve Passe) Cc: peter@spinner.dialix.com, rv@groa.uct.ac.za, freebsd-smp@freebsd.org Subject: (long) Intel SMP info (was -> Re: Intel XXpress - some SMP benchmarks) Message-ID: <199609122113.OAA13651@uruk.org> In-Reply-To: Your message of "Thu, 12 Sep 1996 11:12:53 PDT." <199609121812.LAA07189@phaeton.artisoft.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Terry Lambert <terry@lambert.org> writes: > > > Hint: ALL the APIC ID registers are read/write. > > > > > > Can you see it yet? :-) > > > > we tried that on the XXPRESS and (Russel, please confirm this) an instant > > reset of the hardware. > > Any chance that a write of the ID register acts as an INIT IPI? That's > what seems to be implied. This is not the case. In particular a RESET and INIT are NOT the same thing. I'm not sure what the problem you're having is (I'll try it tonight), but a known bug with the APIC is that it requires all reads and writes to be 32-bit aligned accesses (or weird things can happen). > I suspect that you will need to inventory the processors, then back-fill > the holes for the case where you would get an ID collision during the > shuffling -- ie: if I have n processors, all APIC ID's < (n-1) are left > alone, and only the remainder are rewritten. > > I *believe* that the BP is guranteed an APIC ID of 0. No. The rules are: "default configuration": CPUs must be numbered consecutively starting with 0, in any order. configurations defined in the MP tables: at least one CPU must have id #0, and a vague warning that if you use id numbers too large for other CPUs then the addressing mechanism won't be compatible with all versions of the APIC (i.e. dont use anything more than an 8-bit address can find). The above rules (numbering is arbitrary) have to be following by the OS, or many machines simply aren't going to work. Many Pentium Pro machines using 2 CPUs have the boot CPU as APIC id #1. Also realize that when a processor goes through INIT, the APIC id will be reset to the hardware id. This means if your OS is in the middle of doing a bunch of things and then sends an INIT to one of the CPUs, you have a big potential problem there. To solve this, you either don't reassign APIC ids, or never send an INIT when the system is up and running. All this stuff is in the MPS spec 1.4 and Pentium errata (the 32-bit access thing might not be, I forget if it is a Pentium Pro issue). > You may want to disassemble your MP cold boot BIOS code to see about > the ID assignment; clearly it must be happening in BIOS in any case, > since the PPRO's are "glueless" and would care about which slots they > are put in, otherwise. Quite a bit is determined by the hardware slot it is in. I'm pretty sure that the Pentium and discrete APICs were hard-wired based on wires on the board. I think even the Pentium Pros are, but this is based on a set of wires in the bus protocol in their case, that's why it's "glueless". In a later message, Terry Lambert wrote: > Oh, ugh. Sequent supports 32 processors. I wonder how? I'm pretty sure they don't use anything like the Intel SMP architecture. Steve Passe <smp@csn.net> writes: > >BTW, Some other things we do not do.. We don't set the ERROR LVT to handle > >a non-delivered or failed message. > no, but my original code for the apic_startup() had checks on the APIC_ESR > register, never saw errors. I will put that back later today. Before your operation, write a zero to the APIC_ESR register (I think a read before the write is even necessary... I'll dredge up my notes which tried to make sense of all the eratta sometime tonight or tomorrow). > >I noticed you took out the second STARTUP IPI.. The docs I've been > >reading say "the startup IPI can only be used once after a reset or INIT", > > ... > >the second one is for insurance in case the first one was missed, and that > >the second will normally be ignored. I notice that there is no way in the > > I guess the second one couldn't hurt, but I would rather use a better > means than "if I do it often enough its gotta work". A greater concern > is the INIT/RESET of the 'run bootMP' flavor that the XXPRESS demanded. > If we add the correct timings whats to prevent the STARTUP IPI from > re-running a CPU once it has already started (via RESET), perhaps > double incrementing mp_ncpus? It's not a problem of the first STARTUP being "missed", the problem (which is only present on the Pentium integrated APIC) is as follows: When the Pentium is in the state waiting for a STARTUP, an INIT doesn't get processed, but it is latched. The STARTUP then allows the INIT to be executed, which effectively kills the STARTUP message (this is where the time-delays in the recommended INIT/STARTUP/STARTUP process come from, to allow the CPU to finish the INIT process), so you need another STARTUP message. The problem with trying to just do an INIT, then if that doesn't work, a STARTUP and INIT, is that you're never sure what the state the BIOS puts CPUs in on a lot of the Pentium boxes (you'd also have to wait to see if it responded to each part of the sequence). I spent a lot of time finding out that the ENTIRE sequence was required on even machines that I thought didn't need it (the XXPress box). The Intel folks spent a lot of time generating a single guaranteed sequence that covered all the bases. It's best to simply use the recommended startup sequence and leave it at that, if you wish to support Pentium SMP boxes reliably (again, Pentium Pros dont need this). > >The pppbios.pdf specifically says: "The BSP sends a StartUp APIC message > >broadcast......" One of the various other tables in the P5 docs say that > >startup IPI broadcasts are always edge triggered when used in "all but > >self" mode, so who knows.. :-) > > I see that now, I'm willing to believe it might be doable. I'm not quite sure what you're saying here. If this is from the SMP bootup sections of the various manuals, I remember it talking a lot about such things that were entirely referring to inter-APIC messages done as part of the APIC bus protocol which had no bearing on how the OS was to do CPU startup/etc. > >Another thought.. We do not use the timer on the apic. It has a 32 bit > >read/write register for the "initial count". We could cheat and use that > >as a 32 bit pointer to a cpu-specific data page with each cpu's scratch > >area etc, > > are you certain that we won't want to use it in the future? > another issue is that erich claims accessing the APIC registers is relatively > expensive time-wise. Yes. An APIC read/write is an *uncached* access, which is prety slow. The reason for this is that the APIC is effectively a hardware device, and you're working with it's control ports. I suppose it could have been optimized further, but other things accessed much more often took precidence. On the Pentium Pro, an uncached access has to drain many of the bus pipelines, and is not terribly fast. An L2 hit will certainly be a heck of a lot faster (an order of magnitude in some cases), no matter what CPU you use. Basically, get the APIC id, then calculate everything from there, even if you need to do a small table lookup, it's still going to be faster. > > better off putting in a check for an 82489 apic in the boot code and > > both mine and Russel's boards show an IO APIC version of 17. His CPUs > show APIC vewrsion 16, while mine shows 17 (all numbers taken from the > MP table, NOT the APIC version registers). I can't find a # for the > 82379AB anywhere in its manual. An integrated APIC (or more to the point, any APIC with support for the STARTUP IPI) will have a version of 16 or greater. > >From the manuals: > > 82093AA: 11 > 82379AB: ?? > 82489DX: 01 > > P5: 1x > P6: ?? >From the manual I have, the P6 is hex 11. .... > fixes that. still have a problem in that the makefiles for biosboot etc. > don't search the right path, ie opt_smp is only going to be in the kernel > compile specific directory, is it not? THis only worked because I copied > opt_smp.h to i386/include. Interesting, is it coincidence, or does the biosboot actually need to know about SMP information? -- Erich Stefan Boleyn \_ E-mail (preferred): <erich@uruk.org> Mad Genius wanna-be, CyberMuffin \__ (finger me for other stats) Web: http://www.uruk.org/~erich/ Motto: "I'll live forever or die trying"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199609122113.OAA13651>