Date: Wed, 1 Dec 2004 04:47:51 -0500 (EST) From: "Ketrien I. Saihr-Kenchedra" <ketrien@error404.nls.net> To: "David O'Brien" <obrien@freebsd.org> Cc: freebsd-amd64@freebsd.org Subject: Re: kernel panic with greater that 8 GB of memory Message-ID: <20041201042900.A88450@bahre.achedra.org> In-Reply-To: <20041201084537.GA1621@dragon.nuxi.com> References: <20041129211341.GA26548@troutmask.apl.washington.edu> <20041130183555.GA32237@troutmask.apl.washington.edu> <20041201084537.GA1621@dragon.nuxi.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, 1 Dec 2004, David O'Brien wrote: > There is no 'ccNUMA' setting in the BIOS. A multi-processor Opteron is a > NUMA architecture machine regardless of any BIOS settings. I.E. there is > no way to disable that a MP Opteron is a NUMA machine. > The setting of interest is should the BIOS round-robin interleave > physical addresses across the NUMA nodes[*]. The reference AMI BIOS > refers to this as "Node Interleaving". It should be "DISABLED". Or if > the BIOS speaks of the SRAT table, it should be "ENABLED". While FreeBSD > doesn't use the SRAT table (and cannot until ACPI 3.0 BIOS's); turning on > the SRAT turns off node interleaving. David, trying very hard to be nice, the S2882's a 'Special Case.' Where special should be taken as 'short bus.' VERY short bus. On the S2882/S2885, and even the S4882, the BIOS specifically says, and I quote: 'ccNUMA Support.' No joke. The beauty is that ccNUMA is, you got it, SRAT Table Control, which _disables_ interleave completely. Beautiful, huh? The only way to enable interleave on the S2882/S2885 is to specifically turn 'ccNUMA' off; otherwise it's SRAT only, with no interleave. And no, it's not mentioned anywhere on the S2882s. Only on the S4882, which uses a Phoenix 6.0 reference. (Which Tyan, predictably, did not remove the reference lines _or_ fix the typos on.) The S2882s also use a Phoenix, I believe. So in order to enable Interleave, your only method is that switch. What got me peering curiously here is the placement of the hole; there appears to be a 512MB hole, starting at 4024MB. It certainly is similar to the behavior of 2882 I've looked at; hole is about 1/3rd through memory. On the 2.03 BIOS, still no way to control the PCI hole; that feature is reserved for the S4882 apparently, and even then only size. If the hole _is_ at 4024MB, that would put it past 4096MB and the 4GB limit; on the S2882 the bulk of the peripherals are on the PCI32 bus, not a PCI64 bus. Specifically, disk, fxp(4), bge(4), ACPI, and USB. Presuming I am reading the hole correctly, and bearing in mind that I do not have access to my copy of the PCI2.3 spec, ISTR that the entirety of the PCI memory hole must be within the first 4GB of system memory. (I'd appreciate it if someone could sanity check that.) Presuming that it does need to be within the first 4GB of system memory, I'm wondering if what's being seen here is not a BIOS bug. These boards have a real bad track record with regards to them, so it would not be at all surprising, at least to me. -ksaihr
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20041201042900.A88450>