Date: Mon, 22 Sep 2014 22:35:53 -0700 From: Mark Millard <markmi@dsl-only.net> To: Nathan Whitehorn <nwhitehorn@freebsd.org> Cc: Justin Hibbits <chmeeedalf@gmail.com>, FreeBSD PowerPC ML <freebsd-ppc@freebsd.org> Subject: Re: powerpc64/GENERIC64 use of dcbst vs. dcbf: is the dcbst use really okay? Anyone know? Message-ID: <9C7E071E-2547-47F9-98B3-54F4908C4932@dsl-only.net> In-Reply-To: <5420FD56.2020003@freebsd.org> References: <B3F99C14-ABCF-4F35-B0C6-34B4E5315BAB@dsl-only.net> <5420FD56.2020003@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
I keep seeing if I can get more evidence of what is going on for the = PowerMac G5 boot-hang problem. (I'd like full memory with reliable boots = some day...) In the process I look around and research what I see. And that can lead to such questions even if the question formed is not a = possible explanation of a contribution to the G5 hangs. In this case experiments had shown that dcbf's did not change the = boot-hang problem. But I still had the question. I gather from your explanation that those 4 places that use dcbst do not = have issues with processors fetching "without coherence being enforced". = .__syncicache seemed to be for more general use then the others and so = seemed less likely to have special criteria that might always apply. = That is part of what prompted me to ask: analyzing all the usage to = prove such properties was more than I wanted to take on. Side notes: For Apple's BootX-81: if/when the kernel returns BootX-81 does its own = restore of the openfirmware exception vectors before returning to = openfirmware (returning a -1). So they seem to not presume openfirmware = itself does so in that place. (More paranoia code?) One point of paranoia Apple did not follow in BootX-81: They have dcbf = then the matching icbi with nothing between and do all of the pairs of = those instruction before doing either just one "isync; sync; eieio" = (going into the kernel) or just one "sync; mtmsr ...; isync" on return = from the kernel (kernel exit). =3D=3D=3D Mark Millard markmi at dsl-only.net On Sep 22, 2014, at 9:55 PM, Nathan Whitehorn <nwhitehorn@freebsd.org> = wrote: On 09/22/14 17:18, Mark Millard wrote: > Anyone know why the following is true in FreeBSD (10.1-BETA2, for = example) for kernel vs. openfirmware transitions (in both directions) = for powerpc64/GENERIC64? (And some other places are noted.) The issue is = dcbst vs. dcbf instruction usage. >=20 > (I later quote from pem_64_bit_v3.0.2005jul2005.pdf (from IBM).) Yes, a mix is used. It's done deliberately. dcbf implies dcbst plus a = cache flush, but there is in general no need for an invalidation where = dcbst is used in the kernel. > Some context first... >=20 > Apple's published BootX-81 always saves and restored the Exception = Vectors when going between openfirmware and the kernel: it maintains = separate vectors for the two contexts. In addition it carefully uses = dcbf and icbi no matter if copies to that area at address 0 or to a save = area. And that is followed by isync. (And more, sync and eieio: Apple = seems paranoid.) There's a lot of paranoia that is useful to attach to this process. Open = Firmware is supposed to restore its own exception vectors (and does, in = general) if it needs to. We could imagine doing that too. Doing that = would probably involve a great deal of work implementing the OF callback = infrastructure that we don't currently support, however. Is there a = reason you think the exception vectors are causing a problem? -Nathan > Apple used dcbf instead of dcbst. >=20 > IBM writes of dcbst vs. dcbf: >=20 >> Instruction caches, if they exist, are not required to be consistent = with data caches, memory, or I/O data trans- fers. Software must use the = appropriate cache management instructions to ensure that instruction = caches are kept coherent when instructions are modified by the processor = or by input data transfer. When a processor alters a memory location = that may be contained in an instruction cache, software must ensure that = updates to memory are visible to the instruction fetching mechanism. = Although the instructions to enforce consistency vary among = implementations, the following sequence for a uniprocessor system is = typical:=20 >> 1. dcbst (update memory) >> 2. sync (wait for update) >> 3. icbi (invalidate copy in instruction cache) 4. isync (perform = context synchronization)=20 >> Note: Most operating systems will provide a system service for this = function. These operations are neces- sary because the memory may be = designated as write-back. Since instruction fetching may bypass the data = cache, changes made to items in the data cache may not otherwise be = reflected in memory until after the instruction fetch completes.=20 >> For implementations used in multiprocessor systems, variations on = this sequence may be recommended. For example, in a multiprocessor = system with a unified instruction/data cache (at any level), if = instructions are fetched without coherency being enforced, the preceding = instruction sequence is inadequate. Because the icbi instruction does = not invalidate blocks in a unified cache, a dcbf instruction should be = used instead of a dcbst instruction for this case. >>=20 >=20 > Then the point given that background information... >=20 > FreeBSD's powerpc64/GENERIC64 seems to have a mix of dcbst and dcbf = use. The following have dcbst (unless patched separately at run time): >=20 > 000000000086c1e8 <.agp_apple_unbind_page+0x60> dcbst r0,r0 >=20 > 000000000086c27c <.agp_apple_bind_page+0x64> dcbst r0,r0 >=20 > 00000000008b1b78 <.elf_reloc_internal+0x12c> dcbst r0,r30 >=20 > 00000000008bcd30 <.__syncicache+0x38> dcbst r0,r0 >=20 > That last is used during the openfirmware vs. kernel transitions. The = above are from "objdump -d --prefix-address /boot/kernel/kernel". >=20 > Is the dcbst use risky because of any unified caches at any level on = any of the processors that powerpc64/GENERIC64 is supposed to handle? >=20 >=20 > =3D=3D=3D > Mark Millard > markmi at dsl-only.net >=20
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9C7E071E-2547-47F9-98B3-54F4908C4932>