Date: Sat, 02 Mar 2013 10:50:34 -0700 From: Ian Lepore <ian@FreeBSD.org> To: Tim Kientzle <tim@kientzle.com> Cc: freebsd-arm@FreeBSD.org Subject: Re: PHYSADDR Message-ID: <1362246634.1195.178.camel@revolution.hippie.lan> In-Reply-To: <5B622D1B-4EAE-4184-A194-DD14083A48B6@kientzle.com> References: <E886046B-1612-425B-902B-72D4B0E93618@freebsd.org> <1362068453.1195.40.camel@revolution.hippie.lan> <674A08B3-6600-4B77-8511-9EF54E4B9B1F@FreeBSD.org> <8FEA3237-8ABF-4564-B672-4B4C0C6EF291@kientzle.com> <1362155632.1195.120.camel@revolution.hippie.lan> <5B622D1B-4EAE-4184-A194-DD14083A48B6@kientzle.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, 2013-03-01 at 09:19 -0800, Tim Kientzle wrote: > On Mar 1, 2013, at 8:33 AM, Ian Lepore wrote: >=20 > > On Thu, 2013-02-28 at 09:44 -0800, Tim Kientzle wrote: > >> On Feb 28, 2013, at 8:58 AM, Tim Kientzle wrote: > >>=20 > >>>=20 > >>> On Feb 28, 2013, at 8:20 AM, Ian Lepore wrote: > >>>=20 > >>>> On Wed, 2013-02-27 at 22:27 -0800, Tim Kientzle wrote: > >>>>> Starting to look at what is needed for a Generic ARM kernel. > >>>>> There's a lot here; I sincerely hope I'm not the only one=85 ;-) > >>>>>=20 > >>>>> First up: Can we get rid of PHYSADDR? > >>>>>=20 > >>>>=20 > >>>> If you mean, can we get rid of it within the runtime kernel, I'd s= ay > >>>> yes, because we can use a global variable instead which is easily > >>>> settable in the entry code. > >>>=20 > >>> It doesn't seem to be used in the runtime kernel. As far as > >>> I can see, it's purely a bootstrap concept. > >>>=20 > >=20 > > Well, it's used to set up the early-init page tables in locore.s then > > again to set up the real page tables and related things in initarm() = and > > then I think it isn't used after that, so I should have said "within = the > > kernel init". The main point I was getting at is that I don't think = we > > need a compile-time constant of any sort related to the physical addr= ess > > at which the kernel is loaded. We can get the physical address of th= e > > entry point (_start) using pc-relative math, and we can get the linke= r > > to give us a constant symbol which is the offset of the _start symbol > > within the load image. So we can get the load address at runtime > > without guessing what low-order bits to mask. >=20 > Good. That's the approach I've been eyeing as well; I'm > glad you agree that makes sense. >=20 > >=20 > >>>> On the other hand, I've been working > >>>> towards getting that value set correctly in the kernel elf headers= at > >>>> link time. > >>=20 > >> A-ha! I think I just figured something out. > >>=20 > >> How would the following work: > >>=20 > >> * Rename PHYSADDR to KERNPHYSADDR_BASE > >>=20 > >> * in the top of locore.s, we have a single conditional: > >>=20 > >> #ifdef KERNPHYSADDR_BASE > >> _kpa_base =3D KERNPHYSADDR_BASE; > >> #else > >> _kpa_base =3D pc & 0xF0000000; > >> #endif > >>=20 > >> I think this would DTRT on all of the configurations > >> we currently have in SVN. > >=20 > > Hmm, so the basic assumption is that every SoC will have some physica= l > > memory aligned to a 256mb boundary. >=20 > I'm assuming this only for ARM systems supported by the GENERIC > kernel. >=20 > Ss far as I can tell, the 256mb boundary assumption works for > everything currently in SVN. But the code above does allow you > to custom-build a kernel for a system that doesn't satisfy that; > you just won't be able to run the GENERIC kernel there. >=20 > Eventually, it seems we might pull this information out of the > FDT, but I'm not yet ready to tackle FDT parsing in locore.S. ;-) >=20 > Of course, I'm not certain that it will matter when we're done. > If we only need PHYSADDR to set up the MMU paging, > then we just need to round the _start address down to > the next page boundary, don't we? >=20 > > E.G., there'll never be a SoC with > > memory at 0xN1000000 that doesn't have memory at 0xN0000000. I'm not > > sure that's a safe assumption given things like the rpi where the gpu > > carves off some memory for itself and gives the rest to the arm. It > > works with the way rpi carves up the ram, but I could see similar > > designs that wouldn't work. > >=20 > >>=20 > >> * We redefine KERNPHYSADDR to be an *offset* > >> against _kpa_base. Then we could negotiate a single > >> offset (64k?) that works well on many platforms and use > >> that for the GENERIC kernel. Boot loaders would be > >> responsible for loading the kernel at an address that > >> preserves the KPA_OFFSET. The KPA_OFFSET would > >> be used in the ELF headers. > >>=20 > >> Then there are routine code transformations to use _kpa_base > >> instead of the compile-time symbol and to use > >> _kpa_base + KERNPHYSADDR instead of KERNPHYSADDR. > >>=20 > >> Does this sound reasonable as a starting point? > >>=20 > >=20 > > There are even more assumptions here about what would work in every > > case. Given the basic sequence of boot2->u-boot->ubldr->kernel, ever= y > > one of those components has to load the next component at the physica= l > > address it's linked for, and that has to not overlap the addresses be= ing > > used by the thing doing the loading. The MMU is off during all of th= is > > so we can't just map our way out of any conflicts. >=20 > I've given up entirely on the first two of these being generic. >=20 > I think we have a shot at making the kernel itself generic, > and maybe ubldr could be made truly PIC, but the earlier > boot stages are always going to be highly board-specific. >=20 > So conflicts between the various pieces aren't really > my primary worry at the moment. Since we'll have to > customize the early boot pieces anyway, we can resolve > those on a case-by-case basis. >=20 > The ELF load address vs. where physical memory is located > seems the sticky point. Any ideas for getting around that? > I feel like I have enough ideas to start chipping away at > locore.S if I could just figure out a strategy for the ELF > load address issue. >=20 > (Of course, I still don't understand why the test image I've > been playing with seems able to load the same ubldr on > both RPi and BBone and that ubldr seems to have no trouble > loading a kernel into memory.) We may not have control over anything before ubldr. Not everything is an eval board that you can easily build a custom u-boot for. =20 I'm not sure its safe to assume that (entry-pc & 0xfffff000) is the beginning of the kernel; it's true now but need not be so. But that's no big deal, we can tweak the linker script to give us the offset of the _start symbol so it'll work no matter what. The more I ponder the fixed offset concept for a generic kernel the more it seems that it might be viable, providing that we require the use of ubldr. Then we can make sure that ubldr is always linked at an offset that doesn't overlap the kernel load offset. An offset number like 1mb or 4mb might work well for the kernel; it leaves lots of space for ubldr to be loaded down low in memory. I think putting the loader at a lower address than the kernel is required, because there's no upper bound on the kernel size (I did a 27MB kernel last month -- it contains a huge embedded md filesystem image). This just pushes the real problem into ubldr, though, and that becomes the non-generic component that has to be linked at a different physical address for each SoC. A kernel could still be loaded without ubldr, but it may need to be built in a non-generic way for some platforms. So if we declare that this scheme is for generic kernels loaded by a loader (ubldr or other) that is aware of the generic kernel scheme, there's no need for the physical address fields in the elf headers to be interpretted as a real physical address that would work for a standard elf loader. You can build kernels that work with a standard elf loader, but the generic kernel is not such a thing. =20 In that case, the physical address and entry address fields in the headers are all offsets. If physical ram on a given chip starts at zero, then the headers would accidentally be right for a standard elf loader. Otherwise the generic-aware loader is responsible for filling in some high-order bits when interpretting the headers. The PHYSADDR symbol (and its several name variations) at build time now becomes zero for a generic kernel or non-zero to generate a SoC-specific kernel that can be loaded by a standard elf loader. The symbol doesn't exist at all in the compilation, it's used only by the build system and is passed as a parameter to the linker. There's another problem constant we haven't talked about yet: STARTUP_PAGETABLE_ADDR, used by locore to build the early page tables. It's the absolute physical address of an area of memory that doesn't have any other important data in it, and won't get overwritten by something before initarm() builds the real page tables. I think most SoCs now put it way up high in memory "safely out of the way", but that won't work generically. We need 16K aligned to a 16K boundary, and maybe it would work to use the area immediately preceeding the start of the kernel. We could require the kernel to be linked on a 16k boundary so that we can just blindly subtract 16K from the starting physaddr we calculate; nice and easy. -- Ian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1362246634.1195.178.camel>