Date: Tue, 30 Jul 2019 17:01:57 +0300 From: Toomas Soome <tsoome@me.com> To: "O. Hartmann" <o.hartmann@walstatt.org> Cc: "O. Hartmann" <ohartmann@walstatt.org>, "freebsd-embedded@freebsd.org" <embedded@freebsd.org>, FreeBSD CURRENT <freebsd-current@freebsd.org> Subject: Re: PCengines APU2C4, 12-STABLE: bootloader failure: Panic: free: guard2 fail @ 0x1000 + 2311663946 from Message-ID: <D21DC702-CB58-4AAB-A010-20D0AFECD720@me.com> In-Reply-To: <20190730144331.24f83e0c@freyja> References: <20190724101132.5560377a@thor.intern.walstatt.dynvpn.de> <B7BF6016-E4B1-4B34-9C22-CC7CD789FA79@me.com> <20190724120720.4382397c@thor.intern.walstatt.dynvpn.de> <20190724154901.0c114ebe@thor.intern.walstatt.dynvpn.de> <93EFC481-50A1-491C-87BB-12A35F3302EE@me.com> <20190730144331.24f83e0c@freyja>
next in thread | previous in thread | raw e-mail | index | archive | help
> On 30 Jul 2019, at 15:43, O. Hartmann <o.hartmann@walstatt.org> wrote: >=20 > On Wed, 24 Jul 2019 18:07:22 +0300 > Toomas Soome <tsoome@me.com> wrote: >=20 >>> On 24 Jul 2019, at 16:48, O. Hartmann <ohartmann@walstatt.org> = wrote: >>>=20 >>> -----BEGIN PGP SIGNED MESSAGE----- >>> Hash: SHA256 >>>=20 >>> Am Wed, 24 Jul 2019 12:06:53 +0200 >>> "O. Hartmann" <o.hartmann@walstatt.org = <mailto:o.hartmann@walstatt.org>> >>> schrieb:=20 >>>> -----BEGIN PGP SIGNED MESSAGE----- >>>> Hash: SHA256 >>>>=20 >>>> Am Wed, 24 Jul 2019 12:09:16 +0300 >>>> Toomas Soome <tsoome@me.com> schrieb: >>>>=20 >>>>>> On 24 Jul 2019, at 11:11, O. Hartmann <ohartmann@walstatt.org> = wrote: >>>>>>=20 >>>>>> -----BEGIN PGP SIGNED MESSAGE----- >>>>>> Hash: SHA256 >>>>>>=20 >>>>>> Hallo, >>>>>>=20 >>>>>> on APU2C4 from PCengines with latest firmware apu2_v4.9.0.7.rom, = SeaBIOS >>>>>> rel-1.12.1.3-0-g300e8b7, booting via legacy MBR FreeBSD 12-STABLE >>>>>> r350274 (the same with r350115) fails to boot with an immediate = loader >>>>>> error: >>>>>>=20 >>>>>> [...] >>>>>> SeaBIOS (version rel-1.12.1.3-0-g300e8b7) >>>>>>=20 >>>>>> Press F10 key now for boot menu >>>>>>=20 >>>>>> Booting from Hard Disk... >>>>>> / >>>>>>=20 >>>>>> onsoles: internal video/keyboard =20 >>>>>> IOS drive C: is disk0=20 >>>>>> IOS drive D: is disk1=20 >>>>>> IOS 639kB/3404444kB available memory=20 >>>>>>=20 >>>>>> reeBSD/x86 bootstrap loader, Revision 1.1 =20 >>>>>> Mon Apr 15 21:28:11 CEST 2019 root@thor)=20 >>>>>> anic: free: guard2 fail @ 0x1000 + 2311663946 from >>>>>> X=C3=A7u0=C3=A7}4=C3=A7l$=E2=99=A6=E2=94=9C=C3=AD@=E2=94=A4=E2=99=A0= :2106163957 -> Press a key on the console to reboot >>>>>> <-- [=E2=80=A6] =20 >>>>>=20 >>>>>=20 >>>>> This is definitely something =E2=80=9Cfunny=E2=80=9D, we are = apparently attempting to >>>>> free pointer 0x1000 which is definitely wrong because our heap = should be >>>>> just below 4GB line. Since we do get list of disks printed, also = memory >>>>> and version, it means we get error from interpretator - it is = possible >>>>> the stack did clash with bss and hence the corruption. =20 >>>>=20 >>>> I realized that I have defined=20 >>>>=20 >>>> WITH_KERNEL_RETPOLINE=3DYES >>>>=20 >>>> and since I use to build NanoBSD with -DNO_CLEAN, I'm just now = compiling a >>>> clean NanoBSD with RETPOLINE mitigations disabled so far - trying = to check >>>> whether either of the ways to build causes the issue. >>>>=20 >>>>>=20 >>>>> You can try to press space on first spinner and enter alternate = loader on >>>>> boot: prompt. (enter ?/boot on boot: prompt to see the file list). = =20 >>>>=20 >>>> I try a soon as the build process has finished and if the problem = is then >>>> still present. =20 >>>=20 >>>=20 >>> With a fresh build and no RETPOLINE mitigation (neither kernel nor = world) >>> the phenomenon as described above is still the same. I tried an = alternative >>> loader as requested, but without success. When choosing loader_4th, = I get >>> this error: >>>=20 >>> [...] >>> FreeBSD/x86 boot >>> Default: 0:ad(0p3)/boot/loader >>> boot: /boot/loader_4th/ >>>=20 >>> onsoles: internal video/keyboard >>> IOS drive C: is disk0 >>> IOS drive D: is disk1 >>> IOS 639kB/3404444kB available memory >>>=20 >>> reeBSD/x86 bootstrap loader, Revision 1.1 >>> Wed Jul 24 12:51:12 CEST 2019 root@thor) >>> anic: No heap setup =20 >>> -> Press a key on the console to reboot <=E2=80=94 =20 >>>=20 >>=20 >> Now this is bad. if my math is correct, this system is supposed to = have 3GB >> of RAM, so are there specific build exceptions in place? see >> stand/i386/loader/main.c, function main, after call to bios_getmem(). >>=20 >> rgds, >> toomas >=20 >=20 > Hello Toomas, > the PCengine APU2C4 is supposed to have 4GB of RAM - wouldn't have a = 64bit > system seen the whole range? On 32bit systems, there was a memory hole = I assume > for memory mapped I/O of several PCI devices. This is the first time = I have a > look at the memory reported by the kernel and its confusing me a bit. >=20 The BIOS loader is only running in 32-bit protected mode, we switch to = 64-bit when we start the kernel. With UEFI we do have 32- and 64-bit = loaders, depending on firmware implementation (because we need to use = firmware provided functions), but even there there are buggy systems and = we keep memory usage below 4G line. > I do not have any kind of specific build exceptions in place = targetting the > memory. Of course, for memory usage and image size optimizations I = defined > several WITHOUT_ and WITH_ tags for build and install - but they never = caused > any trouble and have not been changed so far. Unfortunately the only way to identify what is the cause, is to start = inserting debug printf=E2=80=99s into the code paths and to see where we = get blown up. There can be several reasons, and the most common case = still is plain and simple buffer overruns=E2=80=A6 debugging this is = time consuming job. rgds, toomas >=20 >>=20 >>>=20 >>> Loader loader_simp ends up in stuck console with no output: >>>=20 >>> [...] >>> FreeBSD/x86 boot >>> Default: 0:ad(0p3)/boot/loader >>> boot: /boot/loader_4th/ >>>=20 >>> onsoles: internal video/keyboard >>> IOS drive C: is disk0 >>> IOS drive D: is disk1 >>> IOS 639kB/3404444kB available memory >>>=20 >>> reeBSD/x86 bootstrap loader, Revision 1.1 >>> Wed Jul 24 12:59:23 CEST 2019 root@thor) >>> [...] >>>=20 >>> regards >>> oh =20 >>>>=20 >>>>>=20 >>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> Booting 12.0-STABLE #78 r349288: Sat Jun 22 09:10:25 CEST 2019 = amd64 >>>>>> works fine with nothing changed except the OS version. >>>>>>=20 >>>>>>=20 >>>>>> Booting 2.0-STABLE #78 r349288 works fine: >>>>>>=20 >>>>>> [...] >>>>>> SeaBIOS (version rel-1.12.1.3-0-g300e8b7) >>>>>>=20 >>>>>> Press F10 key now for boot menu >>>>>>=20 >>>>>> Booting from Hard Disk... >>>>>> | >>>>>>=20 >>>>>> onsoles: internal video/keyboard =20 >>>>>> IOS drive C: is disk0=20 >>>>>> IOS drive D: is disk1=20 >>>>>> IOS 639kB/3404444kB available memory=20 >>>>>>=20 >>>>>> reeBSD/x86 bootstrap loader, Revision 1.1=20 >>>>>> Mon Apr 15 21:28:11 CEST 2019 root@thor)=20 >>>>>> oading /boot/defaults/loader.conf=20 >>>>>> oading /boot/device.hints=20 >>>>>> oading /boot/loader.conf=20 >>>>>> oading /boot/loader.conf.local=20 >>>>>> Loading kernel... >>>>>> /boot/kernel/kernel text=3D0xb005e8 \ >>>>>> [...] >>>>>>=20 >>>>>> In the message taken from the serial console the first column of >>>>>> characters is lost due to an error in the output which seems = FreeBSD >>>>>> related.=20 >>>>>=20 >>>>> It certainly does look weird - sio_putc() is used in boot2 and = it=E2=80=99s >>>>> implementation is using same principe as comc_putchat() in = comconsole.c >>>>> (even if it is asm versus c code). Since the serial data is = interpreted >>>>> by terminal, it feels more about terminal emulator issue (line >>>>> discipline, cabling, usb to serial dongle?) =20 >>>>=20 >>>> We use here a null modem cabling with an integrated USB-to-UART/TTL >>>> converter, which is attached to a FreeBSD CURRENT (most recent) = box: >>>>=20 >>>> [...] >>>> ugen2.3: <FTDI FT232R USB UART> at usbus2 >>>> uftdi0 on uhub4 >>>> uftdi0: <FT232R USB UART> on usbus2 >>>> [...] >>>>=20 >>>> it is a=20 >>>> StarTech.com 1 Port USB Nullmodem RS232 Adapter Kabel (USB 2.0 FTDI >>>> chipset). >>>>=20 >>>> Regards, >>>> oh >>>>=20 >>>>=20 >>>>>=20 >>>>> rgds, >>>>> toomas >>>>>=20 >>>>>=20 >>>>>>=20 >>>>>> The file /boot/loader.conf.local contains these lines in both, = working >>>>>> and non-working, scenario: >>>>>>=20 >>>>>> [...] >>>>>> boot_serial=3D"YES" >>>>>> # serial speed in bits/s >>>>>> comconsole_speed=3D"115200" >>>>>> console=3D"comconsole" >>>>>>=20 >>>>>> autoboot_delay=3D"0" >>>>>>=20 >>>>>> verbose_loading=3D"YES" >>>>>> loader_logo=3D"orb" >>>>>> beastie_disable=3D"YES" >>>>>>=20 >>>>>> ### Microcode >>>>>> #cpu_microcode_load=3D"YES" # Set this to YES to = load and >>>>>> apply a #cpu_microcode_name=3D"/boot/firmware/intel-ucode.bin" # = Set this >>>>>> to the microcode #cpu_microcode_type=3D"cpu_microcode" # = Required for >>>>>> the kernel to find # the microcode update file. >>>>>>=20 >>>>>>=20 >>>>>> # disable Process Table Isolation >>>>>> #vm.pmap.pti=3D0 >>>>>>=20 >>>>>> kern.geom.label.gptid.enable=3D0 >>>>>>=20 >>>>>> # Limit the phys. memory >>>>>> #hw.physmem=3D1073741824 # 1 G >>>>>> #hw.physmem=3D536870912 # 512 MB >>>>>> #hw.physmem=3D268435456 # 256 MB >>>>>>=20 >>>>>> # Da mehr als 1 igb NIC an Bord! Siehe man igb(4) >>>>>> kern.ipc.nmbclusters=3D757350 >>>>>> #kern.ipc.nmbjumbo9k=3D8192 >>>>>>=20 >>>>>> # NIC >>>>>> #hw.em.max_interrupt_rate=3D32000 >>>>>> hw.em.max_interrupt_rate=3D16000 >>>>>>=20 >>>>>> #If non-zero, enable EXPERIMENTAL feature to improve concurrent = Fortuna >>>>>> performance kern.random.fortuna.concurrent_read=3D"1" >>>>>>=20 >>>>>> # Forward Information Bases (FIBs) >>>>>> net.fibs=3D10 >>>>>> net.add_addr_allfibs=3D0 >>>>>>=20 >>>>>> [...] >>>>>>=20 >>>>>>=20 >>>>>> Again, with the exact same setting 12-STABLE r349288 boots fine, >>>>>> rr350274 doesn't. FreeBSD 12-STABLE r >>>>>>=20 >>>>>> Can someone please help? >>>>>>=20 >>>>>> Thanks in advance, oh =20 > [...] > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to = "freebsd-current-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?D21DC702-CB58-4AAB-A010-20D0AFECD720>