From owner-freebsd-embedded@freebsd.org Tue Jul 30 14:02:11 2019 Return-Path: Delivered-To: freebsd-embedded@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6326DA4398 for ; Tue, 30 Jul 2019 14:02:11 +0000 (UTC) (envelope-from tsoome@me.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46ED69AFC5 for ; Tue, 30 Jul 2019 14:02:11 +0000 (UTC) (envelope-from tsoome@me.com) Received: by mailman.nyi.freebsd.org (Postfix) id 4676DA4397; Tue, 30 Jul 2019 14:02:11 +0000 (UTC) Delivered-To: embedded@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 46341A4396 for ; Tue, 30 Jul 2019 14:02:11 +0000 (UTC) (envelope-from tsoome@me.com) Received: from mr85p00im-zteg06023901.me.com (mr85p00im-zteg06023901.me.com [17.58.23.192]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1A5FB9AFC1 for ; Tue, 30 Jul 2019 14:02:11 +0000 (UTC) (envelope-from tsoome@me.com) Received: from nazgul.lan (148-52-235-80.sta.estpak.ee [80.235.52.148]) by mr85p00im-zteg06023901.me.com (Postfix) with ESMTPSA id A93DC9609C6; Tue, 30 Jul 2019 14:02:00 +0000 (UTC) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Subject: Re: PCengines APU2C4, 12-STABLE: bootloader failure: Panic: free: guard2 fail @ 0x1000 + 2311663946 from From: Toomas Soome In-Reply-To: <20190730144331.24f83e0c@freyja> Date: Tue, 30 Jul 2019 17:01:57 +0300 Cc: "O. Hartmann" , "freebsd-embedded@freebsd.org" , FreeBSD CURRENT Content-Transfer-Encoding: quoted-printable Message-Id: References: <20190724101132.5560377a@thor.intern.walstatt.dynvpn.de> <20190724120720.4382397c@thor.intern.walstatt.dynvpn.de> <20190724154901.0c114ebe@thor.intern.walstatt.dynvpn.de> <93EFC481-50A1-491C-87BB-12A35F3302EE@me.com> <20190730144331.24f83e0c@freyja> To: "O. Hartmann" X-Mailer: Apple Mail (2.3445.104.11) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-07-30_07:, , signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1011 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1812120000 definitions=main-1907300146 X-Rspamd-Queue-Id: 46ED69AFC5 X-Spamd-Bar: ------ X-Spamd-Result: default: False [-6.98 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.982,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Jul 2019 14:02:11 -0000 > On 30 Jul 2019, at 15:43, O. Hartmann wrote: >=20 > On Wed, 24 Jul 2019 18:07:22 +0300 > Toomas Soome wrote: >=20 >>> On 24 Jul 2019, at 16:48, O. Hartmann = wrote: >>>=20 >>> -----BEGIN PGP SIGNED MESSAGE----- >>> Hash: SHA256 >>>=20 >>> Am Wed, 24 Jul 2019 12:06:53 +0200 >>> "O. Hartmann" > >>> schrieb:=20 >>>> -----BEGIN PGP SIGNED MESSAGE----- >>>> Hash: SHA256 >>>>=20 >>>> Am Wed, 24 Jul 2019 12:09:16 +0300 >>>> Toomas Soome schrieb: >>>>=20 >>>>>> On 24 Jul 2019, at 11:11, O. Hartmann = wrote: >>>>>>=20 >>>>>> -----BEGIN PGP SIGNED MESSAGE----- >>>>>> Hash: SHA256 >>>>>>=20 >>>>>> Hallo, >>>>>>=20 >>>>>> on APU2C4 from PCengines with latest firmware apu2_v4.9.0.7.rom, = SeaBIOS >>>>>> rel-1.12.1.3-0-g300e8b7, booting via legacy MBR FreeBSD 12-STABLE >>>>>> r350274 (the same with r350115) fails to boot with an immediate = loader >>>>>> error: >>>>>>=20 >>>>>> [...] >>>>>> SeaBIOS (version rel-1.12.1.3-0-g300e8b7) >>>>>>=20 >>>>>> Press F10 key now for boot menu >>>>>>=20 >>>>>> Booting from Hard Disk... >>>>>> / >>>>>>=20 >>>>>> onsoles: internal video/keyboard =20 >>>>>> IOS drive C: is disk0=20 >>>>>> IOS drive D: is disk1=20 >>>>>> IOS 639kB/3404444kB available memory=20 >>>>>>=20 >>>>>> reeBSD/x86 bootstrap loader, Revision 1.1 =20 >>>>>> Mon Apr 15 21:28:11 CEST 2019 root@thor)=20 >>>>>> anic: free: guard2 fail @ 0x1000 + 2311663946 from >>>>>> X=C3=A7u0=C3=A7}4=C3=A7l$=E2=99=A6=E2=94=9C=C3=AD@=E2=94=A4=E2=99=A0= :2106163957 -> Press a key on the console to reboot >>>>>> <-- [=E2=80=A6] =20 >>>>>=20 >>>>>=20 >>>>> This is definitely something =E2=80=9Cfunny=E2=80=9D, we are = apparently attempting to >>>>> free pointer 0x1000 which is definitely wrong because our heap = should be >>>>> just below 4GB line. Since we do get list of disks printed, also = memory >>>>> and version, it means we get error from interpretator - it is = possible >>>>> the stack did clash with bss and hence the corruption. =20 >>>>=20 >>>> I realized that I have defined=20 >>>>=20 >>>> WITH_KERNEL_RETPOLINE=3DYES >>>>=20 >>>> and since I use to build NanoBSD with -DNO_CLEAN, I'm just now = compiling a >>>> clean NanoBSD with RETPOLINE mitigations disabled so far - trying = to check >>>> whether either of the ways to build causes the issue. >>>>=20 >>>>>=20 >>>>> You can try to press space on first spinner and enter alternate = loader on >>>>> boot: prompt. (enter ?/boot on boot: prompt to see the file list). = =20 >>>>=20 >>>> I try a soon as the build process has finished and if the problem = is then >>>> still present. =20 >>>=20 >>>=20 >>> With a fresh build and no RETPOLINE mitigation (neither kernel nor = world) >>> the phenomenon as described above is still the same. I tried an = alternative >>> loader as requested, but without success. When choosing loader_4th, = I get >>> this error: >>>=20 >>> [...] >>> FreeBSD/x86 boot >>> Default: 0:ad(0p3)/boot/loader >>> boot: /boot/loader_4th/ >>>=20 >>> onsoles: internal video/keyboard >>> IOS drive C: is disk0 >>> IOS drive D: is disk1 >>> IOS 639kB/3404444kB available memory >>>=20 >>> reeBSD/x86 bootstrap loader, Revision 1.1 >>> Wed Jul 24 12:51:12 CEST 2019 root@thor) >>> anic: No heap setup =20 >>> -> Press a key on the console to reboot <=E2=80=94 =20 >>>=20 >>=20 >> Now this is bad. if my math is correct, this system is supposed to = have 3GB >> of RAM, so are there specific build exceptions in place? see >> stand/i386/loader/main.c, function main, after call to bios_getmem(). >>=20 >> rgds, >> toomas >=20 >=20 > Hello Toomas, > the PCengine APU2C4 is supposed to have 4GB of RAM - wouldn't have a = 64bit > system seen the whole range? On 32bit systems, there was a memory hole = I assume > for memory mapped I/O of several PCI devices. This is the first time = I have a > look at the memory reported by the kernel and its confusing me a bit. >=20 The BIOS loader is only running in 32-bit protected mode, we switch to = 64-bit when we start the kernel. With UEFI we do have 32- and 64-bit = loaders, depending on firmware implementation (because we need to use = firmware provided functions), but even there there are buggy systems and = we keep memory usage below 4G line. > I do not have any kind of specific build exceptions in place = targetting the > memory. Of course, for memory usage and image size optimizations I = defined > several WITHOUT_ and WITH_ tags for build and install - but they never = caused > any trouble and have not been changed so far. Unfortunately the only way to identify what is the cause, is to start = inserting debug printf=E2=80=99s into the code paths and to see where we = get blown up. There can be several reasons, and the most common case = still is plain and simple buffer overruns=E2=80=A6 debugging this is = time consuming job. rgds, toomas >=20 >>=20 >>>=20 >>> Loader loader_simp ends up in stuck console with no output: >>>=20 >>> [...] >>> FreeBSD/x86 boot >>> Default: 0:ad(0p3)/boot/loader >>> boot: /boot/loader_4th/ >>>=20 >>> onsoles: internal video/keyboard >>> IOS drive C: is disk0 >>> IOS drive D: is disk1 >>> IOS 639kB/3404444kB available memory >>>=20 >>> reeBSD/x86 bootstrap loader, Revision 1.1 >>> Wed Jul 24 12:59:23 CEST 2019 root@thor) >>> [...] >>>=20 >>> regards >>> oh =20 >>>>=20 >>>>>=20 >>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> Booting 12.0-STABLE #78 r349288: Sat Jun 22 09:10:25 CEST 2019 = amd64 >>>>>> works fine with nothing changed except the OS version. >>>>>>=20 >>>>>>=20 >>>>>> Booting 2.0-STABLE #78 r349288 works fine: >>>>>>=20 >>>>>> [...] >>>>>> SeaBIOS (version rel-1.12.1.3-0-g300e8b7) >>>>>>=20 >>>>>> Press F10 key now for boot menu >>>>>>=20 >>>>>> Booting from Hard Disk... >>>>>> | >>>>>>=20 >>>>>> onsoles: internal video/keyboard =20 >>>>>> IOS drive C: is disk0=20 >>>>>> IOS drive D: is disk1=20 >>>>>> IOS 639kB/3404444kB available memory=20 >>>>>>=20 >>>>>> reeBSD/x86 bootstrap loader, Revision 1.1=20 >>>>>> Mon Apr 15 21:28:11 CEST 2019 root@thor)=20 >>>>>> oading /boot/defaults/loader.conf=20 >>>>>> oading /boot/device.hints=20 >>>>>> oading /boot/loader.conf=20 >>>>>> oading /boot/loader.conf.local=20 >>>>>> Loading kernel... >>>>>> /boot/kernel/kernel text=3D0xb005e8 \ >>>>>> [...] >>>>>>=20 >>>>>> In the message taken from the serial console the first column of >>>>>> characters is lost due to an error in the output which seems = FreeBSD >>>>>> related.=20 >>>>>=20 >>>>> It certainly does look weird - sio_putc() is used in boot2 and = it=E2=80=99s >>>>> implementation is using same principe as comc_putchat() in = comconsole.c >>>>> (even if it is asm versus c code). Since the serial data is = interpreted >>>>> by terminal, it feels more about terminal emulator issue (line >>>>> discipline, cabling, usb to serial dongle?) =20 >>>>=20 >>>> We use here a null modem cabling with an integrated USB-to-UART/TTL >>>> converter, which is attached to a FreeBSD CURRENT (most recent) = box: >>>>=20 >>>> [...] >>>> ugen2.3: at usbus2 >>>> uftdi0 on uhub4 >>>> uftdi0: on usbus2 >>>> [...] >>>>=20 >>>> it is a=20 >>>> StarTech.com 1 Port USB Nullmodem RS232 Adapter Kabel (USB 2.0 FTDI >>>> chipset). >>>>=20 >>>> Regards, >>>> oh >>>>=20 >>>>=20 >>>>>=20 >>>>> rgds, >>>>> toomas >>>>>=20 >>>>>=20 >>>>>>=20 >>>>>> The file /boot/loader.conf.local contains these lines in both, = working >>>>>> and non-working, scenario: >>>>>>=20 >>>>>> [...] >>>>>> boot_serial=3D"YES" >>>>>> # serial speed in bits/s >>>>>> comconsole_speed=3D"115200" >>>>>> console=3D"comconsole" >>>>>>=20 >>>>>> autoboot_delay=3D"0" >>>>>>=20 >>>>>> verbose_loading=3D"YES" >>>>>> loader_logo=3D"orb" >>>>>> beastie_disable=3D"YES" >>>>>>=20 >>>>>> ### Microcode >>>>>> #cpu_microcode_load=3D"YES" # Set this to YES to = load and >>>>>> apply a #cpu_microcode_name=3D"/boot/firmware/intel-ucode.bin" # = Set this >>>>>> to the microcode #cpu_microcode_type=3D"cpu_microcode" # = Required for >>>>>> the kernel to find # the microcode update file. >>>>>>=20 >>>>>>=20 >>>>>> # disable Process Table Isolation >>>>>> #vm.pmap.pti=3D0 >>>>>>=20 >>>>>> kern.geom.label.gptid.enable=3D0 >>>>>>=20 >>>>>> # Limit the phys. memory >>>>>> #hw.physmem=3D1073741824 # 1 G >>>>>> #hw.physmem=3D536870912 # 512 MB >>>>>> #hw.physmem=3D268435456 # 256 MB >>>>>>=20 >>>>>> # Da mehr als 1 igb NIC an Bord! Siehe man igb(4) >>>>>> kern.ipc.nmbclusters=3D757350 >>>>>> #kern.ipc.nmbjumbo9k=3D8192 >>>>>>=20 >>>>>> # NIC >>>>>> #hw.em.max_interrupt_rate=3D32000 >>>>>> hw.em.max_interrupt_rate=3D16000 >>>>>>=20 >>>>>> #If non-zero, enable EXPERIMENTAL feature to improve concurrent = Fortuna >>>>>> performance kern.random.fortuna.concurrent_read=3D"1" >>>>>>=20 >>>>>> # Forward Information Bases (FIBs) >>>>>> net.fibs=3D10 >>>>>> net.add_addr_allfibs=3D0 >>>>>>=20 >>>>>> [...] >>>>>>=20 >>>>>>=20 >>>>>> Again, with the exact same setting 12-STABLE r349288 boots fine, >>>>>> rr350274 doesn't. FreeBSD 12-STABLE r >>>>>>=20 >>>>>> Can someone please help? >>>>>>=20 >>>>>> Thanks in advance, oh =20 > [...] > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to = "freebsd-current-unsubscribe@freebsd.org"