Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 15 Oct 2018 21:00:17 -0700
From:      Mark Millard <marklmi@yahoo.com>
To:        FreeBSD PowerPC ML <freebsd-ppc@freebsd.org>, FreeBSD Toolchain <freebsd-toolchain@freebsd.org>, FreeBSD Mailing List <freebsd-hackers@freebsd.org>
Cc:        Justin Hibbits <jrh29@alumni.cwru.edu>
Subject:   Re: FYI: powerpc64 headbuilt via devel/powerpc64-xtoolchain-gcc and C++ exceptions for user code built by system-clang or devel/powerpc64-gcc (as of head -r339076 and ports -r480180)
Message-ID:  <3B69D483-AEBA-47CA-B140-7445089EB064@yahoo.com>
In-Reply-To: <0539C16B-1603-4639-914A-0308578C7262@yahoo.com>
References:  <E1167EAA-3F90-4F0A-9F51-53CFE1461617@yahoo.com> <0539C16B-1603-4639-914A-0308578C7262@yahoo.com>

next in thread | previous in thread | raw e-mail | index | archive | help
[I've found the problem at the low level for
my context of using WITH_LIBCPLUSPLUS=3D with
the likes of devel/powerpc64-gcc but I do
not have a solution for WITH_LIBCPLUSPLUS=3D
so far. I give details of what I found.]

On 2018-Oct-14, at 12:40 AM, Mark Millard <marklmi at yahoo.com> wrote:

> On 2018-Oct-12, at 1:59 PM, Mark Millard <marklmi at yahoo.com> wrote:
>=20
>> I built a powerpc64 head -r339076 via devel/powerpc64-gcc
>> and the like that built clang as cc as well (and
>> WITHOUT_LIB32). This included use of base/binutils to
>> the the powerpc64 set up. The system and kernel are
>> non-debug builds (but with symbols). [system-clang is not
>> used for buildworld buildkernel because of known
>> issues (last I tried).]
>>=20
>> booting, buildworld, buildkernel, poudriere building
>> what totaled to be somewhat under 400 ports all seem
>> to work. But . . .
>>=20
>> It been a long time since I've done something analogous
>> and a significant item in the result is different than in
>> the past once I started testing the throwing of C++
>> exceptions in code produced by system-clang or by
>> devel/powerpc64-gcc :
>>=20
>> Such code ends up stuck using around 100% of a CPU.
>> An example is the program:
>>=20
>> # more exception_test.cpp
>> #include <exception>
>>=20
>> int main(void)
>> {
>>   try { throw std::exception(); }
>>   catch (std::exception& e) {}
>>   return 0;
>> }
>>=20
>> For system-clang it ended up with:
>>=20
>> # ldd a.out
>> a.out:
>> 	libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x81006d000)
>> 	libcxxrt.so.1 =3D> /lib/libcxxrt.so.1 (0x810184000)
>> 	libm.so.5 =3D> /lib/libm.so.5 (0x8101ab000)
>> 	libc.so.7 =3D> /lib/libc.so.7 (0x8101eb000)
>> 	libgcc_s.so.1 =3D> /lib/libgcc_s.so.1 (0x810554000)
>>=20
>> That program goes into an possibly unbounded execution.
>> (Historically when this program had problems it would
>> stop and produce a core file.)
>>=20
>> When compiled by devel/powerpc64-gcc the a.out that results
>> does the same thing. ( =
/usr/local/bin/powerpc64-unknown-freebsd12.0-c++=20
>> as the compiler path ) So this is not really clang specific
>> in any way. This ended up with:
>>=20
>> # ldd a.out
>> a.out:
>> 	libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x81006d000)
>> 	libcxxrt.so.1 =3D> /lib/libcxxrt.so.1 (0x810184000)
>> 	libm.so.5 =3D> /lib/libm.so.5 (0x8101ab000)
>> 	libgcc_s.so.1 =3D> /lib/libgcc_s.so.1 (0x8101eb000)
>> 	libc.so.7 =3D> /lib/libc.so.7 (0x810211000)
>>=20
>> (That should not have involved clang or llvm at all.)
>>=20
>> But compiled by lang/gcc8's g++8 the a.out that results works
>> fine. This ends up with:
>>=20
>> # ldd a.out
>> a.out:
>> 	libstdc++.so.6 =3D> /usr/local/lib/gcc8/libstdc++.so.6 =
(0x81006e000)
>> 	libm.so.5 =3D> /lib/libm.so.5 (0x8102c7000)
>> 	libgcc_s.so.1 =3D> /lib/libgcc_s.so.1 (0x810307000)
>> 	libc.so.7 =3D> /lib/libc.so.7 (0x81032d000)
>>=20
>> It is not clear if using base/gcc as system cc
>> would do any better than using system-clang does
>> or than devel/powerpc64-gcc does: it is sort of
>> a variant of devel/powerpc64-gcc .
>>=20
>> It will probably be some time before I figure out
>> much about what is going on.
>>=20
>> Two things common to the problem cases are:
>>=20
>> libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x81006d000)
>> libcxxrt.so.1 =3D> /lib/libcxxrt.so.1 (0x810184000)
>>=20
>> lang/gcc8 avoids those being involved.
>>=20
>>=20
>> Notes:
>>=20
>> . . .
>>=20
>> WITHOUT_LIB32=3D is because, for every post-gcc 4.2.1
>> that I've tried, the lib32 produced misuses R30 in
>> crtbeginS code (vs. the ABI for FreeBSD) and 32-bit
>> code just produces core files from the bad so-called
>> address dereference that results.
>>=20
>> I'd rather have throwing C++ exceptions working and
>> lack of lib32 than have lib32 but not have throwing
>> C++ exceptions working. But at the moment how to have
>> such is not obvious when fairly modern compilers
>> and toolchains are involved.=20
>=20
> Here is what I've found so far.
>=20
> The code is looping in the following routine.
> (I've inserted 2 NOTE: lines for what the
> sustained looping is like.)
>=20
. . . (the routine was _Unwind_RaiseException) . . .


So far I've found that the following in
_Unwind_RaiseException stays invariant once
initialized --despite the uw_frame_state_for
and uw_update_context calls in
_Unwind_RaiseException 's loop that normally
update fs:


(gdb) print fs
$15 =3D {regs =3D {reg =3D {{loc =3D {reg =3D 0, offset =3D 0, exp =3D =
0x0}, how =3D REG_UNSAVED} <repeats 31 times>, {loc =3D {reg =3D =
18446744073709551608, offset =3D -8,=20
          exp =3D 0xfffffffffffffff8 <Error reading address =
0xfffffffffffffff8: Bad address>}, how =3D REG_SAVED_OFFSET}, {loc =3D =
{reg =3D 0, offset =3D 0, exp =3D 0x0}, how =3D REG_UNSAVED} <repeats 33 =
times>, {
        loc =3D {reg =3D 16, offset =3D 16, exp =3D 0x10 <Error reading =
address 0x10: Bad address>}, how =3D REG_SAVED_OFFSET}, {loc =3D {reg =3D =
0, offset =3D 0, exp =3D 0x0}, how =3D REG_UNSAVED} <repeats 80 times>},=20=

    prev =3D 0x0}, cfa_offset =3D 0, cfa_reg =3D 1, cfa_exp =3D 0x0, =
cfa_how =3D CFA_REG_OFFSET, pc =3D 0x8101999f8, personality =3D 0, =
data_align =3D -8, code_align =3D 4, retaddr_column =3D 65,=20
  fde_encoding =3D 27 '\033', lsda_encoding =3D 255 '?', saw_z =3D 1 =
'\001', signal_frame =3D 0 '\0', eh_ptr =3D 0x0}

It turns out that pc value 0x8101999f8 is a little after the libcxxrt.so
call to _Unwind_RaiseException that is in throw_exception. But
_Unwind_RaiseException returning would be a failure and would end up
in a non-returning, error-reporting code path.

In other words: this is not an appropriate context for following the
return path to unwind out of _Unwind_RaiseException and its internal
caller (throw_exception).

It got to that address from lr containing the address of the
instruction after the one that does bl to the
_Unwind_RaiseException routine.

Overall it needs to unwind past this in the normal case but is stuck
handling the error/no-return case as "the" case.

Supporting details follow.



What lead up to 0x8101999f8 for initialization was
the lr value related to calling _Unwind_RaiseException
(see the address in lr below, also copied to r5):

. . .
(gdb) c
Continuing.

Breakpoint 9, <signal handler called>
1: x/i $pc  0x8101f35d8 <_Unwind_RaiseException+216>:	bl      =
0x8101f2bc0 <uw_init_context_1>
Current language:  auto; currently minimal

where the register values being supplied are (see
r5 and lr):

(gdb) info reg
r0             0x8101999f0	34629851632
r1             0x3fffffffffffc320	4611686018427372320
r2             0x810217900	34630367488
r3             0x3fffffffffffd280	4611686018427376256
r4             0x3fffffffffffd930	4611686018427377968
r5             0x8101999f0	34629851632
r6             0xa0	160
r7             0x0	0
r8             0x1	1
r9             0x8101aac10	34629921808
r10            0x1	1
r11            0x28	40
r12            0x28000282	671089282
r13            0x81005d020	34628554784
r14            0x0	0
r15            0x0	0
r16            0x0	0
r17            0x0	0
r18            0x0	0
r19            0x0	0
r20            0x0	0
r21            0x0	0
r22            0x0	0
r23            0x0	0
r24            0x0	0
r25            0x0	0
r26            0x0	0
r27            0x3fffffffffffd280	4611686018427376256
r28            0x810041060	34628440160
r29            0x3fffffffffffc390	4611686018427372432
r30            0x3fffffffffffcd10	4611686018427374864
r31            0x810041008	34628440072
pc             0x8101f35d8	34630219224
ps             0x0	0
cr             0x0	0
lr             0x8101999f0	34629851632
ctr            0x8101f3500	34630219008
xer            0x0	0
fpscr          0xfff80000	-524288
vscr           0x0	0
vrsave         0x0	0


The pc listed in print fs (0x8101999f8) is in the
following from libcxxrt, as is the value in r5 and
lr (0x8101999f0): (some blank lines inserted to
help identify the area and some related material)

(gdb) disass throw_exception
Dump of assembler code for function throw_exception:
0x0000000810199960 <throw_exception+0>:	mflr    r0
0x0000000810199964 <throw_exception+4>:	std     r31,-8(r1)
0x0000000810199968 <throw_exception+8>:	mr      r31,r3
0x000000081019996c <throw_exception+12>:	std     r0,16(r1)
0x0000000810199970 <throw_exception+16>:	stdu    r1,-128(r1)
0x0000000810199974 <throw_exception+20>:	bl      0x810197ab0 =
<thread_info>
0x0000000810199978 <throw_exception+24>:	ld      r10,8(r3)
0x000000081019997c <throw_exception+28>:	mr      r9,r3
0x0000000810199980 <throw_exception+32>:	cmpdi   cr7,r10,0
0x0000000810199984 <throw_exception+36>:	std     r10,24(r31)
0x0000000810199988 <throw_exception+40>:	beq-    cr7,0x810199a10 =
<throw_exception+176>
0x000000081019998c <throw_exception+44>:	ld      r10,0(r9)
0x0000000810199990 <throw_exception+48>:	cmpdi   cr7,r10,0
0x0000000810199994 <throw_exception+52>:	std     r10,32(r31)
0x0000000810199998 <throw_exception+56>:	beq-    cr7,0x8101999d0 =
<throw_exception+112>
0x000000081019999c <throw_exception+60>:	lwz     r10,48(r9)
0x00000008101999a0 <throw_exception+64>:	addi    r3,r31,88
0x00000008101999a4 <throw_exception+68>:	addi    r10,r10,1
0x00000008101999a8 <throw_exception+72>:	stw     r10,48(r9)
0x00000008101999ac <throw_exception+76>:	bl      0x81018e500 =
<00000018.plt_call._Unwind_RaiseException@@GCC_3.0>
0x00000008101999b0 <throw_exception+80>:	ld      r2,40(r1)
0x00000008101999b4 <throw_exception+84>:	addi    r1,r1,128
0x00000008101999b8 <throw_exception+88>:	mr      r4,r31
0x00000008101999bc <throw_exception+92>:	ld      r0,16(r1)
0x00000008101999c0 <throw_exception+96>:	ld      r31,-8(r1)
0x00000008101999c4 <throw_exception+100>:	mtlr    r0
0x00000008101999c8 <throw_exception+104>:	b       0x8101996b0 =
<report_failure>
0x00000008101999cc <throw_exception+108>:	nop
0x00000008101999d0 <throw_exception+112>:	nop
0x00000008101999d4 <throw_exception+116>:	addi    r3,r31,88
0x00000008101999d8 <throw_exception+120>:	ld      r10,-30008(r2)
0x00000008101999dc <throw_exception+124>:	std     r10,32(r31)
0x00000008101999e0 <throw_exception+128>:	lwz     r10,48(r9)
0x00000008101999e4 <throw_exception+132>:	addi    r10,r10,1
0x00000008101999e8 <throw_exception+136>:	stw     r10,48(r9)

0x00000008101999ec <throw_exception+140>:	bl      0x81018e500 =
<00000018.plt_call._Unwind_RaiseException@@GCC_3.0>

0x00000008101999f0 <throw_exception+144>:	ld      r2,40(r1)
0x00000008101999f4 <throw_exception+148>:	addi    r1,r1,128

0x00000008101999f8 <throw_exception+152>:	mr      r4,r31

0x00000008101999fc <throw_exception+156>:	ld      r0,16(r1)
0x0000000810199a00 <throw_exception+160>:	ld      r31,-8(r1)
0x0000000810199a04 <throw_exception+164>:	mtlr    r0
0x0000000810199a08 <throw_exception+168>:	b       0x8101996b0 =
<report_failure>

0x0000000810199a0c <throw_exception+172>:	nop
0x0000000810199a10 <throw_exception+176>:	nop
0x0000000810199a14 <throw_exception+180>:	ld      r10,-30000(r2)
0x0000000810199a18 <throw_exception+184>:	std     r10,24(r31)
0x0000000810199a1c <throw_exception+188>:	b       0x81019998c =
<throw_exception+44>
0x0000000810199a20 <throw_exception+192>:	.long 0x0
0x0000000810199a24 <throw_exception+196>:	.long 0x90001
0x0000000810199a28 <throw_exception+200>:	lwz     r0,0(r1)
End of assembler dump.

For:           0x00000008101999f8 (above)
objdump shows:   00000000000159f8 (below):

0000000000015960 <.__cxa_end_catch+0x460> mflr    r0
0000000000015964 <.__cxa_end_catch+0x464> std     r31,-8(r1)
0000000000015968 <.__cxa_end_catch+0x468> mr      r31,r3
000000000001596c <.__cxa_end_catch+0x46c> std     r0,16(r1)
0000000000015970 <.__cxa_end_catch+0x470> stdu    r1,-128(r1)
0000000000015974 <.__cxa_end_catch+0x474> bl      0000000000013ab0 =
<._ZdaPv+0x590>
0000000000015978 <.__cxa_end_catch+0x478> ld      r10,8(r3)
000000000001597c <.__cxa_end_catch+0x47c> mr      r9,r3
0000000000015980 <.__cxa_end_catch+0x480> cmpdi   cr7,r10,0
0000000000015984 <.__cxa_end_catch+0x484> std     r10,24(r31)
0000000000015988 <.__cxa_end_catch+0x488> beq     cr7,0000000000015a10 =
<.__cxa_end_catch+0x510>
000000000001598c <.__cxa_end_catch+0x48c> ld      r10,0(r9)
0000000000015990 <.__cxa_end_catch+0x490> cmpdi   cr7,r10,0
0000000000015994 <.__cxa_end_catch+0x494> std     r10,32(r31)
0000000000015998 <.__cxa_end_catch+0x498> beq     cr7,00000000000159d0 =
<.__cxa_end_catch+0x4d0>
000000000001599c <.__cxa_end_catch+0x49c> lwz     r10,48(r9)
00000000000159a0 <.__cxa_end_catch+0x4a0> addi    r3,r31,88
00000000000159a4 <.__cxa_end_catch+0x4a4> addi    r10,r10,1
00000000000159a8 <.__cxa_end_catch+0x4a8> stw     r10,48(r9)
00000000000159ac <.__cxa_end_catch+0x4ac> bl      000000000000a500 =
<CXXABI_1.3@@CXXABI_1.3+0xa500>
00000000000159b0 <.__cxa_end_catch+0x4b0> ld      r2,40(r1)
00000000000159b4 <.__cxa_end_catch+0x4b4> addi    r1,r1,128
00000000000159b8 <.__cxa_end_catch+0x4b8> mr      r4,r31
00000000000159bc <.__cxa_end_catch+0x4bc> ld      r0,16(r1)
00000000000159c0 <.__cxa_end_catch+0x4c0> ld      r31,-8(r1)
00000000000159c4 <.__cxa_end_catch+0x4c4> mtlr    r0
00000000000159c8 <.__cxa_end_catch+0x4c8> b       00000000000156b0 =
<.__cxa_end_catch+0x1b0>
00000000000159cc <.__cxa_end_catch+0x4cc> nop
00000000000159d0 <.__cxa_end_catch+0x4d0> nop
00000000000159d4 <.__cxa_end_catch+0x4d4> addi    r3,r31,88
00000000000159d8 <.__cxa_end_catch+0x4d8> ld      r10,-30008(r2)
00000000000159dc <.__cxa_end_catch+0x4dc> std     r10,32(r31)
00000000000159e0 <.__cxa_end_catch+0x4e0> lwz     r10,48(r9)
00000000000159e4 <.__cxa_end_catch+0x4e4> addi    r10,r10,1
00000000000159e8 <.__cxa_end_catch+0x4e8> stw     r10,48(r9)

00000000000159ec <.__cxa_end_catch+0x4ec> bl      000000000000a500 =
<CXXABI_1.3@@CXXABI_1.3+0xa500>

00000000000159f0 <.__cxa_end_catch+0x4f0> ld      r2,40(r1)
00000000000159f4 <.__cxa_end_catch+0x4f4> addi    r1,r1,128

00000000000159f8 <.__cxa_end_catch+0x4f8> mr      r4,r31

00000000000159fc <.__cxa_end_catch+0x4fc> ld      r0,16(r1)
0000000000015a00 <.__cxa_end_catch+0x500> ld      r31,-8(r1)
0000000000015a04 <.__cxa_end_catch+0x504> mtlr    r0
0000000000015a08 <.__cxa_end_catch+0x508> b       00000000000156b0 =
<.__cxa_end_catch+0x1b0>

0000000000015a0c <.__cxa_end_catch+0x50c> nop
0000000000015a10 <.__cxa_end_catch+0x510> nop
0000000000015a14 <.__cxa_end_catch+0x514> ld      r10,-30000(r2)
0000000000015a18 <.__cxa_end_catch+0x518> std     r10,24(r31)
0000000000015a1c <.__cxa_end_catch+0x51c> b       000000000001598c =
<.__cxa_end_catch+0x48c>
0000000000015a20 <.__cxa_end_catch+0x520> .long 0x0
0000000000015a24 <.__cxa_end_catch+0x524> .long 0x90001
0000000000015a28 <.__cxa_end_catch+0x528> lwz     r0,0(r1)

And dwarfdump shows starting at 0x00015960 as:

<  117><0x00015960:0x00015a2c><><cie offset 0x0000101c::cie index     =
0><fde offset 0x00001018 length: 0x00000030>
       <eh aug data len 0x0>
        0x00015960: <off cfa=3D00(r1) >=20
        0x00015968: <off cfa=3D00(r1) > <off r31=3D-8(cfa) > <off r65=3Dr0=
 >=20
        0x00015974: <off cfa=3D128(r1) > <off r31=3D-8(cfa) > <off =
r65=3D16(cfa) >=20
        0x000159b8: <off cfa=3D00(r1) > <off r31=3D-8(cfa) > <off =
r65=3D16(cfa) >=20
        0x000159c8: <off cfa=3D00(r1) >=20

        0x000159d0: <off cfa=3D128(r1) > <off r31=3D-8(cfa) > <off =
r65=3D16(cfa) >=20

        0x000159f8: <off cfa=3D00(r1) > <off r31=3D-8(cfa) > <off =
r65=3D16(cfa) >=20

        0x00015a08: <off cfa=3D00(r1) >=20
        0x00015a10: <off cfa=3D128(r1) > <off r31=3D-8(cfa) > <off =
r65=3D16(cfa) >=20
 fde section offset 4120 0x00001018 cie offset for fde: 4124 0x0000101c
         0 DW_CFA_advance_loc 8  (2 * 4)
         1 DW_CFA_register r65 =3D r0
         4 DW_CFA_offset r31 -8  (1 * -8)
         6 DW_CFA_advance_loc 12  (3 * 4)
         7 DW_CFA_def_cfa_offset 128
        10 DW_CFA_offset_extended_sf r65 16  (-2 * -8)
        13 DW_CFA_advance_loc 68  (17 * 4)
        14 DW_CFA_remember_state
        15 DW_CFA_def_cfa_offset 0
        17 DW_CFA_advance_loc 16  (4 * 4)
        18 DW_CFA_restore_extended r65
        20 DW_CFA_restore r31
        21 DW_CFA_advance_loc 8  (2 * 4)
        22 DW_CFA_restore_state
        23 DW_CFA_advance_loc 40  (10 * 4)
        24 DW_CFA_remember_state
        25 DW_CFA_def_cfa_offset 0
        27 DW_CFA_advance_loc 16  (4 * 4)
        28 DW_CFA_restore_extended r65
        30 DW_CFA_restore r31
        31 DW_CFA_advance_loc 8  (2 * 4)
        32 DW_CFA_restore_state
        33 DW_CFA_nop
        34 DW_CFA_nop

/usr/src/contrib/libstdc++/libsupc++/eh_throw.cc has something
that /usr/src/contrib/libcxxrt/exception.cc does not have in
its error handling code path: a use of __cxa_begin_catch=20
in __cxxabiv1::__cxa_throw :

#ifdef _GLIBCXX_SJLJ_EXCEPTIONS
  _Unwind_SjLj_RaiseException (&header->unwindHeader);
#else
  _Unwind_RaiseException (&header->unwindHeader);
#endif

  // Some sort of unwinding error.  Note that terminate is a handler.
  __cxa_begin_catch (&header->unwindHeader);
  std::terminate ();

It looks to me like __cxa_begin_catch might do either of
terminate or _Unwind_Resume and that the (conditional)
_Unwind_Resume case is possibly needed here for the
normal execution.

There are also no other calls (bl's) before the terminate:
more directly indicated to not return.

I do not know if one or both of those points helped the code
unwind correctly or not. But it seems suggestive.


Other notes:

I've demonstrated the problem on my prior head
-r333594 environment that had been build via a
6.3 vintage of devel/powerpc64-gcc (but that was
then updated to ports -r469844 and so had 6.4 of
devel/powerpc64-gcc installed). Also: base/binutils
was of a 2.30 vintage and base/gcc was of a 6.3
vintage. (system-clang was not cc, base/gcc
provided system-cc.)

Compiling to produce the a.out via:

/usr/bin/powerpc64-unknown-freebsd12.0-g++

(and so via a 6.3 vintage g++) made no
difference. It still had the problem.

I have taken to having buildworld buildkernel
use -gdwarf-2 so that /usr/libexec/gdb has
access to the information in a format it is
(mostly) designed for. (/usr/local/bin/gdb is
broken by the thrown-C++-exception problem that
I'm investigating: gdb internally throws
exceptions in normal operation.)


=3D=3D=3D
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3B69D483-AEBA-47CA-B140-7445089EB064>