From owner-freebsd-current@freebsd.org Thu Jul 26 01:52:20 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C35331059B4A for ; Thu, 26 Jul 2018 01:52:19 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic303-22.consmr.mail.ne1.yahoo.com (sonic303-22.consmr.mail.ne1.yahoo.com [66.163.188.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5A1C076118 for ; Thu, 26 Jul 2018 01:52:18 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: 2ONrIdgVM1kzNP1UhgikjtoqRFlnWUBRn92JMboXGZu7fNVUONBOMT.i3hXhY8S uas1R9_3Z7CRQP.1.qJxzxwYYtTw_.bm0ob9qs7ExSJJ9_CTPvIBSF92mMQZ4UHUbTXCkc9Rp6s. 7009ATxrOcr96wdRqZcYskobWv4.8ezWE7dtDGVJ3.PsAybE63kJqiZyqJAhf7R8Vzyq5to3VZpN HJ3kl0aYFwHC9oqjNZn46QE6ZdcaoNDZogqvhZ610bbf0x6wSSuDq1KA1H9bnW5uVJgD6G4kCXqL tm531I3u8UoXeCKdVtoK.CkYKOoql46gY6eO887J6VAgFXwldbr07MTx67cCOvFFlCZ3U7q.wcH. 5krtCcNSdlTuYtvy.oRaZ._am2L1lG.DrEVYM3ZRvJ._rkvdqVlueXXmi2Uk.t6WguynElAhFpVv TKoCDRnNABOhWINr01oVbcWW8YHgvjnlHwrvxxdY64gSshJp_awAKbNKT7rbldc24wICWFVWRZyZ 3yi7VzxDu6YXeW.wIEY5UKc4TfXc6YYWY39f8gQW92nbemI.pZ2cj0h9Kq7O0sEbbqKtwIXxrZuQ xPS.9dc29aNDNeHL8qdXMZE0E1V4CL1YWNe0.QWQmaaMywU.tdFHjPd.bta.MvNJWXRw8yYQ33XG 4zDAfdZhvIIwivF5YxuQY15j1cCEDmG0sWIAb5dXBSpkkzezigS0hVbRk8QyrHcKMMxcBbvaLKh8 lObyno_pe93Jab9ayFkaTd_Epv2ajCRK7B0Oipq.iLmvXfeaevHtw3ktfuStoETyUAGkuOt.PCAr GME2jag_xphU6RgQc5Sj5B1a1M3JYtXSarIM.5m1HG8SujaIcpWHvdv6F7yL0f.J0wxh1deSYBLV Z4Z9flLr_qP3cg50ybV4AsgeNnShpk5NfxEnsFg0FfBl3bQ.ufH4g4P5QGZqOOWEbprzJztr9_po svHdgQuEQL3hAqwX4kNIBYnVM5Y6mHAhxvyZ4RGs1M7.zcU8L9H9qyvGfjGT_R4iz3p0X5pQJ81L e9.fZLJKWE0qYcf1l0cbH8g-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic303.consmr.mail.ne1.yahoo.com with HTTP; Thu, 26 Jul 2018 01:52:12 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp402.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 2ee2ee847328818624f94d2bfa3bfe23; Thu, 26 Jul 2018 01:52:08 +0000 (UTC) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: head -r336568 and -r336570 appears to have made ci.freebsg.org's FreeBSD-head-amd64-gcc fail either than it had been (error: operand type 'struct *' is incompatible with argument 1 of '__atomic_fetch_add') From: Mark Millard In-Reply-To: <9D40F38E-F1DC-4A3F-8792-09AD30D8802B@yahoo.com> Date: Wed, 25 Jul 2018 18:52:05 -0700 Cc: Konstantin Belousov , FreeBSD Current Content-Transfer-Encoding: quoted-printable Message-Id: References: <95fdbf29-6c11-77a6-27a3-2d0dc30f1668@FreeBSD.org> <788B1EE7-EFC9-4AD4-9FD1-9876D0121189@yahoo.com> <9D40F38E-F1DC-4A3F-8792-09AD30D8802B@yahoo.com> To: John Baldwin X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Jul 2018 01:52:20 -0000 On 2018-Jul-25, at 2:10 PM, Mark Millard wrote: > On 2018-Jul-25, at 10:09 AM, Mark Millard = wrote: >=20 >> On 2018-Jul-25, at 8:39 AM, John Baldwin wrote: >>=20 >>> On 7/24/18 11:39 PM, Mark Millard wrote: >>>> On 2018-Jul-24, at 10:32 PM, Mark Millard = wrote: >>>>=20 >>>>> https://ci.freebsd.org/job/FreeBSD-head-amd64-gcc/6597/consoleText >>>>> (head -r336573 after the prior 6596's -r336565 ): >>>>>=20 >>>>> --- all_subdir_lib/ofed --- >>>>> In file included from = /workspace/src/contrib/ofed/librdmacm/cma.h:43:0, >>>>> from /workspace/src/contrib/ofed/librdmacm/acm.c:42: >>>>> /workspace/src/contrib/ofed/librdmacm/cma.h: In function = 'fastlock_init': >>>>> /workspace/src/contrib/ofed/librdmacm/cma.h:60:2: error: invalid = initializer >>>>> atomic_store(&lock->cnt, 0); >>>>> ^ >>>>> In file included from = /workspace/src/contrib/ofed/librdmacm/acm.c:42:0: >>>>> /workspace/src/contrib/ofed/librdmacm/cma.h: In function = 'fastlock_acquire': >>>>> /workspace/src/contrib/ofed/librdmacm/cma.h:68:2: error: operand = type 'struct *' is incompatible with argument 1 of = '__atomic_fetch_add' >>>>> if (atomic_fetch_add(&lock->cnt, 1) > 0) >>>>> ^~ >>>>> /workspace/src/contrib/ofed/librdmacm/cma.h: In function = 'fastlock_release': >>>>> /workspace/src/contrib/ofed/librdmacm/cma.h:73:2: error: operand = type 'struct *' is incompatible with argument 1 of = '__atomic_fetch_sub' >>>>> if (atomic_fetch_sub(&lock->cnt, 1) > 1) >>>>> ^~ >>>>> . . . >>>>> --- all_subdir_lib/ofed --- >>>>> *** [acm.o] Error code 1 >>>>>=20 >>>>>=20 >>>>> https://ci.freebsd.org/job/FreeBSD-head-amd64-gcc/6621/consoleText = ( for >>>>> -r336700 ) still shows this type of error. >>>>=20 >>>>=20 >>>> [I should have a subject with "head -r336568 through -r336570 . . = .".] >>>>=20 >>>> =46rom what I can tell looking around having something like: >>>>=20 >>>> if (atomic_fetch_add(&lock->cnt, 1) > 0) >>>>=20 >>>> involve a __atomic_fetch_add indicates that: >>>>=20 >>>> = /usr/local/lib/gcc/x86_64-unknown-freebsd12.0/6.4.0/include/stdatomic.h >>>>=20 >>>> was in use instead of FreeBSD's stdatomic.h file. >>>>=20 >>>> If this is right, then the issue may be tied to head -r335782 >>>> implicitly changing the order of the include file directory >>>> searching for builds via the devel/*-gcc . >>>>=20 >>>> (I reverted -r335782 in my environment some time ago and have >>>> not run into this problem in my context so far.) >>>=20 >>> C11 atomics should work fine with compiler-provided headers since = they >>> are a part of the language (and the system stdatomic.h simply = attempts >>> to mimic the compiler-provided header in case it is missing). >>>=20 >>> Simple standalone tests of _Atomic(int) with GCC don't trigger those >>> failures when using its stdatomic.h, so there is probably something = else >>> going on with kernel includes being used while building the library, >>> etc. The last time we had this issue with stdarg.h it was because a >>> header shared between the kernel and userland always used = '' >>> which is correct for the kernel but not for userland. >>=20 >> I did misread the headers. FreeBSD has the likes of: >>=20 >> #if defined(__CLANG_ATOMICS) >> . . . >> #define atomic_fetch_add_explicit(object, operand, order) = \ >> __c11_atomic_fetch_add(object, operand, order) >> . . . >> #elif defined(__GNUC_ATOMICS) >> . . . >> #define atomic_fetch_add_explicit(object, operand, order) = \ >> __atomic_fetch_add(&(object)->__val, operand, order) >> . . . >> #endif >> . . . >> #define atomic_fetch_add(object, operand) = \ >> atomic_fetch_add_explicit(object, operand, memory_order_seq_cst) >>=20 >> so __atomic_fetch_add would occur. >>=20 >> But so far I do not see the problem with -r335782 reverted. I last = built >> -r336693 last night via devel/amd64-gcc (via xtoolchain). >>=20 >> =46rom what I can tell FreeBSD defines: >>=20 >> #if !defined(__CLANG_ATOMICS) >> #define _Atomic(T) struct { volatile T = __val; } >> #endif >>=20 >> and that struct is being used in &(object)->__val is what the >> error reports are about. So that would be, for example, >>=20 >> &(&lock->cnt)->__val >>=20 >> This would appear to suggest that __val itself had a type meeting: >>=20 >> operand type struct >>=20 >> for T in _Atomic(T) . >>=20 >> (This is independent of just what the issue traces back to: just >> the net result on ci.freebsd.org . No claim that you are right >> or wrong here. I'll not be looking any more until this afternoon >> or night.) >=20 > Going in a somewhat different direction . . . >=20 > Looking around I found https://bugs.llvm.org/show_bug.cgi?id=3D26462 > which is titled: >=20 > 26462 =E2=80=93 GCC/clang C11 _Atomic incompatibility >=20 > It appears that the normal source of platform ABI definitions are > not explicit/detailed in the area and allow for incompatibilities > in this area. clang and gcc made differing choices absent being > constrained to match. >=20 > An example (a powerpc64 context was indicated): >=20 > struct A16 { char val[16]; };=20 > _Atomic struct A16 a16;=20 > // GCC: > _Static_assert(_Alignof(a16) =3D=3D 16, "");=20 > // Clang: > _Static_assert(_Alignof(a16) =3D=3D 1, "");=20 >=20 >=20 > Non-power-of-2 is a general problem > (not a powerpc64 context from what I can > tell): >=20 > struct A3 { char val[3]; }; > _Atomic struct A3 a3; > // GCC: > _Static_assert(sizeof(a3) =3D=3D 3, ""); > _Static_assert(_Alignof(a3) =3D=3D 1, ""); > // Clang: > _Static_assert(sizeof(a3) =3D=3D 4, ""); > _Static_assert(_Alignof(a3) =3D=3D 4, ""); >=20 >=20 >=20 > Comment 6 (by John McCall) is relevant: >=20 > QUOTE > Anyway, while I prefer the Clang rule, the GCC rule is defensible, as = are any number of other rules. The important point, however, is that = having this discussion is not the right approach to solving this = problem. The layout of _Atomic(T) is ABI. ABI rules are not generally = determined by compiler implementors making things up as they go along, = or at least they shouldn't be. The Darwin ABI for _Atomic is the rule = implemented in Clang, which we actually did think about carefully when = we adopted it. Other platforms need to make their own call, and it = probably shouldn't just be "whatever's implemented in GCC", especially = on other platforms where GCC is not the system compiler. > END QUOTE >=20 >=20 > (I do nto claim to have proivided all the material that should > be read in https://bugs.llvm.org/show_bug.cgi?id=3D26462 .) >=20 > It may be that FreeBSD needs to be the source of the ABI definitions > involved if clang and gcc freeBSD builds are to be interoperable in > this area. But this could mean avoiding builtins? >=20 > If any of this is inlined and so not behind a more stable interface, > it looks like clang and gcc can not be mixed for the same instances > of various _Atomic possibilities. >=20 >=20 Going back to the code being compiled and=20 confirming your note: ( /usr/src/contrib/ofed/librdmacm/cma.h ) typedef struct { sem_t sem; _Atomic(int) cnt; } fastlock_t; . . . static inline void fastlock_acquire(fastlock_t *lock) { if (atomic_fetch_add(&lock->cnt, 1) > 0) sem_wait(&lock->sem); } So lock->cnt is an _Atomic(int) , i.e., struct { volatile int __val; } so overall: typedef struct { sem_t sem; struct { volatile int __val; } cnt; } fastlock_t; for which: atomic_fetch_add(&lock->cnt, 1) has for A filled-in in the C11 language official: atomic_fetch_add (volatile A* obj, M arg) (generic function) being: atomic_fetch_add (volatile struct { volatile int __val; }* obj, M arg) and a direct type-check of that notation for obj would find: operand type 'struct *' and that would propagate to GCC's translation to __atomic_fetch_add via gcc's stdatomic.h : #define atomic_fetch_add(PTR, VAL) __atomic_fetch_add ((PTR), (VAL), = \ __ATOMIC_SEQ_CST) So, yes, it looks like the: #if !defined(__CLANG_ATOMICS) #define _Atomic(T) struct { volatile T __val; } #endif overriding the GCC compiler's _Atomic(T) notation is inappropriate if one is trying for the GCC atomic ABI (via GCC's builtins). (Such is not the same as the clang atomic ABI for at least some T's on some architectures.) (But if always being compatible with clang's atomic ABI was a criteria, I'm not sure what things should be like when compiling via gcc.) =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)