From owner-freebsd-amd64@freebsd.org Mon Jul 30 11:56:59 2018 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 51E971053CA1; Mon, 30 Jul 2018 11:56:59 +0000 (UTC) (envelope-from elenamihailescu22@gmail.com) Received: from mail-oi0-x22e.google.com (mail-oi0-x22e.google.com [IPv6:2607:f8b0:4003:c06::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D8F6D95C00; Mon, 30 Jul 2018 11:56:58 +0000 (UTC) (envelope-from elenamihailescu22@gmail.com) Received: by mail-oi0-x22e.google.com with SMTP id q11-v6so20658373oic.12; Mon, 30 Jul 2018 04:56:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to:cc; bh=64uKS1lktvxfxsm1mHU7il6rxsNopLY9JC2V9LQz0Aw=; b=gw63Xk55yZ6BRFLEFfX1oFIaeQOMirfZEH3sp6JTVX7fZAzXJhVhzmBBFs8OHDHRNE 8S0X48T/mL+leXs8dT78IoCEdwzUeOPLlfHyYoiGwZlvDe2lM4ywO9zpPOjtps4vSM83 4FQyLy74rukQmtw3nSQKWfFNBSTIdEGm7FHpEDjHieTdgw9/indjnX9tyZMsPCirkJmi hNre5xAWFRPUDRxC92AEays9TKjBwfX+1IWy18oMD5n/j7xL1birEyrHatALWVrZlF1T KuNfcFeLyhJmXwI4ILZ6tFbOpIBRUXceVHYw+85qDja7XYgHscdFnO+7/t+xq0JH+/bl K7QA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=64uKS1lktvxfxsm1mHU7il6rxsNopLY9JC2V9LQz0Aw=; b=smvKzCglQsoEprIHGNoRCnFGtUOqsT1CDmyHHAqhSbgFlna1G4ThiE60xFWW+xDrCb 9pBTEPvbFRfHxT2WJLBVvRSYF1ruG5mNDmL6030P27XxmhNIswxRBFwbxPNTEme5noll c9SC+PNbBXgY1KOMwzfTKm1I0h6Q5JJcfRxNwr6jAlzR5xsUCFlyKIgeYfjHeDoxn+J/ M9bUraB6V0VnRGjbvP6OduSVWebOP06PYkyy3htIb0O3jjbvPPPziwk88VjY9qtseKmk ctOmSplaivA3INCBTxxS6wWirhlaSPYwhpe9JsIP+qdec5ce8lHRFrE3gjLxnJjV8wMu 8PNw== X-Gm-Message-State: AOUpUlENWPVgcjhck5s/BQt9HXxAYm1im4zZ7Gta4gLavJkFwlu9DN2u Vb+gPTQQ2qAvuhtXBry1P3QPnFkxZkVgpkUMd20CUw== X-Google-Smtp-Source: AAOMgpdA/mA45z9B5XuQm7vzaxGDJze9fOt9240GW3y1oiGBlamLbOrRihU4Ogpd9AtlxIPIjfPl+dd37Oj8h3HAnQU= X-Received: by 2002:aca:ed45:: with SMTP id l66-v6mr18550434oih.40.1532951817444; Mon, 30 Jul 2018 04:56:57 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a9d:3c4e:0:0:0:0:0 with HTTP; Mon, 30 Jul 2018 04:56:27 -0700 (PDT) From: Elena Mihailescu Date: Mon, 30 Jul 2018 14:56:27 +0300 Message-ID: Subject: bhyve internals related to FreeBSD memory mechanisms To: freebsd-virtualization@freebsd.org, freebsd-amd64@freebsd.org, freebsd-hackers@freebsd.org Cc: Mihai Carabas , Darius Mihai Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jul 2018 11:56:59 -0000 Hello, I'll start by giving some context before asking my questions. Currently, we are trying to implement a live migration feature for bhyve. In order to do that, we want to mark the guest memory copy-on-write. As we've previously discussed this problem on the freebsd-virtualization list [1], the vm_entry structure that contains the guest memory needs to have the MAP_ENTRY_COW and MAP_ENTRY_NEEDS_COPY flags set. The way the migration mechanism will work in "rounds" is described bellow (it is a pre-copy live migration technique): - in the first round, after starting the procedure and checking the compatibility of the two systems, the entire guest memory will be sent at the destination - in the second round, after the transfer is completed, only the differences (pages that were written/dirtied since the first round started) will be sent to the destination ... - in the n-th round, only the differences between this round and the round (n - 1) will be sent. Before the last round, the guest will be stopped, and the remaining memory will be sent along with the CPU state. We need the COW mechanism to determine changes between rounds. As for the number of rounds, there could be maximum 10 rounds. the value is set without running any benchmarks to limit potential overhead). The number of rounds will be decided later based on the test results. The number of new objects on the source will be also not very high since the number of steps is limited. After the migration process is completed, the guest from the source system will be destroyed and those new created object will be no longer needed (will be discarded). ---- As we could see by inspecting and debugging the bhyve code, the object (currently, we are using a 512MB bhyve guest, for more physical memory assigned to the virtual machine, there could be more objects) that describes the guest memory, is pointed to by two vm_entry structures from two different vmspace structures: - the first one is the vmspace that describes host's virtual memory - the second one is a separated vmspace structure created by the hypervisor when creating the virtual machine. I have several questions about the FreeBSD's memory management and bhyve's internals because I wasn't able to determine this yet by myself. The first one is that if someone knows whether that object that describes the guest memory is contained only by the two vm_entry structures, or whether it is contained by other entries. We could not find if it so or not. As far as I could tell, the COW can be set only for vm_entry structures. Is there a way to set as copy-on-write just certain pages or maybe just the object and not its vm_entry structure? I want to know if there is a more fine grain mechanism to set just parts of the memory as COW. We need a finer granularity when setting pages as copy-on-write because we encountered some issues: - virtio mechanisms are working by having a shared memory region between host and guest and while transferring the guest memory state, the pages that are involved in the virtio communication do not need to be set COW. - if we are trying to mark the vm_entry that contains the guest memory as COW from the host vmspace, the virtio devices will crash the guest eventually (some assertions about IOV and operation types will fail). We know that we should not set that memory as copy-on-write because it is not the way the guest sees its physical memory but, - if we set the vm_entry that contains the object with guest memory as COW from the dedicated vmspace created for the guest, the virtio devices will not fail assertions anymore, but after some time it seems that the guest filesystem is corrupted. Usually, we can start the guest normally after entering in single user mode and running fsck. Sometime, we need to install the virtual machine again. Also, after setting the vm_entry from the guest dedicated vmspace as COW, the two vm_entry will have different views of the guest memory: - the vm_entry from the guest dedicated vmspace will point to a new object (of course, after a first write access) - the vm_entry from the host vmspace that contains the guest memory will point to the old object that now has as backing object the new created object. Another question might be if it is ok to "change" the object from the host's vm_entry to point to the backing object. In this case, the two entries will point again to the same object. This might imply to remap/redo the references contained by the old object to point to the new object. [1] http://freebsd.1045724.x6.nabble.com/Inspect-pages-created-after-a-vm-object-is-marked-as-copy-on-write-td6266552.html Thank you, Elena From owner-freebsd-amd64@freebsd.org Tue Jul 31 20:36:19 2018 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D70CD10644FE for ; Tue, 31 Jul 2018 20:36:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 73ECE7BCDD for ; Tue, 31 Jul 2018 20:36:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 38E4F10644FC; Tue, 31 Jul 2018 20:36:19 +0000 (UTC) Delivered-To: amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 27CD410644FB for ; Tue, 31 Jul 2018 20:36:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BDF7B7BCD7 for ; Tue, 31 Jul 2018 20:36:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 0FC7913010 for ; Tue, 31 Jul 2018 20:36:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w6VKaHmZ020845 for ; Tue, 31 Jul 2018 20:36:17 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w6VKaHO5020844 for amd64@FreeBSD.org; Tue, 31 Jul 2018 20:36:17 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: amd64@FreeBSD.org Subject: [Bug 179282] [amd64] [patch] add Intel SMAP for FreeBSD-CURRENT Date: Tue, 31 Jul 2018 20:36:17 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: amd64 X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: emaste@freebsd.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: amd64@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jul 2018 20:36:20 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D179282 --- Comment #8 from Ed Maste --- ifunc implementation committed in r336876 --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-amd64@freebsd.org Wed Aug 1 01:46:36 2018 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 48050106C6D5 for ; Wed, 1 Aug 2018 01:46:36 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic307-12.consmr.mail.ne1.yahoo.com (sonic307-12.consmr.mail.ne1.yahoo.com [66.163.190.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6A23D8DC5E for ; Wed, 1 Aug 2018 01:46:35 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: AmAk3AMVM1nj3.KdGSoy.C9GIdTViRbWmwHXGTRvVJC9UHGZjxDciEk7XvVnINx qfGlKAB__xqm5pdM.ta3aE.jI3k94Y.JQqnGkUf7sZsUPqmtrvN1xzv8Cz4zAfO036BOkT_4slX7 ylBd5mkcxAx3R8Bjs7YbLHLLlzrqv0OKqKG_lbFpmPSm4thkgtDSU2OM9eY6ibliXHUT9D3wqgeQ 76j8frSPT3fULoWYPJsMu5gtMjUZFO9LkNu4I4VKH1scc9KCcLu4p.JujHSTx.rf919xGYKYm2v5 bFSKs_c1doFVhBtcCE3pyW79lugMcG7qtMgtUffBCfr7eND.PrpVeGZ.aL8x8HvphGylTYS_jP2B ll2JXi0heXS2qY6YxLRwfOYcW9V5Ci5RxZfXp39qoqQk8cN6MpES4_XFSBtpF3lenUUSNWqTB_7E Qme61qGEZ2dRYI1bA.BKyIBbQvYWrnaiKXwrsQXVcukODguDzHN_CsukS6IM5rWTK4CFbm5Us0s6 qdCdJp6LBYCmTJ0bxM4.iVOJGuao04g2NEUDODiTXisZlcgFUmeDw6f6m9JBS2rF0lfhChhF6.SP Bjv2LtHnWSR6fPmwqwpyI23XAABF1V_RYaJ_PNPnzBiFy1SHetalDVPVTeGu7Gq9yGdKC4.kwFPA 4vLgWpthEBOixOqrh9IdjxV2a1S91LIM5OktplkLxc6ETzvb70ekSkPrAVK0tumJBwIRqqmfEpLZ 3QRq7n3aSihq_SFUfjxoZGZ2lty6.ZueBDI4VDJeIwnJ4BcW8SqxElEfwH5OQT.KSWVGow8yEMf1 OttbB0vp5iNczy1fHUM29Ya2D1M8VQN83pQ2fKTUQL5_Nnp9miBm9gDiS.NUq05B709CVyyMXTWI 35bScd68dluT3DRCX1ClJOf7xhahXROZ5lVKHIvdiPv7Jz5jmxCCeKz7B_Ynv.SOau.pZalpC5yb 4.zmv8xmwV29nJM9zFncTCGxjiV_UrGT44066BDRONPwEy1BN3CMzTsNMPvCaVQydVXlJi3j5Vl5 LxnYEEjGQByPOVrFK Received: from sonic.gate.mail.ne1.yahoo.com by sonic307.consmr.mail.ne1.yahoo.com with HTTP; Wed, 1 Aug 2018 01:46:34 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp425.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 8038f0495d12a8775d8b876066527fc5; Wed, 01 Aug 2018 01:46:33 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: svn commit: r335873 - in head: . sys/amd64/amd64 sys/amd64/include sys/conf sys/i386/i386 sys/i386/include sys/sys sys/vm Message-Id: <226713AF-FC51-4A44-93E1-8264B1BD2C8A@yahoo.com> Date: Tue, 31 Jul 2018 18:46:31 -0700 To: mmacy@FreeBSD.org, svn-src-head@freebsd.org, freebsd-hackers@freebsd.org, freebsd-amd64@freebsd.org X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Aug 2018 01:46:36 -0000 > Author: mmacy > Date: Mon Jul 2 19:48:38 2018 > New Revision: 335873 > URL:=20 > https://svnweb.freebsd.org/changeset/base/335873 >=20 >=20 > Log: > inline atomics and allow tied modules to inline locks > =20 > - inline atomics in modules on i386 and amd64 (they were always > inline on other arches) > - allow modules to opt in to inlining locks by specifying > MODULE_TIED=3D1 in the makefile I recently found the following about ABI incompatibilities between clang and gcc relative to C11 language based atomics: https://bugs.llvm.org/show_bug.cgi?id=3D26462 26462 =E2=80=93 GCC/clang C11 _Atomic incompatibility So are there implications about building the kernel vs. modules that overall mix the toolchains once modules are loaded? Do the toolchains need to match, at least for amd64 and i386 TARGET_ARCH 's? For reference as an introduction to the material in llvm's 26462 . . . It appears that the normal source of platform ABI definitions are not explicit/detailed in the area and allow for incompatibilities in this area. clang and gcc made differing choices absent being constrained to match. An example (a powerpc64 context was indicated): struct A16 { char val[16]; };=20 _Atomic struct A16 a16;=20 // GCC: _Static_assert(_Alignof(a16) =3D=3D 16, "");=20 // Clang: _Static_assert(_Alignof(a16) =3D=3D 1, "");=20 Non-power-of-2 is a general problem (not a powerpc64 context from what I can tell): struct A3 { char val[3]; }; _Atomic struct A3 a3; // GCC: _Static_assert(sizeof(a3) =3D=3D 3, ""); _Static_assert(_Alignof(a3) =3D=3D 1, ""); // Clang: _Static_assert(sizeof(a3) =3D=3D 4, ""); _Static_assert(_Alignof(a3) =3D=3D 4, ""); Comment 6 (by John McCall) is relevant: QUOTE Anyway, while I prefer the Clang rule, the GCC rule is defensible, as = are any number of other rules. The important point, however, is that = having this discussion is not the right approach to solving this = problem. The layout of _Atomic(T) is ABI. ABI rules are not generally = determined by compiler implementors making things up as they go along, = or at least they shouldn't be. The Darwin ABI for _Atomic is the rule = implemented in Clang, which we actually did think about carefully when = we adopted it. Other platforms need to make their own call, and it = probably shouldn't just be "whatever's implemented in GCC", especially = on other platforms where GCC is not the system compiler. END QUOTE =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-amd64@freebsd.org Wed Aug 1 07:57:16 2018 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1299A105CF57; Wed, 1 Aug 2018 07:57:16 +0000 (UTC) (envelope-from kib@freebsd.org) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 86430871C4; Wed, 1 Aug 2018 07:57:15 +0000 (UTC) (envelope-from kib@freebsd.org) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w717v4hI067904 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 1 Aug 2018 10:57:07 +0300 (EEST) (envelope-from kib@freebsd.org) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w717v4hI067904 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w717v4iV067903; Wed, 1 Aug 2018 10:57:04 +0300 (EEST) (envelope-from kib@freebsd.org) X-Authentication-Warning: tom.home: kostik set sender to kib@freebsd.org using -f Date: Wed, 1 Aug 2018 10:57:04 +0300 From: Konstantin Belousov To: Mark Millard Cc: mmacy@FreeBSD.org, svn-src-head@freebsd.org, freebsd-hackers@freebsd.org, freebsd-amd64@freebsd.org Subject: Re: svn commit: r335873 - in head: . sys/amd64/amd64 sys/amd64/include sys/conf sys/i386/i386 sys/i386/include sys/sys sys/vm Message-ID: <20180801075704.GS40119@kib.kiev.ua> References: <226713AF-FC51-4A44-93E1-8264B1BD2C8A@yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <226713AF-FC51-4A44-93E1-8264B1BD2C8A@yahoo.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-Mailman-Approved-At: Wed, 01 Aug 2018 11:28:30 +0000 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Aug 2018 07:57:16 -0000 On Tue, Jul 31, 2018 at 06:46:31PM -0700, Mark Millard via freebsd-amd64 wrote: > > Author: mmacy > > Date: Mon Jul 2 19:48:38 2018 > > New Revision: 335873 > > URL: > > https://svnweb.freebsd.org/changeset/base/335873 > > > > > > Log: > > inline atomics and allow tied modules to inline locks > > > > - inline atomics in modules on i386 and amd64 (they were always > > inline on other arches) > > - allow modules to opt in to inlining locks by specifying > > MODULE_TIED=1 in the makefile > > I recently found the following about ABI incompatibilities > between clang and gcc relative to C11 language based > atomics: > > https://bugs.llvm.org/show_bug.cgi?id=26462 > > 26462 ??? GCC/clang C11 _Atomic incompatibility > > > So are there implications about building the kernel > vs. modules that overall mix the toolchains once > modules are loaded? Do the toolchains need to match, > at least for amd64 and i386 TARGET_ARCH 's? This is irrelevant since kernel does not use C11 atomics, we roll our own version, which tries to follow C11 model. Lack of the ABI for atomics is one of the reason to not use compiler C11 atomics in kernel and C runtime. From owner-freebsd-amd64@freebsd.org Wed Aug 1 14:41:59 2018 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 389AC105F216 for ; Wed, 1 Aug 2018 14:41:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id BB425774D9 for ; Wed, 1 Aug 2018 14:41:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 6CEC5105F212; Wed, 1 Aug 2018 14:41:58 +0000 (UTC) Delivered-To: amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 570AA105F211 for ; Wed, 1 Aug 2018 14:41:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D6B30774D5 for ; Wed, 1 Aug 2018 14:41:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id E07B51C72D for ; Wed, 1 Aug 2018 14:41:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w71EfuZB089157 for ; Wed, 1 Aug 2018 14:41:56 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w71EfuLm089156 for amd64@FreeBSD.org; Wed, 1 Aug 2018 14:41:56 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: amd64@FreeBSD.org Subject: [Bug 212539] readelf -w consumes a lot of memory with GCC-compiled kernels Date: Wed, 01 Aug 2018 14:41:56 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: emaste@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Aug 2018 14:41:59 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D212539 --- Comment #1 from Ed Maste --- Which readelf is this (GNU or ELF Tool Chain)? --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-amd64@freebsd.org Wed Aug 1 15:46:04 2018 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 25AC0106146B for ; Wed, 1 Aug 2018 15:46:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id B5EF47A5AB for ; Wed, 1 Aug 2018 15:46:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 774FF1061468; Wed, 1 Aug 2018 15:46:03 +0000 (UTC) Delivered-To: amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6602A1061467 for ; Wed, 1 Aug 2018 15:46:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 07BE07A5A9 for ; Wed, 1 Aug 2018 15:46:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 35B0A1CFDB for ; Wed, 1 Aug 2018 15:46:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w71Fk2Bg079075 for ; Wed, 1 Aug 2018 15:46:02 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w71Fk2WE079074 for amd64@FreeBSD.org; Wed, 1 Aug 2018 15:46:02 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: amd64@FreeBSD.org Subject: [Bug 212539] readelf -w consumes a lot of memory with GCC-compiled kernels Date: Wed, 01 Aug 2018 15:46:02 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: markj@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Aug 2018 15:46:04 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D212539 --- Comment #2 from Mark Johnston --- It's elftoolchain. The issue is still reproducible with at least gcc 6. --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-amd64@freebsd.org Fri Aug 3 21:13:09 2018 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 682A41051007 for ; Fri, 3 Aug 2018 21:13:09 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-21.consmr.mail.gq1.yahoo.com (sonic317-21.consmr.mail.gq1.yahoo.com [98.137.66.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E730D7587C for ; Fri, 3 Aug 2018 21:13:08 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: 4WSHpvUVM1l7OpOTmJW3nxJmARMzGOhHWSWrA.d7Ww98uI.TKeEMGwToYb_Tvjj MBs1NkbV9r4zoWnt1rWlKm0FhLMt1Kqzvywq7tvrSb_QnX7AoxsKN4IMubB8Uz4W0ZtW.CdD8XkL jhIu4c7RP1VlM8JeQxwGdXEXJfJFBZN.y3DTliALakz724iZt6IabC_5MA8Y_jkNKWD6sqenI0wW uN1utFoLaoAW.diaGhX4tukuZNzutq0AxCabGYHplztxWCJJmWRWAGKpEHa2EsuxT873Gc7BJQrA bhDs0dKbdjUPCTAQ_RUPKFhkgTynqGMOOeODH7Mr11dRKKAriUR7FmT8e0r.4DaFMs_O8ZO8Fo.m RkR9d2RxhhSZw4cKmXrxUrduZEG9fG4GcYeErylxMgyyZhgP10NLYJEzjSmLtupZT1wwrn.EB61v limR_ZkETv4Dm6jJwlJ_McNvNP_SNxF.xEGn5OY30bednnj22Qyp5zQlsKFGQo25NiYCOiscweLl d7JriiHNOu2s3UcGbPuirzVeZTddCaaDdvQv7FVSdseGXOva3tzhdCzjfhLUbT9_rLDwIfpKVY1B acvachwn8N0VW3Z6XUZYO5Uypbn7W02oONgepVU3Cv8Quy9_sGSu2fmC9oP5OaPwxbjARHOnLmnm qokRjfyOjmsOgpWchPZRcJ3dfk0pF2WG05clwb0Oc2z9CchBr78KiezHgLdAENeFXl8FtXRxFXKT .Ydrslm0TlFSPKcwzHbOPG.QfRGCG.lhTApaMnXCxuVqWtkTuaWlLdC.fKcaXgvUh0URE9rPfZWk xAa0fSNB9J21eWRemM15MaMxsTVJTLGDm8kSYCN.XAvJSmiZsqUNEXwqtbmsM7OiVzjnNA8fvRqf uqm_dik4AwDUucZk5PULvGcQX_uz9ucXY0xQYcDqzdri8MsvH1co1IIZeu1WV7Ne3q1abdec0JME sOPSZwa1rxVzqtW2gwriSCDfwl0HLL7If1WJjPpzwZ0gx6kaJDCYKnhno4LIHVqmPhyvuN4lXPqJ gdf2plRAEgbwGEfo79g-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.gq1.yahoo.com with HTTP; Fri, 3 Aug 2018 21:13:01 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp412.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 37df12fa32bf2493899491448ee57466; Fri, 03 Aug 2018 21:12:56 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: svn commit: r335873 - in head: . sys/amd64/amd64 sys/amd64/include sys/conf sys/i386/i386 sys/i386/include sys/sys sys/vm From: Mark Millard In-Reply-To: <20180801075704.GS40119@kib.kiev.ua> Date: Fri, 3 Aug 2018 14:12:55 -0700 Cc: mmacy@FreeBSD.org, svn-src-head@freebsd.org, freebsd-hackers@freebsd.org, freebsd-amd64@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <3910BFAA-45CF-4A5F-AF08-A47585BFF9C2@yahoo.com> References: <226713AF-FC51-4A44-93E1-8264B1BD2C8A@yahoo.com> <20180801075704.GS40119@kib.kiev.ua> To: Konstantin Belousov X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Aug 2018 21:13:09 -0000 On 2018-Aug-1, at 12:57 AM, Konstantin Belousov = wrote: >=20 > On Tue, Jul 31, 2018 at 06:46:31PM -0700, Mark Millard via = freebsd-amd64 wrote: >>> Author: mmacy >>> Date: Mon Jul 2 19:48:38 2018 >>> New Revision: 335873 >>> URL:=20 >>> https://svnweb.freebsd.org/changeset/base/335873 >>>=20 >>>=20 >>> Log: >>> inline atomics and allow tied modules to inline locks >>>=20 >>> - inline atomics in modules on i386 and amd64 (they were always >>> inline on other arches) >>> - allow modules to opt in to inlining locks by specifying >>> MODULE_TIED=3D1 in the makefile >>=20 >> I recently found the following about ABI incompatibilities >> between clang and gcc relative to C11 language based >> atomics: >>=20 >> https://bugs.llvm.org/show_bug.cgi?id=3D26462 >>=20 >> 26462 ??? GCC/clang C11 _Atomic incompatibility >>=20 >>=20 >> So are there implications about building the kernel >> vs. modules that overall mix the toolchains once >> modules are loaded? Do the toolchains need to match, >> at least for amd64 and i386 TARGET_ARCH 's? > This is irrelevant since kernel does not use C11 atomics, we roll > our own version, which tries to follow C11 model. >=20 > Lack of the ABI for atomics is one of the reason to not use compiler > C11 atomics in kernel and C runtime. Thanks for the information: good to know. There are other things around that use C11 atomics, such as in: src/contrib/ofed/librdmacm/ (This is part of why https://reviews.freebsd.org/D16585 is out for updating stdatomic.h (and sys/cdefs.h ): so that fairly modern gcc vintages can build world with such involved.) =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)