From owner-freebsd-arch@FreeBSD.ORG Mon Nov 3 16:48:55 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A570EB5E for ; Mon, 3 Nov 2014 16:48:55 +0000 (UTC) Received: from mail-ie0-f173.google.com (mail-ie0-f173.google.com [209.85.223.173]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 712F0A3E for ; Mon, 3 Nov 2014 16:48:55 +0000 (UTC) Received: by mail-ie0-f173.google.com with SMTP id tr6so5660880ieb.32 for ; Mon, 03 Nov 2014 08:48:49 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:from:content-type:subject:message-id:date :to:mime-version; bh=+BQzq9iyLo0ZORqL3l8sz6OY6Xve5f/LEDkG/X/E3HE=; b=iXwpKLKNvl2O9rx9X8zJPJJjI8wESTPPd+sAG9GBW65AHIC9abw0sQ8KuK2Boj7twf 3ZeHDGGcgac4XCEwBQ9x8vHF7Vq4Y9XIWmfFfbZ7JLHbToc935asODFLH3shaSUjfv6n tw5iEvADrsOt+A17UWR2d5lSRewuVwZyrMirQYDu/qc7Hzt0AU1C4M57ISQIwxBvL5NC KD4f4irOv7xoEg/al7tkIFvhVK/rnzPn7KQmHPM1ybsDnW8QMIJHbQMKZmMvapeU/CEA h0XUB4VGdKs8jx/4X40972VtduR2F3W+Iyzznj3SyYTazP2fR017INDuvG240tiHhmTJ gzsg== X-Gm-Message-State: ALoCoQkZvDZNP619B2uaJ4vF6CW3wJ7wiXrCAWeL+tfl2sgX6zwgKC3T2AqwWjGlmlYGg00b9Eko X-Received: by 10.50.3.67 with SMTP id a3mr17399742iga.42.1415033329235; Mon, 03 Nov 2014 08:48:49 -0800 (PST) Received: from bsdimp.bsdimp.com ([50.253.99.174]) by mx.google.com with ESMTPSA id xb4sm3873569igc.11.2014.11.03.08.48.48 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 03 Nov 2014 08:48:48 -0800 (PST) Sender: Warner Losh From: Warner Losh Content-Type: multipart/signed; boundary="Apple-Mail=_FAE250A3-3FB8-462F-ADD5-A8E277D90B05"; protocol="application/pgp-signature"; micalg=pgp-sha512 Subject: Why do we have @ in modules builds? Message-Id: <3285BC54-05D8-41DB-88FE-BAD681A3E45B@bsdimp.com> Date: Mon, 3 Nov 2014 09:48:46 -0700 To: "freebsd-arch@freebsd.org" Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) X-Mailer: Apple Mail (2.1878.6) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Nov 2014 16:48:55 -0000 --Apple-Mail=_FAE250A3-3FB8-462F-ADD5-A8E277D90B05 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Does anybody recall why we have @ symlink in our module builds? I=92m constantly working around issues that this creates. Maybe it is = time to eliminate it? So I=92ve posted the following review: https://reviews.freebsd.org/D1100 Warner --Apple-Mail=_FAE250A3-3FB8-462F-ADD5-A8E277D90B05 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJUV7HuAAoJEGwc0Sh9sBEA/0wQAJvlaLnP1fUpdJ6EJn57+H1Z d8o6cafZvPdW3zGTwoOuHkofCr5c8QF7WTfq2Xt3/RKo1pn7tyHBsv1xWgUdyQao qwpKie1Dka9FiNK8F8Qng7nTKnjNS/27jmVE9wSB1WqM4aaeGK+rQvSgY0NHI/Ul npspOBoQ1qC9Do9TBJfp39btdkGnBWzVDchtEQ9ufJgXmLi6FLrgnRhSWcEwtdmA 083Oltm4Gb34gBxg+ELQwi4s1Xsmu5s7f3DWM/OaBCroTSARh7/w4IceKqIYGgwx Envg86uHkny7JnlcPVSoEEc8VhEG0DW2e0HfG6/0Aimo//DCqrKbZwMfVotiMIiz nrh3LUOQQ9NKplgGNpHIa8p5GfYAjdE2Mm4uStfvYtwKFcoo/7TH/C7bt9QLCGIp 17+GYJhYYGozQF+CvfZlJRn6lkdFCKDHA7UtK6lSx+FEzjG3DUz7T0WdbODMN69i R5zYSBAFadmqmne6mhPpOXU9rmhDZaWHxZ9fbPY3YKv1//VTvJpKZG+fWFDPKT9W bL5cIAgg4j+8MrZESffCXG5csPc1Vyyl2zhzYKvYcG7K+WDW62eOjlR9S7c5YymB 3boHMyv4/eqOUd04wyYsypk6+QWBES0PXIx9qJsnJbwHJx2HSB8lqiHOQBY0ToVM b8OiuVvD/fWmNy29XEqX =82L5 -----END PGP SIGNATURE----- --Apple-Mail=_FAE250A3-3FB8-462F-ADD5-A8E277D90B05-- From owner-freebsd-arch@FreeBSD.ORG Mon Nov 3 19:09:06 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B1852E4C for ; Mon, 3 Nov 2014 19:09:06 +0000 (UTC) Received: from elvis.mu.org (elvis.mu.org [IPv6:2001:470:1f05:b76::196]) by mx1.freebsd.org (Postfix) with ESMTP id 9F639BEF for ; Mon, 3 Nov 2014 19:09:06 +0000 (UTC) Received: from AlfredMacbookAir.local (unknown [129.253.54.225]) by elvis.mu.org (Postfix) with ESMTPSA id 42613341F83D; Mon, 3 Nov 2014 11:09:06 -0800 (PST) Message-ID: <5457D2D0.8080201@freebsd.org> Date: Mon, 03 Nov 2014 11:09:04 -0800 From: Alfred Perlstein Organization: FreeBSD User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: freebsd-arch@freebsd.org, Warner Losh Subject: Re: Why do we have @ in modules builds? References: <3285BC54-05D8-41DB-88FE-BAD681A3E45B@bsdimp.com> In-Reply-To: <3285BC54-05D8-41DB-88FE-BAD681A3E45B@bsdimp.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Nov 2014 19:09:06 -0000 On 11/3/14, 8:48 AM, Warner Losh wrote: > Does anybody recall why we have @ symlink in our module builds? > I’m constantly working around issues that this creates. Maybe it is time > to eliminate it? > > So I’ve posted the following review: https://reviews.freebsd.org/D1100 > > Warner > Thank you!!!! I hate the '@' thing as well, I figured it was just due to someone not wanting 'grep -r' to work. :) Just to check, after your changes... will you still be able to do something like: cd $HOME svn co https://.../base/head/sys/modules/foo_module cd foo_module make depend all install -Alfred From owner-freebsd-arch@FreeBSD.ORG Mon Nov 3 19:22:29 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 294F9495 for ; Mon, 3 Nov 2014 19:22:29 +0000 (UTC) Received: from mail-ig0-f178.google.com (mail-ig0-f178.google.com [209.85.213.178]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E4239D87 for ; Mon, 3 Nov 2014 19:22:28 +0000 (UTC) Received: by mail-ig0-f178.google.com with SMTP id a13so5203918igq.17 for ; Mon, 03 Nov 2014 11:22:22 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:content-type:mime-version:subject:from :in-reply-to:date:cc:message-id:references:to; bh=oL4nnS/ltdq8jLyHe2XJrOESI+9JiOxasAK3Uoj1eis=; b=OwZFyU0IHtk7DnG+3YZObsjMqo5xa6CI758W65s+Piwm+YTdrCc3ivIKMtmQpkNVZM 1gEDPtfcGMbgGzx5RjZRhgx3mo7Old9q5fetiGcBGFVcodBHMTaRuBjq8ArPcKG0F9w5 CM20ML2HyWFaW3IeJSl5ciRKjKCqCOdOfTVAeJPLYgMFUt4kV5CK9hiHc7bpzpqw8Ae2 EOI8oC6TUHu/jH/xEBxIyZTrILy3phK7fpyBW+09Jzqjv1snZuCsQ6h2Hx5pDHF1mDFB cfD92JTwU55LjiLD63Yel7s/t7DsmkT8gUT/qVrignsldWEtr9gJejOO3MJiLRMlpJ4/ 0aUA== X-Gm-Message-State: ALoCoQnM7JZOXIHHGrJnxo1Yq92nLzg5ajCg7CLcPGlrmXlu0e6W+7JFu+5ubegVOdIAGMJQqSS4 X-Received: by 10.50.66.179 with SMTP id g19mr18564592igt.8.1415042542654; Mon, 03 Nov 2014 11:22:22 -0800 (PST) Received: from bsdimp.bsdimp.com ([50.253.99.174]) by mx.google.com with ESMTPSA id fm2sm4045240igb.6.2014.11.03.11.22.21 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 03 Nov 2014 11:22:21 -0800 (PST) Sender: Warner Losh Content-Type: multipart/signed; boundary="Apple-Mail=_7256C275-0A51-41AC-AD83-8DF8975DEEE3"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Why do we have @ in modules builds? From: Warner Losh In-Reply-To: <5457D2D0.8080201@freebsd.org> Date: Mon, 3 Nov 2014 12:22:20 -0700 Message-Id: <64C28D7A-AD81-4990-B95C-47E81C0E4F0A@bsdimp.com> References: <3285BC54-05D8-41DB-88FE-BAD681A3E45B@bsdimp.com> <5457D2D0.8080201@freebsd.org> To: Alfred Perlstein X-Mailer: Apple Mail (2.1878.6) Cc: freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Nov 2014 19:22:29 -0000 --Apple-Mail=_7256C275-0A51-41AC-AD83-8DF8975DEEE3 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Nov 3, 2014, at 12:09 PM, Alfred Perlstein = wrote: >=20 > On 11/3/14, 8:48 AM, Warner Losh wrote: >> Does anybody recall why we have @ symlink in our module builds? >> I=92m constantly working around issues that this creates. Maybe it is = time >> to eliminate it? >>=20 >> So I=92ve posted the following review: = https://reviews.freebsd.org/D1100 >>=20 >> Warner >>=20 > Thank you!!!! I hate the '@' thing as well, I figured it was just due = to someone not wanting 'grep -r' to work. :) >=20 > Just to check, after your changes... will you still be able to do = something like: >=20 > cd $HOME > svn co https://.../base/head/sys/modules/foo_module > cd foo_module > make depend all install It should behave the same it does today. I=92m replacing a ln -s = ${SYSDIR} @ with an in-line replacement of @ with ${SYSDIR}. Warner --Apple-Mail=_7256C275-0A51-41AC-AD83-8DF8975DEEE3 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJUV9XsAAoJEGwc0Sh9sBEA2zwP/i+BufKZOu96NpBHK4KfCXSA S+DjNWVvyJXXmK9CUIg85BCrH+uP112mY4P5yMSouicz00Aw1Z87WAsXqIbeR3Ae UJ0AWnFAjfJBzY2l7RW6KT9PbNqEGjUklpCrDbpWS/QAQvFL9q9WUerXXocm+NRW BZk3YoJZFssVepBTkT2BcWxi0obPIlN8dX0I+tcaMDKoPy3XmYJlV2i5g0+sPLh7 g1gKYTJthdSu6QLjP3gbAcdqJjWovBaAlxoeQrdKujJFL1oJ0IcAJucFUcgDc8bC 8fqKsk7vlP/Hf004ppyBBINePVYlR74e3XaGszAMLq+PopTXZkCptDA3djoN/ZB/ 7bD+d6ol55WGRw0/V6Imc7r7apKv6Ie/RD4EHYZMZZcjDA06gjqnLsPHczLNoalb Y5dK5RX4KuU3qO/408DAWYhlYlvkhBc90sM046p2p3TyK8rAFag0I7A8t690Ot2Y gbROrfVxNkO0ZhZpalT/W0v4lGowaJVDQ8DKR/P2znbhEc5NG2VD8CCN75bntBLm yilXMk/zC1DWTt15In7kouKWDbZxl0c4q0eaTkdCWzxTt0TxVwlwN6JMf77+8YaW y8RTt5/SwsAyrA+zI5xarZryvnof5dQWiggLrusOF4QBPoIlwZ+MAXoGYwmFBM72 vrZUgWZzIZ2YnDRKEeV/ =cEsK -----END PGP SIGNATURE----- --Apple-Mail=_7256C275-0A51-41AC-AD83-8DF8975DEEE3-- From owner-freebsd-arch@FreeBSD.ORG Mon Nov 3 19:30:01 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4549B80C for ; Mon, 3 Nov 2014 19:30:01 +0000 (UTC) Received: from elvis.mu.org (elvis.mu.org [IPv6:2001:470:1f05:b76::196]) by mx1.freebsd.org (Postfix) with ESMTP id 31773DE7 for ; Mon, 3 Nov 2014 19:30:01 +0000 (UTC) Received: from AlfredMacbookAir.local (unknown [129.253.54.225]) by elvis.mu.org (Postfix) with ESMTPSA id 0CBFE341F84E; Mon, 3 Nov 2014 11:30:01 -0800 (PST) Message-ID: <5457D7B7.5050503@freebsd.org> Date: Mon, 03 Nov 2014 11:29:59 -0800 From: Alfred Perlstein Organization: FreeBSD User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: freebsd-arch@freebsd.org, Warner Losh Subject: Re: Why do we have @ in modules builds? References: <3285BC54-05D8-41DB-88FE-BAD681A3E45B@bsdimp.com> <5457D2D0.8080201@freebsd.org> <64C28D7A-AD81-4990-B95C-47E81C0E4F0A@bsdimp.com> In-Reply-To: <64C28D7A-AD81-4990-B95C-47E81C0E4F0A@bsdimp.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Nov 2014 19:30:01 -0000 On 11/3/14, 11:22 AM, Warner Losh wrote: > On Nov 3, 2014, at 12:09 PM, Alfred Perlstein wrote: > >> On 11/3/14, 8:48 AM, Warner Losh wrote: >>> Does anybody recall why we have @ symlink in our module builds? >>> I’m constantly working around issues that this creates. Maybe it is time >>> to eliminate it? >>> >>> So I’ve posted the following review: https://reviews.freebsd.org/D1100 >>> >>> Warner >>> >> Thank you!!!! I hate the '@' thing as well, I figured it was just due to someone not wanting 'grep -r' to work. :) >> >> Just to check, after your changes... will you still be able to do something like: >> >> cd $HOME >> svn co https://.../base/head/sys/modules/foo_module >> cd foo_module >> make depend all install > It should behave the same it does today. I’m replacing a ln -s ${SYSDIR} @ with an in-line replacement of @ with ${SYSDIR}. Excellent! I did a history check to see why these came to be and it looks like it was just an oversight to use "@ links" instead of just SYSDIR as far as I can tell... -Alfred > Warner > From owner-freebsd-arch@FreeBSD.ORG Mon Nov 3 20:14:29 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 83C3226F for ; Mon, 3 Nov 2014 20:14:29 +0000 (UTC) Received: from mail-ie0-f178.google.com (mail-ie0-f178.google.com [209.85.223.178]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 497D9352 for ; Mon, 3 Nov 2014 20:14:29 +0000 (UTC) Received: by mail-ie0-f178.google.com with SMTP id rp18so6112195iec.9 for ; Mon, 03 Nov 2014 12:14:22 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:content-type:mime-version:subject:from :in-reply-to:date:cc:message-id:references:to; bh=0g1hXMbkvaL9fxtCxnLyrwsio4Rw2EXw44GPvKQcfWE=; b=MuYznL6gC+zSDRSpK9UXtQi96tLQMB9eTFbGQl9/FusbmG154ijvrUrvK3E0RaEieV +fk8Q+L7K7BWlHnolx67a/h+59lCqE3Q33RxfgrRg06kKVavcbxU6xVElBwVYB6IYZtj fW3MgMpK+AeLWfDKncuZEDxiJ0LPTbYjCVTXALtdjAMvqKw76qYNfYa3bOup+khw0Jtr 3da7Qnmf2uYauw3CZQucPsJKzsWBYxL88EzKS8TtyNG4iIJAnUoGoW2296eozQSUpG9k +Xr4o3CGgX2OX3ax6r/pFHEt7ePFoCu1byh1fYO4K9/t8tRiNkUo/0YfGAOak22hRGq+ Cj4g== X-Gm-Message-State: ALoCoQkCAKh/c+tfz8u6I5XQjuZXphcHJUxoXsNqAtPRyxwUu3nP/wTIp5YNdg9AqcUMV7+UB3GW X-Received: by 10.50.29.71 with SMTP id i7mr19166125igh.4.1415045661956; Mon, 03 Nov 2014 12:14:21 -0800 (PST) Received: from bsdimp.bsdimp.com ([50.253.99.174]) by mx.google.com with ESMTPSA id jd6sm4091455igb.16.2014.11.03.12.14.20 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 03 Nov 2014 12:14:21 -0800 (PST) Sender: Warner Losh Content-Type: multipart/signed; boundary="Apple-Mail=_5681B2AB-E1D3-4F11-83CD-499C1B8C4074"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Why do we have @ in modules builds? From: Warner Losh In-Reply-To: <5457D7B7.5050503@freebsd.org> Date: Mon, 3 Nov 2014 13:14:18 -0700 Message-Id: <9A113544-16B9-4C6E-8AC0-BF718C47A243@bsdimp.com> References: <3285BC54-05D8-41DB-88FE-BAD681A3E45B@bsdimp.com> <5457D2D0.8080201@freebsd.org> <64C28D7A-AD81-4990-B95C-47E81C0E4F0A@bsdimp.com> <5457D7B7.5050503@freebsd.org> To: Alfred Perlstein X-Mailer: Apple Mail (2.1878.6) Cc: freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Nov 2014 20:14:29 -0000 --Apple-Mail=_5681B2AB-E1D3-4F11-83CD-499C1B8C4074 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Nov 3, 2014, at 12:29 PM, Alfred Perlstein = wrote: >=20 > On 11/3/14, 11:22 AM, Warner Losh wrote: >> On Nov 3, 2014, at 12:09 PM, Alfred Perlstein = wrote: >>=20 >>> On 11/3/14, 8:48 AM, Warner Losh wrote: >>>> Does anybody recall why we have @ symlink in our module builds? >>>> I=92m constantly working around issues that this creates. Maybe it = is time >>>> to eliminate it? >>>>=20 >>>> So I=92ve posted the following review: = https://reviews.freebsd.org/D1100 >>>>=20 >>>> Warner >>>>=20 >>> Thank you!!!! I hate the '@' thing as well, I figured it was just = due to someone not wanting 'grep -r' to work. :) >>>=20 >>> Just to check, after your changes... will you still be able to do = something like: >>>=20 >>> cd $HOME >>> svn co https://.../base/head/sys/modules/foo_module >>> cd foo_module >>> make depend all install >> It should behave the same it does today. I=92m replacing a ln -s = ${SYSDIR} @ with an in-line replacement of @ with ${SYSDIR}. > Excellent! >=20 > I did a history check to see why these came to be and it looks like it = was just an oversight to use "@ links" instead of just SYSDIR as far as = I can tell... Back in the deep, dark past of FreeBSD=92s build system: >Revision 32813 - Mon Jan 26 20:36:38 1998 UTC (16 years, 9 months ago) = by bde=20 > >Generate symlinks to the "sys" and directories and put >them in the include path. This fixes recent breakage of the syscons >LKMs and general brokenness of the include paths (headers under >/usr/include were used in many cases). it was introduced. This is still in the lkm era. This pre-dates SYSDIR = being meaningful. That had to wait until: >Revision 59097 - Sat Apr 8 17:20:00 2000 UTC (14 years, 6 months ago) = by imp=20 >Add support for compiling kernel modules outside of the tree. If you >do not have the kernel you wish to compile against in either >/usr/src/sys or /sys, then you will need to set SYSDIR to point to the >sys directory of the source tree that contians the source. > >Also, minor tweaks to the load/unload targets from Bruce. > >I've had this through several make worlds, as well as using it on a >daily basis for the past couple of weeks to build modules needed for >testing at Timing Solutions. > >Reviewed and revised by: bde >Work sponsored by: Timing Solutions so I=92m only 14 years tardy in =93finishing=94 this change set :) Warner --Apple-Mail=_5681B2AB-E1D3-4F11-83CD-499C1B8C4074 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJUV+IaAAoJEGwc0Sh9sBEAIpcQAOBLKticcoxF5sYu4zYHmIAR Xb38iCplsBhAmx6hmiBZXXKumiqok4SgVgM+DCG+jh5nclJygXuN7JGCaDZU8yep YKSqG481GjrwMExOXEPK4irRqhLn/V79TkS5OP+pumdapRJWQsFwDT7j9BlDF2y+ iNWF8ra8oDHDLryo7b20Kx6AgM0Jt99qWyweYjCsVVE32XjM2WNWjH3JmCAK4+ix 6eymBnVTY2nx0NGH8scaDLuOM/RgLElMR7Fv6DeCFer4bHLurQ+6UZSiIROYFue/ AxASfRch8JIhESWIE/Gf6SG1EcBgMUJpVoWE7/lyEi3ACh0/+ESdbkOM43qHd7+Y 3mtZl5BA3AMffYyYo1lZY4VrE0xxCJ3EtmUFoLW7BaaFXw90r35vGaXviJqIpaRj ZeknG1PtH+ORR6bLTmYPpfsIgv36l6JTI781wp1LkxlMKhk3sOaxFnX8TTjVNwnK otBENUlXUhv5SYVSCwxn7oTZS7KuxviaeGc8gO80MjhD1o4yD89CBrjo9nOkooOZ 0QyESWQ5zsypNLcmu+yWSMACHWWOMBVgmj3JZyrf+T5mwELUo4YU4R6MMuEE/82f NXA4zU1L96q2anBj6nDY3KQCZFOXl7a7d0/Nx61tz4fO5nrf/jTeFE8bKr6dfvtb vAzjDk7s+7pJ3gD1Cf8A =07nM -----END PGP SIGNATURE----- --Apple-Mail=_5681B2AB-E1D3-4F11-83CD-499C1B8C4074-- From owner-freebsd-arch@FreeBSD.ORG Mon Nov 3 20:56:45 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0242AA31 for ; Mon, 3 Nov 2014 20:56:45 +0000 (UTC) Received: from hergotha.csail.mit.edu (wollman-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B28F599A for ; Mon, 3 Nov 2014 20:56:44 +0000 (UTC) Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.14.9/8.14.9) with ESMTP id sA3Kuc3B006301; Mon, 3 Nov 2014 15:56:39 -0500 (EST) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.14.9/8.14.4/Submit) id sA3KubAk006298; Mon, 3 Nov 2014 15:56:37 -0500 (EST) (envelope-from wollman) Date: Mon, 3 Nov 2014 15:56:37 -0500 (EST) From: Garrett Wollman Message-Id: <201411032056.sA3KubAk006298@hergotha.csail.mit.edu> To: imp@bsdimp.com Subject: Re: Why do we have @ in modules builds? X-Newsgroups: mit.lcs.mail.freebsd-arch In-Reply-To: <3285BC54-05D8-41DB-88FE-BAD681A3E45B@bsdimp.com> References: <3285BC54-05D8-41DB-88FE-BAD681A3E45B@bsdimp.com> Organization: none X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (hergotha.csail.mit.edu [127.0.0.1]); Mon, 03 Nov 2014 15:56:39 -0500 (EST) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=disabled version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on hergotha.csail.mit.edu Cc: freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Nov 2014 20:56:45 -0000 In article <3285BC54-05D8-41DB-88FE-BAD681A3E45B@bsdimp.com>, Warner Losh writes: >Does anybody recall why we have @ symlink in our module builds? Back in the mists of ancient time, it was a way to allow easy relocation of kernel builds -- that way, there was only one symlink to change rather than two (or perhaps even more). I'm reasonably convinced it's my fault, anyway. Forgive me: I was still young then. -GAWollman From owner-freebsd-arch@FreeBSD.ORG Mon Nov 3 22:15:04 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DB60641B for ; Mon, 3 Nov 2014 22:15:03 +0000 (UTC) Received: from mail109.syd.optusnet.com.au (mail109.syd.optusnet.com.au [211.29.132.80]) by mx1.freebsd.org (Postfix) with ESMTP id 9DD22258 for ; Mon, 3 Nov 2014 22:15:02 +0000 (UTC) Received: from c122-106-147-133.carlnfd1.nsw.optusnet.com.au (c122-106-147-133.carlnfd1.nsw.optusnet.com.au [122.106.147.133]) by mail109.syd.optusnet.com.au (Postfix) with ESMTPS id 3665CD684B7; Tue, 4 Nov 2014 09:15:00 +1100 (AEDT) Date: Tue, 4 Nov 2014 09:14:59 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Warner Losh Subject: Re: Why do we have @ in modules builds? In-Reply-To: <3285BC54-05D8-41DB-88FE-BAD681A3E45B@bsdimp.com> Message-ID: <20141104082203.J2282@besplex.bde.org> References: <3285BC54-05D8-41DB-88FE-BAD681A3E45B@bsdimp.com> MIME-Version: 1.0 X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=dMCfxopb c=1 sm=1 tr=0 a=7NqvjVvQucbO2RlWB8PEog==:117 a=PO7r1zJSAAAA:8 a=JzwRw_2MAAAA:8 a=nlC_4_pT8q9DhB4Ho9EA:9 a=cz2ZRIgtxKwA:10 a=wJWlkF7cXJYA:10 a=i0JV2VhHQkC0FAWM-98A:9 a=45ClL6m2LaAA:10 Content-Type: TEXT/PLAIN; charset=X-UNKNOWN; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Nov 2014 22:15:04 -0000 On Mon, 3 Nov 2014, Warner Losh wrote: > Does anybody recall why we have @ symlink in our module builds? > I=92m constantly working around issues that this creates. Maybe it is tim= e > to eliminate it? It is probably because I finished implementing '@' for modules better than for kernels. wollman introduced '@', but didn't change config(8) to generate it. My version of config does generate it, and I always use it for kernels, but I never committed this. My kernel object tree is under /usr/obj, and my source tree is hacked to point to it. E.g., /sys/i386/compile -> /usr/obj/usr/src/sys/compile (should be to .../sys/i386/compile). '@' then points back to /usr/src/sys. Modules have more varied paths than kernels, so @ is even more useful for them. -current (ref11-i386) also uses /usr/obj, but instead of '@' it uses: % S=3D/usr/src/sys % ... % .if !defined(S) % .if exists(./@/.) % S=3D=09./@ % .else % S=3D=09../../.. % .endif % .endif % .include "$S/conf/kern.pre.mk" %=20 % INCLUDES+=3D -I$S/contrib/libfdt The default S wouldn't work at all. Config doesn't support '@', but suppor= ts 'S' instead. (It seems to only do this when necessary -- it doesn't do it for my source-relative object trees on the same machine.) The generated Makefile then uses $S throughout as before. This hasn't chan= ged since at least wollman's change in ~1993. Using $S instead of its expansio= n keeps the makefile fairly readable. The main advantage of '@' is that it reduces the unreadability of make output and .depend files. When '@' is used, $S expands to just "./@". This keeps individual pathnames fairly short and makes it easier to see the basenames. $S might be much longer than the above, depending on the depth of the source tree and on realpath-like expansion of symlinks. make output remains fairly unreadable due to other spam in it. The spam that it used to have for -D options has been adequately replaced by spam from -W options. Without automatic generation of the paths, finding the path to the sources is fragile. To find the top level directory, kmod.mk still seems to just search .., ../.., /sys and /usr/src/sys. No trace of SRCDIR in the Aug 6 RELENG_11 version. I think I wrote most of that. It is better than nothing. Modularity is also very broken for programs. Most programs now depend on src.opts.mk, which doesn't exist for programs outside of the source tree. src.opts.mk sort of goes in the opposite and worse direction than SRCDIR. With a single source tree, you can hard-code SRCDIR in a single file near src.opts.mk and not have to guess it. Actually, the mk directory is still hard to find. Bruce From owner-freebsd-arch@FreeBSD.ORG Wed Nov 5 09:02:21 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3828C19A for ; Wed, 5 Nov 2014 09:02:21 +0000 (UTC) Received: from mail-wi0-x22c.google.com (mail-wi0-x22c.google.com [IPv6:2a00:1450:400c:c05::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B7A9B8B5 for ; Wed, 5 Nov 2014 09:02:20 +0000 (UTC) Received: by mail-wi0-f172.google.com with SMTP id bs8so11714047wib.11 for ; Wed, 05 Nov 2014 01:02:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=p/5ceOF7Rpm8/x4XQOCwn2j6Ttsr3V6lwsfpS9ItldA=; b=d/RiW/M6fX+We3ICxIUU2cyOzIhfxNtSH2B/iH6hQ71nAHP51GkycawgDgsbZmhTb4 6ZHTuYYo4EOsLlMr6jpcdrwRaQ6IeJIt1FIRJgIs2q24iRYDI0yHQ1szywGZCubHq1kw KYGjVsi0nod2BH7dtadUMnZTDKcb22RAyLPg1WAkJIVkRtZjkFJCORSweuqIY81J2vrk H7gCAgV+6QTkfQL1dbs3uGuldyNbrqyltsB4yokp/Gg+N5r26ljYgv9xekuGeAoAOgn4 LJUNo4RwCLAOJ6+uoUY1lwIgoZ5me4NCThDwAU/8ln/Rz5M5a66/4I+i7eyTdsLrfxCO 2ing== X-Received: by 10.180.94.70 with SMTP id da6mr3930083wib.69.1415178138951; Wed, 05 Nov 2014 01:02:18 -0800 (PST) Received: from ivaldir.etoilebsd.net ([2001:41d0:8:db4c::1]) by mx.google.com with ESMTPSA id cv7sm3334849wjc.3.2014.11.05.01.02.17 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Nov 2014 01:02:17 -0800 (PST) Sender: Baptiste Daroussin Date: Wed, 5 Nov 2014 10:02:15 +0100 From: Baptiste Daroussin To: Warner Losh Subject: Re: PIE/PIC support on base Message-ID: <20141105090215.GF10388@ivaldir.etoilebsd.net> References: <315B4DC5-0E04-4F6B-BBB0-477D049025BF@bsdimp.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="udcq9yAoWb9A4FsZ" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Cc: PaX Team , FreeBSD Arch , Shawn Webb X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 09:02:21 -0000 --udcq9yAoWb9A4FsZ Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Oct 17, 2014 at 08:05:57AM -0600, Warner Losh wrote: > [[cc trimmed ]] >=20 > On Oct 17, 2014, at 7:46 AM, Shawn Webb wrote: >=20 > > On Fri, Oct 17, 2014 at 9:41 AM, Warner Losh wrote: > >=20 > > On Oct 17, 2014, at 2:05 AM, Shawn Webb wrote: > >=20 > > > On Fri, Oct 17, 2014 at 3:53 AM, Jeremie Le Hen wro= te: > > > > > >> On Fri, Oct 17, 2014 at 12:15 AM, Shawn Webb wro= te: > > >>> > > >>> > > >>> On Thu, Oct 16, 2014 at 5:59 PM, Jeremie Le Hen w= rote: > > >>>> > > >>>> On Thu, Oct 16, 2014 at 8:21 PM, David Carlier > > >>>> wrote: > > >>>>> > > >>>>> I chose the "atomic" approach, at the moment very few binaries are > > >>>>> concerned at the moment. So I applied INCLUDE_PIC_ARCHIVE in the > > >> needed > > >>>>> libraries plus created WITH_PIE which add fPIE/fpie -pie flags on= ly if > > >>>>> you > > >>>>> include (which include ...) otherw= ise > > >>>>> other > > >>>>> binaries include as usual hence does not apply. Look > > >>>>> reasonable approach ? > > >>>> > > >>>> I think I understand what you mean. But I think PIE is commonplace > > >>>> nowadays and I don't understand what you win by not enabling it for > > >>>> the whole system. Is it a performance concern? Is it to preserve > > >>>> conservative minds from to much change? :) > > >>> > > >>> > > >>> Looping in Kostik, Bryan Drewery, the PaX team, Hunger, and Sean Br= uno. > > >>> > > >>> On i386, there is a performance cost due to not having an extra reg= ister > > >>> available for the relocation work that has to happen. PIE doesn't c= arry > > >> much > > >>> of a performance penalty on amd64, though it still does carry some = on > > >> first > > >>> resolution of functions (due to the extra relocation step the RTLD = has to > > >>> worry about). On amd64, after symbol resolution has taken place, th= ere > > >> is no > > >>> further performance penalty due to amd64 having an extra register t= o use > > >> for > > >>> PIE/PIC. I'm unsure what, if any, performance penalty PIE carries o= n ARM, > > >>> AArch64, and sparc64. > > >>> > > >>> Certain folk would prefer to see PIE enabled only in certain > > >> applications. > > >>> /bin/ls can't really make much use of PIE. But sshd can. I personal= ly > > >> would > > >>> like to see all of base's applications compiled as PIEs, but that's= a > > >> long > > >>> ways off. It took OpenBSD several years to accomplish that. Having > > >> certain > > >>> high-visibility applications (like sshd, inetd, etc) is a great sta= rt. > > >>> Providing a framework for application developers to opt their appli= cation > > >>> into PIE is another great start. > > >>> > > >>> Those are my two cents. > > >> > > >> OK. As long as i386 is still an important architecture, it can make > > >> sense to enable this on a per-binary basis if we don't want to have a > > >> discrepancy between archs. Also I buy your argument on /bin/ls but I > > >> was challenging to enable for the whole system because I wonder if > > >> there aren't some unexpected attack surfaces, besides the obvious on= es > > >> (servers). > > >> > > >> Do you know what took so much time to OpenBSD? > > > > > > > > > In a private conversation with Theo, I realized that my recollection = of the > > > time it took OpenBSD to compile all of base as PIEs was wrong. Quotin= g him: > > > > > > "It took 5 people approximately 3 months to debug it, activate it, and > > > start shipping it the next release. That was on amd64, for all > > > dynamically linked binaries, except one (a gcc bug took some time to > > > find). The next architectures followed about 1 or 2 per 6-month > > > release." > > > > > > Given that only one person has worked on this in the past (me) and no= w the > > > task has been delegated to another (David Carlier), I think we're doi= ng > > > okay on our end. There's a lot of moving parts, and neither of us ful= ly > > > understand all of them completely. We're working on it in HardenedBSD= , in > > > the hardened/current/pie branch. > > > > > > I'm thinking we might try for a WITH_PIE knob (and *not* use USE_PIE)= and > > > have certain high-profile applications opt-in to PIE until we work ou= t all > > > the details for everything en masse. Baptiste did bring up a good poi= nt > > > with INTERNALLIB and I'm unsure of how we should handle that. > >=20 > > WITH_PIE or WITHOUT_PIE controls, on a global basis, via the MK_PIE > > variable, whether or not the user wants to turn on this feature for tho= se > > program that can do PIE. Designating which programs do, or don=E2=80=99= t, > > use PIE simply must be done with a different variable. I posted a bit o= f a > > rant about the current state of things that suggested a couple of > > alternatives as well as giving some history as to why some options > > aren=E2=80=99t to be used and the history behind some of my reactions. = :) > >=20 > > For this reason, I think WITH_PIE, as I understand your proposal, > > likely isn=E2=80=99t a good fit with other WITH_xxx variables used in t= he src > > tree today. > >=20 > > Gotcha. To be honest, I found your email a tad bit confusing. Are you s= uggesting we create an ENABLE_feature framework? Or are you suggesting we h= ave a USE_PIE flag? Or are you suggesting something different entirely (and= if you are, what?)? >=20 > I=E2=80=99m saying we don=E2=80=99t have a good framework at the moment t= o do this. We > have several bad ones that all have their pitfalls. This is one reason I = had > the fast reaction to NO_PIE, then a minute later said =E2=80=9Cgo ahead a= nd use > it and I=E2=80=99ll fix it.=E2=80=9D I=E2=80=99m still cool with that pos= ition, btw. >=20 > As for a name, that can be debated a lot, but I=E2=80=99d like to see so= mething > new, easy to use and unambiguous. If you are looking for a suggestion > for that name, let=E2=80=99s go with WANTS_PIE. Only Makefiles can set it. >=20 > WANTS_PIE undefined means do the default behavior as defined by the > current MK_PIE setting and perhaps system policy. =E2=80=9CGo with this f= low." >=20 > WANTS_PIE=3Dyes means that if MK_PIE is =E2=80=9Cyes=E2=80=9D, then do PI= E things for > this thing we=E2=80=99re building. If MK_PIE is =E2=80=9Cno=E2=80=9D, tho= ugh PIE is disabled for > everything. >=20 > WANTS_PIE=3Dno means that if MK_PIE is =E2=80=9Cyes=E2=80=9D, then disabl= e doing PIE > things for this component. If MK_PIE is no, it is also disabled. >=20 > This could also be extended to NEEDS_foo, which says =E2=80=9CI need foo = to > build, and if MK_foo is set to no, don=E2=80=99t build me.=E2=80=9D I don= =E2=80=99t think anything > that you are doing falls into this category though. >=20 > WANTS/NEEDS also avoids the historical use of USE in the ports tree > possibly creating confusion.=20 >=20 > If you go with WANTS_PIE, then you wouldn=E2=80=99t need bad.*.pie.mk. >=20 > Comments? >=20 > Warner On amd64 WANTS_PIE will be useless as we can easily activate PIE on every p= laces For i386 we would propably prefer cherry picking the what we want to see bu= ilt with PIE. Don't know for other arches. So here is what I do propose: if MK_PIE=3Dno: no PIE at all if MK_PIE=3Dyes: - on amd64/(platforms without performance penalty): build everything with P= IE from libs to prog - on i386/(platforms with performance penalty): build with PIE if WANTS_PIE is defined. So the difference with the previous approach are: - No way to opt out PIE for a single binary either totally disable or enabl= e (I have encountered no binary so far in the base system which fails with PIE enabled - again only tested on amd64) - Activate PIE for both binaries and libraries (no reason not to include libraries) If we are ok with this I will publish a patch with does it in the next coup= le of week. We can start by only supporting amd64 as fully PIE first then slowly add ot= her platforms. What do you think? regards, Bapt --udcq9yAoWb9A4FsZ Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlRZ55cACgkQ8kTtMUmk6EyougCgoIiULNL3x49jBDJKcCOK5M4z QQsAn00ZLWfRKv7A+87qLiM92jOvfbIn =m1vr -----END PGP SIGNATURE----- --udcq9yAoWb9A4FsZ-- From owner-freebsd-arch@FreeBSD.ORG Wed Nov 5 09:26:24 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 16B959A9; Wed, 5 Nov 2014 09:26:24 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AD592AE6; Wed, 5 Nov 2014 09:26:23 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id sA59QErb018051 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 5 Nov 2014 11:26:14 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua sA59QErb018051 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id sA59QE7d018050; Wed, 5 Nov 2014 11:26:14 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 5 Nov 2014 11:26:14 +0200 From: Konstantin Belousov To: Baptiste Daroussin Subject: Re: PIE/PIC support on base Message-ID: <20141105092614.GB53947@kib.kiev.ua> References: <315B4DC5-0E04-4F6B-BBB0-477D049025BF@bsdimp.com> <20141105090215.GF10388@ivaldir.etoilebsd.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20141105090215.GF10388@ivaldir.etoilebsd.net> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: PaX Team , Shawn Webb , FreeBSD Arch X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 09:26:24 -0000 On Wed, Nov 05, 2014 at 10:02:15AM +0100, Baptiste Daroussin wrote: > On Fri, Oct 17, 2014 at 08:05:57AM -0600, Warner Losh wrote: > > [[cc trimmed ]] > > > > On Oct 17, 2014, at 7:46 AM, Shawn Webb wrote: > > > > > On Fri, Oct 17, 2014 at 9:41 AM, Warner Losh wrote: > > > > > > On Oct 17, 2014, at 2:05 AM, Shawn Webb wrote: > > > > > > > On Fri, Oct 17, 2014 at 3:53 AM, Jeremie Le Hen wrote: > > > > > > > >> On Fri, Oct 17, 2014 at 12:15 AM, Shawn Webb wrote: > > > >>> > > > >>> > > > >>> On Thu, Oct 16, 2014 at 5:59 PM, Jeremie Le Hen wrote: > > > >>>> > > > >>>> On Thu, Oct 16, 2014 at 8:21 PM, David Carlier > > > >>>> wrote: > > > >>>>> > > > >>>>> I chose the "atomic" approach, at the moment very few binaries are > > > >>>>> concerned at the moment. So I applied INCLUDE_PIC_ARCHIVE in the > > > >> needed > > > >>>>> libraries plus created WITH_PIE which add fPIE/fpie -pie flags only if > > > >>>>> you > > > >>>>> include (which include ...) otherwise > > > >>>>> other > > > >>>>> binaries include as usual hence does not apply. Look > > > >>>>> reasonable approach ? > > > >>>> > > > >>>> I think I understand what you mean. But I think PIE is commonplace > > > >>>> nowadays and I don't understand what you win by not enabling it for > > > >>>> the whole system. Is it a performance concern? Is it to preserve > > > >>>> conservative minds from to much change? :) > > > >>> > > > >>> > > > >>> Looping in Kostik, Bryan Drewery, the PaX team, Hunger, and Sean Bruno. > > > >>> > > > >>> On i386, there is a performance cost due to not having an extra register > > > >>> available for the relocation work that has to happen. PIE doesn't carry > > > >> much > > > >>> of a performance penalty on amd64, though it still does carry some on > > > >> first > > > >>> resolution of functions (due to the extra relocation step the RTLD has to > > > >>> worry about). On amd64, after symbol resolution has taken place, there > > > >> is no > > > >>> further performance penalty due to amd64 having an extra register to use > > > >> for > > > >>> PIE/PIC. I'm unsure what, if any, performance penalty PIE carries on ARM, > > > >>> AArch64, and sparc64. > > > >>> > > > >>> Certain folk would prefer to see PIE enabled only in certain > > > >> applications. > > > >>> /bin/ls can't really make much use of PIE. But sshd can. I personally > > > >> would > > > >>> like to see all of base's applications compiled as PIEs, but that's a > > > >> long > > > >>> ways off. It took OpenBSD several years to accomplish that. Having > > > >> certain > > > >>> high-visibility applications (like sshd, inetd, etc) is a great start. > > > >>> Providing a framework for application developers to opt their application > > > >>> into PIE is another great start. > > > >>> > > > >>> Those are my two cents. > > > >> > > > >> OK. As long as i386 is still an important architecture, it can make > > > >> sense to enable this on a per-binary basis if we don't want to have a > > > >> discrepancy between archs. Also I buy your argument on /bin/ls but I > > > >> was challenging to enable for the whole system because I wonder if > > > >> there aren't some unexpected attack surfaces, besides the obvious ones > > > >> (servers). > > > >> > > > >> Do you know what took so much time to OpenBSD? > > > > > > > > > > > > In a private conversation with Theo, I realized that my recollection of the > > > > time it took OpenBSD to compile all of base as PIEs was wrong. Quoting him: > > > > > > > > "It took 5 people approximately 3 months to debug it, activate it, and > > > > start shipping it the next release. That was on amd64, for all > > > > dynamically linked binaries, except one (a gcc bug took some time to > > > > find). The next architectures followed about 1 or 2 per 6-month > > > > release." > > > > > > > > Given that only one person has worked on this in the past (me) and now the > > > > task has been delegated to another (David Carlier), I think we're doing > > > > okay on our end. There's a lot of moving parts, and neither of us fully > > > > understand all of them completely. We're working on it in HardenedBSD, in > > > > the hardened/current/pie branch. > > > > > > > > I'm thinking we might try for a WITH_PIE knob (and *not* use USE_PIE) and > > > > have certain high-profile applications opt-in to PIE until we work out all > > > > the details for everything en masse. Baptiste did bring up a good point > > > > with INTERNALLIB and I'm unsure of how we should handle that. > > > > > > WITH_PIE or WITHOUT_PIE controls, on a global basis, via the MK_PIE > > > variable, whether or not the user wants to turn on this feature for those > > > program that can do PIE. Designating which programs do, or don???t, > > > use PIE simply must be done with a different variable. I posted a bit of a > > > rant about the current state of things that suggested a couple of > > > alternatives as well as giving some history as to why some options > > > aren???t to be used and the history behind some of my reactions. :) > > > > > > For this reason, I think WITH_PIE, as I understand your proposal, > > > likely isn???t a good fit with other WITH_xxx variables used in the src > > > tree today. > > > > > > Gotcha. To be honest, I found your email a tad bit confusing. Are you suggesting we create an ENABLE_feature framework? Or are you suggesting we have a USE_PIE flag? Or are you suggesting something different entirely (and if you are, what?)? > > > > I???m saying we don???t have a good framework at the moment to do this. We > > have several bad ones that all have their pitfalls. This is one reason I had > > the fast reaction to NO_PIE, then a minute later said ???go ahead and use > > it and I???ll fix it.??? I???m still cool with that position, btw. > > > > As for a name, that can be debated a lot, but I???d like to see something > > new, easy to use and unambiguous. If you are looking for a suggestion > > for that name, let???s go with WANTS_PIE. Only Makefiles can set it. > > > > WANTS_PIE undefined means do the default behavior as defined by the > > current MK_PIE setting and perhaps system policy. ???Go with this flow." > > > > WANTS_PIE=yes means that if MK_PIE is ???yes???, then do PIE things for > > this thing we???re building. If MK_PIE is ???no???, though PIE is disabled for > > everything. > > > > WANTS_PIE=no means that if MK_PIE is ???yes???, then disable doing PIE > > things for this component. If MK_PIE is no, it is also disabled. > > > > This could also be extended to NEEDS_foo, which says ???I need foo to > > build, and if MK_foo is set to no, don???t build me.??? I don???t think anything > > that you are doing falls into this category though. > > > > WANTS/NEEDS also avoids the historical use of USE in the ports tree > > possibly creating confusion. > > > > If you go with WANTS_PIE, then you wouldn???t need bad.*.pie.mk. > > > > Comments? > > > > Warner > > On amd64 WANTS_PIE will be useless as we can easily activate PIE on every places > For i386 we would propably prefer cherry picking the what we want to see built > with PIE. Don't know for other arches. > > So here is what I do propose: > if MK_PIE=no: no PIE at all > if MK_PIE=yes: > - on amd64/(platforms without performance penalty): build everything with PIE > from libs to prog See below. > - on i386/(platforms with performance penalty): build with PIE if WANTS_PIE > is defined. > > So the difference with the previous approach are: > - No way to opt out PIE for a single binary either totally disable or enable (I > have encountered no binary so far in the base system which fails with PIE > enabled - again only tested on amd64) > - Activate PIE for both binaries and libraries (no reason not to include > libraries) What does it mean 'PIE for library' ? There is simply no such thing. Also, I strongly oppose compiling everything with PIC, even on amd64. I described somewhere else that using PIC code changes symbol lookup rules for binaries. So despite not having performance impact, the thing does impact runtime behaviour in subtle ways. The most affected programs are those which support dynamic modules. Also, what is the state of static binaries + PIE ? Do our binutils support this at all ? The csu is definitely not ready for 'everything PIE'. > > If we are ok with this I will publish a patch with does it in the next couple of > week. > > We can start by only supporting amd64 as fully PIE first then slowly add other > platforms. > > What do you think? > > regards, > Bapt From owner-freebsd-arch@FreeBSD.ORG Wed Nov 5 09:32:25 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 10EE4B3B; Wed, 5 Nov 2014 09:32:25 +0000 (UTC) Received: from mail-oi0-x22a.google.com (mail-oi0-x22a.google.com [IPv6:2607:f8b0:4003:c06::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BE2CFBC9; Wed, 5 Nov 2014 09:32:24 +0000 (UTC) Received: by mail-oi0-f42.google.com with SMTP id a3so11155394oib.29 for ; Wed, 05 Nov 2014 01:32:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ZrH9JRaT4wNol8N4Ydw8n6oCVPcuJShi3f7Ep71l7Fc=; b=xrBu0Fd/YtR8B0ggvQxxtLvcRFQa8xIw7CUYmFyQmmA8RlrYeMX3Obdkei0oyYtZx/ VfhbgiKstG2a0kSqUJWv8r2mfmW0xkhShuelhO1QrJZA02tpNPl9pWLZ5OKl0dXsIApT 19J8rXiO28H3YnghrKGoq+9EvYPQiwykLFgwE0wdng2X9fBgIxjM4kC6adW8Wmeu9w7N fptnKulMC0dPkM3INK4F3DYXuMZuGmvU0EVyNwr1atgx0EbnYg+2XI/v7qTJvBlQ6rWn g600R04u109LG6g0mAjgD08CKymycwQULL8iX/6MGp7HAF0BGMMHILTOF7EWNc1bs3zC 1uFA== MIME-Version: 1.0 X-Received: by 10.202.61.8 with SMTP id k8mr24729421oia.3.1415179944019; Wed, 05 Nov 2014 01:32:24 -0800 (PST) Received: by 10.202.208.150 with HTTP; Wed, 5 Nov 2014 01:32:23 -0800 (PST) In-Reply-To: <20141105090215.GF10388@ivaldir.etoilebsd.net> References: <315B4DC5-0E04-4F6B-BBB0-477D049025BF@bsdimp.com> <20141105090215.GF10388@ivaldir.etoilebsd.net> Date: Wed, 5 Nov 2014 10:32:23 +0100 Message-ID: Subject: Re: PIE/PIC support on base From: Oliver Pinter To: Baptiste Daroussin Content-Type: text/plain; charset=ISO-8859-1 Cc: PaX Team , Shawn Webb , FreeBSD Arch X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 09:32:25 -0000 On 11/5/14, Baptiste Daroussin wrote: > On Fri, Oct 17, 2014 at 08:05:57AM -0600, Warner Losh wrote: >> [[cc trimmed ]] >> >> On Oct 17, 2014, at 7:46 AM, Shawn Webb wrote: >> >> > On Fri, Oct 17, 2014 at 9:41 AM, Warner Losh wrote: >> > >> > On Oct 17, 2014, at 2:05 AM, Shawn Webb wrote: >> > >> > > On Fri, Oct 17, 2014 at 3:53 AM, Jeremie Le Hen >> > > wrote: >> > > >> > >> On Fri, Oct 17, 2014 at 12:15 AM, Shawn Webb >> > >> wrote: >> > >>> >> > >>> >> > >>> On Thu, Oct 16, 2014 at 5:59 PM, Jeremie Le Hen >> > >>> wrote: >> > >>>> >> > >>>> On Thu, Oct 16, 2014 at 8:21 PM, David Carlier >> > >>>> wrote: >> > >>>>> >> > >>>>> I chose the "atomic" approach, at the moment very few binaries >> > >>>>> are >> > >>>>> concerned at the moment. So I applied INCLUDE_PIC_ARCHIVE in the >> > >> needed >> > >>>>> libraries plus created WITH_PIE which add fPIE/fpie -pie flags >> > >>>>> only if >> > >>>>> you >> > >>>>> include (which include ...) >> > >>>>> otherwise >> > >>>>> other >> > >>>>> binaries include as usual hence does not apply. >> > >>>>> Look >> > >>>>> reasonable approach ? >> > >>>> >> > >>>> I think I understand what you mean. But I think PIE is >> > >>>> commonplace >> > >>>> nowadays and I don't understand what you win by not enabling it >> > >>>> for >> > >>>> the whole system. Is it a performance concern? Is it to preserve >> > >>>> conservative minds from to much change? :) >> > >>> >> > >>> >> > >>> Looping in Kostik, Bryan Drewery, the PaX team, Hunger, and Sean >> > >>> Bruno. >> > >>> >> > >>> On i386, there is a performance cost due to not having an extra >> > >>> register >> > >>> available for the relocation work that has to happen. PIE doesn't >> > >>> carry >> > >> much >> > >>> of a performance penalty on amd64, though it still does carry some >> > >>> on >> > >> first >> > >>> resolution of functions (due to the extra relocation step the RTLD >> > >>> has to >> > >>> worry about). On amd64, after symbol resolution has taken place, >> > >>> there >> > >> is no >> > >>> further performance penalty due to amd64 having an extra register to >> > >>> use >> > >> for >> > >>> PIE/PIC. I'm unsure what, if any, performance penalty PIE carries on >> > >>> ARM, >> > >>> AArch64, and sparc64. >> > >>> >> > >>> Certain folk would prefer to see PIE enabled only in certain >> > >> applications. >> > >>> /bin/ls can't really make much use of PIE. But sshd can. I >> > >>> personally >> > >> would >> > >>> like to see all of base's applications compiled as PIEs, but that's >> > >>> a >> > >> long >> > >>> ways off. It took OpenBSD several years to accomplish that. Having >> > >> certain >> > >>> high-visibility applications (like sshd, inetd, etc) is a great >> > >>> start. >> > >>> Providing a framework for application developers to opt their >> > >>> application >> > >>> into PIE is another great start. >> > >>> >> > >>> Those are my two cents. >> > >> >> > >> OK. As long as i386 is still an important architecture, it can make >> > >> sense to enable this on a per-binary basis if we don't want to have >> > >> a >> > >> discrepancy between archs. Also I buy your argument on /bin/ls but I >> > >> was challenging to enable for the whole system because I wonder if >> > >> there aren't some unexpected attack surfaces, besides the obvious >> > >> ones >> > >> (servers). >> > >> >> > >> Do you know what took so much time to OpenBSD? >> > > >> > > >> > > In a private conversation with Theo, I realized that my recollection >> > > of the >> > > time it took OpenBSD to compile all of base as PIEs was wrong. Quoting >> > > him: >> > > >> > > "It took 5 people approximately 3 months to debug it, activate it, >> > > and >> > > start shipping it the next release. That was on amd64, for all >> > > dynamically linked binaries, except one (a gcc bug took some time to >> > > find). The next architectures followed about 1 or 2 per 6-month >> > > release." >> > > >> > > Given that only one person has worked on this in the past (me) and now >> > > the >> > > task has been delegated to another (David Carlier), I think we're >> > > doing >> > > okay on our end. There's a lot of moving parts, and neither of us >> > > fully >> > > understand all of them completely. We're working on it in HardenedBSD, >> > > in >> > > the hardened/current/pie branch. >> > > >> > > I'm thinking we might try for a WITH_PIE knob (and *not* use USE_PIE) >> > > and >> > > have certain high-profile applications opt-in to PIE until we work out >> > > all >> > > the details for everything en masse. Baptiste did bring up a good >> > > point >> > > with INTERNALLIB and I'm unsure of how we should handle that. >> > >> > WITH_PIE or WITHOUT_PIE controls, on a global basis, via the MK_PIE >> > variable, whether or not the user wants to turn on this feature for >> > those >> > program that can do PIE. Designating which programs do, or don't, >> > use PIE simply must be done with a different variable. I posted a bit of >> > a >> > rant about the current state of things that suggested a couple of >> > alternatives as well as giving some history as to why some options >> > aren't to be used and the history behind some of my reactions. :) >> > >> > For this reason, I think WITH_PIE, as I understand your proposal, >> > likely isn't a good fit with other WITH_xxx variables used in the src >> > tree today. >> > >> > Gotcha. To be honest, I found your email a tad bit confusing. Are you >> > suggesting we create an ENABLE_feature framework? Or are you suggesting >> > we have a USE_PIE flag? Or are you suggesting something different >> > entirely (and if you are, what?)? >> >> I'm saying we don't have a good framework at the moment to do this. We >> have several bad ones that all have their pitfalls. This is one reason I >> had >> the fast reaction to NO_PIE, then a minute later said "go ahead and use >> it and I'll fix it." I'm still cool with that position, btw. >> >> As for a name, that can be debated a lot, but I'd like to see something >> new, easy to use and unambiguous. If you are looking for a suggestion >> for that name, let's go with WANTS_PIE. Only Makefiles can set it. >> >> WANTS_PIE undefined means do the default behavior as defined by the >> current MK_PIE setting and perhaps system policy. "Go with this flow." >> >> WANTS_PIE=yes means that if MK_PIE is "yes", then do PIE things for >> this thing we're building. If MK_PIE is "no", though PIE is disabled for >> everything. >> >> WANTS_PIE=no means that if MK_PIE is "yes", then disable doing PIE >> things for this component. If MK_PIE is no, it is also disabled. >> >> This could also be extended to NEEDS_foo, which says "I need foo to >> build, and if MK_foo is set to no, don't build me." I don't think >> anything >> that you are doing falls into this category though. >> >> WANTS/NEEDS also avoids the historical use of USE in the ports tree >> possibly creating confusion. >> >> If you go with WANTS_PIE, then you wouldn't need bad.*.pie.mk. >> >> Comments? >> >> Warner > > On amd64 WANTS_PIE will be useless as we can easily activate PIE on every > places > For i386 we would propably prefer cherry picking the what we want to see > built > with PIE. Don't know for other arches. > > So here is what I do propose: > if MK_PIE=no: no PIE at all > if MK_PIE=yes: > - on amd64/(platforms without performance penalty): build everything with > PIE > from libs to prog > - on i386/(platforms with performance penalty): build with PIE if WANTS_PIE > is defined. > > So the difference with the previous approach are: > - No way to opt out PIE for a single binary either totally disable or enable > (I > have encountered no binary so far in the base system which fails with PIE > enabled - again only tested on amd64) > - Activate PIE for both binaries and libraries (no reason not to include > libraries) > > If we are ok with this I will publish a patch with does it in the next > couple of > week. > > We can start by only supporting amd64 as fully PIE first then slowly add > other > platforms. > > What do you think? I think for us (HardenedBSD) this plan is good. If for FreeBSD not, we will integrate the test patch in HardenedBSD. > > regards, > Bapt > From owner-freebsd-arch@FreeBSD.ORG Wed Nov 5 11:38:44 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 65319BBD for ; Wed, 5 Nov 2014 11:38:44 +0000 (UTC) Received: from mail-wi0-x234.google.com (mail-wi0-x234.google.com [IPv6:2a00:1450:400c:c05::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F1627ADB for ; Wed, 5 Nov 2014 11:38:43 +0000 (UTC) Received: by mail-wi0-f180.google.com with SMTP id hi2so1739278wib.1 for ; Wed, 05 Nov 2014 03:38:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:subject:message-id:mime-version:content-type :content-disposition:user-agent; bh=PF3SbsJYYKaljvGybQaWcAzcgYpunjMYFNh1/Caqsew=; b=Nl+BJiqoeVID1VnS8S93dUKvSEilWfWpwcuWgm3zO+rbDXIYKfy7DP5f4shldGtioi HJBELcxNIS+WrlSU3GJr/fVy7kbEz3xNI0xBx6P6o++GFHvdU1IxWoHdgvwsU2rQbLoM 4KjZ5ciaHQ6C9LzrQ5v/mMeQ5ZHJjHAMBnv3QfGmvD2aZjX3ObXLYCn1frRFLD+DQ3O8 AqT5fQqwz9kpWZmyokl3W6U1UGIK217kcWU0DdsWGBw8Ds5YyDNKuVvizhauTcdh/C5C 83fp8dz8Fjg/Y5kfSxlVcFZTi+CwQmMTPopMA8Im7ZuvaOLcJ4y63iBqcNjQ27KPMd7k v3FA== X-Received: by 10.194.76.101 with SMTP id j5mr52444594wjw.57.1415187522304; Wed, 05 Nov 2014 03:38:42 -0800 (PST) Received: from ivaldir.etoilebsd.net ([2001:41d0:8:db4c::1]) by mx.google.com with ESMTPSA id l4sm3790221wjx.14.2014.11.05.03.38.41 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Nov 2014 03:38:41 -0800 (PST) Sender: Baptiste Daroussin Date: Wed, 5 Nov 2014 12:38:39 +0100 From: Baptiste Daroussin To: arch@FreeBSD.org Subject: Overlinking in base Message-ID: <20141105113839.GG10388@ivaldir.etoilebsd.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="OFj+1YLvsEfSXdCH" Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 11:38:44 -0000 --OFj+1YLvsEfSXdCH Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi all, We do overlink a lot in the base system. I wrote a script to check overlinking on the final binaries and running it on /usr/bin files I got the following results: https://wiki.freebsd.org/Overlinking Most of the overlink are due to LDADD having all the recursive libraries which for dynamic linking we do not need. We might need them for static linking, but I think we should fix those. I think anyway a binary Makefile should only declare its direct dependencies and not the dependencies of its dependencies. I was already proposing in the past to import pkgconf in base to solve this as the library will declare its dependencies in case of dynamic linking and/or in case of static linking (which can be different) This can also be done in pure make(1) regards, Bapt --OFj+1YLvsEfSXdCH Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlRaDD8ACgkQ8kTtMUmk6EysJQCfRQIgRa5leygkPsUqDWiR6uLt W/0AnjXevvLQikLsZsxebRIpmxdtycki =lZWp -----END PGP SIGNATURE----- --OFj+1YLvsEfSXdCH-- From owner-freebsd-arch@FreeBSD.ORG Wed Nov 5 11:49:01 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1261BED8 for ; Wed, 5 Nov 2014 11:49:01 +0000 (UTC) Received: from mail-wg0-x22e.google.com (mail-wg0-x22e.google.com [IPv6:2a00:1450:400c:c00::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 91AD0BD4 for ; Wed, 5 Nov 2014 11:49:00 +0000 (UTC) Received: by mail-wg0-f46.google.com with SMTP id x13so694600wgg.33 for ; Wed, 05 Nov 2014 03:48:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=YXj7gqNRDV7AdryLVk3F/3jHjfJ5NgO4k5ERoYsTtW4=; b=vnVKKeW/gXm4cmIa8oTcZSXt/JSfnTUBWOvWDflvKFYoXCizyjKXOgCz1Hq/zMHEzU OQENAEcGUiobd62LMNBfAfwlDvFOeFjKKV4ujUg5ap1V26N1hWtDDK0PwPsKlzB7l0dM Jzpgw+vO4JqWUuc+QCqODtHBt/NE3TX58LjGR8rdLtsuxDLcIPFNNBbXwQ3dEyTQ6khQ CmLHiFYDjjrwyr9w0hXmrdyXHBm1U9gv9k+TuoYLLlrIVMAOY6QJQmkb/RnxbMXkX0MY TRRsor+VZnd4KLud+Mv4ZZ19rmDCtFeinSkYoyUgEypVhTKZAQDe7W3zCk3IOcETfzfU zTKg== X-Received: by 10.180.107.230 with SMTP id hf6mr25556493wib.31.1415188138769; Wed, 05 Nov 2014 03:48:58 -0800 (PST) Received: from ivaldir.etoilebsd.net ([2001:41d0:8:db4c::1]) by mx.google.com with ESMTPSA id dw9sm4326879wib.0.2014.11.05.03.48.57 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Nov 2014 03:48:57 -0800 (PST) Sender: Baptiste Daroussin Date: Wed, 5 Nov 2014 12:48:55 +0100 From: Baptiste Daroussin To: Konstantin Belousov Subject: Re: PIE/PIC support on base Message-ID: <20141105114855.GH10388@ivaldir.etoilebsd.net> References: <315B4DC5-0E04-4F6B-BBB0-477D049025BF@bsdimp.com> <20141105090215.GF10388@ivaldir.etoilebsd.net> <20141105092614.GB53947@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="wj9ZLJVQDRFjGSdK" Content-Disposition: inline In-Reply-To: <20141105092614.GB53947@kib.kiev.ua> User-Agent: Mutt/1.5.23 (2014-03-12) Cc: PaX Team , FreeBSD Arch , Shawn Webb X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 11:49:01 -0000 --wj9ZLJVQDRFjGSdK Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Nov 05, 2014 at 11:26:14AM +0200, Konstantin Belousov wrote: > On Wed, Nov 05, 2014 at 10:02:15AM +0100, Baptiste Daroussin wrote: > > On Fri, Oct 17, 2014 at 08:05:57AM -0600, Warner Losh wrote: > > > [[cc trimmed ]] > > >=20 > > > On Oct 17, 2014, at 7:46 AM, Shawn Webb wrote: > > >=20 > > > > On Fri, Oct 17, 2014 at 9:41 AM, Warner Losh wrote: > > > >=20 > > > > On Oct 17, 2014, at 2:05 AM, Shawn Webb wrote: > > > >=20 > > > > > On Fri, Oct 17, 2014 at 3:53 AM, Jeremie Le Hen = wrote: > > > > > > > > > >> On Fri, Oct 17, 2014 at 12:15 AM, Shawn Webb = wrote: > > > > >>> > > > > >>> > > > > >>> On Thu, Oct 16, 2014 at 5:59 PM, Jeremie Le Hen wrote: > > > > >>>> > > > > >>>> On Thu, Oct 16, 2014 at 8:21 PM, David Carlier > > > > >>>> wrote: > > > > >>>>> > > > > >>>>> I chose the "atomic" approach, at the moment very few binarie= s are > > > > >>>>> concerned at the moment. So I applied INCLUDE_PIC_ARCHIVE in = the > > > > >> needed > > > > >>>>> libraries plus created WITH_PIE which add fPIE/fpie -pie flag= s only if > > > > >>>>> you > > > > >>>>> include (which include ...) ot= herwise > > > > >>>>> other > > > > >>>>> binaries include as usual hence does not apply.= Look > > > > >>>>> reasonable approach ? > > > > >>>> > > > > >>>> I think I understand what you mean. But I think PIE is common= place > > > > >>>> nowadays and I don't understand what you win by not enabling i= t for > > > > >>>> the whole system. Is it a performance concern? Is it to pres= erve > > > > >>>> conservative minds from to much change? :) > > > > >>> > > > > >>> > > > > >>> Looping in Kostik, Bryan Drewery, the PaX team, Hunger, and Sea= n Bruno. > > > > >>> > > > > >>> On i386, there is a performance cost due to not having an extra= register > > > > >>> available for the relocation work that has to happen. PIE doesn= 't carry > > > > >> much > > > > >>> of a performance penalty on amd64, though it still does carry s= ome on > > > > >> first > > > > >>> resolution of functions (due to the extra relocation step the R= TLD has to > > > > >>> worry about). On amd64, after symbol resolution has taken place= , there > > > > >> is no > > > > >>> further performance penalty due to amd64 having an extra regist= er to use > > > > >> for > > > > >>> PIE/PIC. I'm unsure what, if any, performance penalty PIE carri= es on ARM, > > > > >>> AArch64, and sparc64. > > > > >>> > > > > >>> Certain folk would prefer to see PIE enabled only in certain > > > > >> applications. > > > > >>> /bin/ls can't really make much use of PIE. But sshd can. I pers= onally > > > > >> would > > > > >>> like to see all of base's applications compiled as PIEs, but th= at's a > > > > >> long > > > > >>> ways off. It took OpenBSD several years to accomplish that. Hav= ing > > > > >> certain > > > > >>> high-visibility applications (like sshd, inetd, etc) is a great= start. > > > > >>> Providing a framework for application developers to opt their a= pplication > > > > >>> into PIE is another great start. > > > > >>> > > > > >>> Those are my two cents. > > > > >> > > > > >> OK. As long as i386 is still an important architecture, it can = make > > > > >> sense to enable this on a per-binary basis if we don't want to h= ave a > > > > >> discrepancy between archs. Also I buy your argument on /bin/ls b= ut I > > > > >> was challenging to enable for the whole system because I wonder = if > > > > >> there aren't some unexpected attack surfaces, besides the obviou= s ones > > > > >> (servers). > > > > >> > > > > >> Do you know what took so much time to OpenBSD? > > > > > > > > > > > > > > > In a private conversation with Theo, I realized that my recollect= ion of the > > > > > time it took OpenBSD to compile all of base as PIEs was wrong. Qu= oting him: > > > > > > > > > > "It took 5 people approximately 3 months to debug it, activate it= , and > > > > > start shipping it the next release. That was on amd64, for all > > > > > dynamically linked binaries, except one (a gcc bug took some time= to > > > > > find). The next architectures followed about 1 or 2 per 6-month > > > > > release." > > > > > > > > > > Given that only one person has worked on this in the past (me) an= d now the > > > > > task has been delegated to another (David Carlier), I think we're= doing > > > > > okay on our end. There's a lot of moving parts, and neither of us= fully > > > > > understand all of them completely. We're working on it in Hardene= dBSD, in > > > > > the hardened/current/pie branch. > > > > > > > > > > I'm thinking we might try for a WITH_PIE knob (and *not* use USE_= PIE) and > > > > > have certain high-profile applications opt-in to PIE until we wor= k out all > > > > > the details for everything en masse. Baptiste did bring up a good= point > > > > > with INTERNALLIB and I'm unsure of how we should handle that. > > > >=20 > > > > WITH_PIE or WITHOUT_PIE controls, on a global basis, via the MK_PIE > > > > variable, whether or not the user wants to turn on this feature for= those > > > > program that can do PIE. Designating which programs do, or don???t, > > > > use PIE simply must be done with a different variable. I posted a b= it of a > > > > rant about the current state of things that suggested a couple of > > > > alternatives as well as giving some history as to why some options > > > > aren???t to be used and the history behind some of my reactions. :) > > > >=20 > > > > For this reason, I think WITH_PIE, as I understand your proposal, > > > > likely isn???t a good fit with other WITH_xxx variables used in the= src > > > > tree today. > > > >=20 > > > > Gotcha. To be honest, I found your email a tad bit confusing. Are y= ou suggesting we create an ENABLE_feature framework? Or are you suggesting = we have a USE_PIE flag? Or are you suggesting something different entirely = (and if you are, what?)? > > >=20 > > > I???m saying we don???t have a good framework at the moment to do thi= s. We > > > have several bad ones that all have their pitfalls. This is one reaso= n I had > > > the fast reaction to NO_PIE, then a minute later said ???go ahead and= use > > > it and I???ll fix it.??? I???m still cool with that position, btw. > > >=20 > > > As for a name, that can be debated a lot, but I???d like to see some= thing > > > new, easy to use and unambiguous. If you are looking for a suggestion > > > for that name, let???s go with WANTS_PIE. Only Makefiles can set it. > > >=20 > > > WANTS_PIE undefined means do the default behavior as defined by the > > > current MK_PIE setting and perhaps system policy. ???Go with this flo= w." > > >=20 > > > WANTS_PIE=3Dyes means that if MK_PIE is ???yes???, then do PIE things= for > > > this thing we???re building. If MK_PIE is ???no???, though PIE is dis= abled for > > > everything. > > >=20 > > > WANTS_PIE=3Dno means that if MK_PIE is ???yes???, then disable doing = PIE > > > things for this component. If MK_PIE is no, it is also disabled. > > >=20 > > > This could also be extended to NEEDS_foo, which says ???I need foo to > > > build, and if MK_foo is set to no, don???t build me.??? I don???t thi= nk anything > > > that you are doing falls into this category though. > > >=20 > > > WANTS/NEEDS also avoids the historical use of USE in the ports tree > > > possibly creating confusion.=20 > > >=20 > > > If you go with WANTS_PIE, then you wouldn???t need bad.*.pie.mk. > > >=20 > > > Comments? > > >=20 > > > Warner > >=20 > > On amd64 WANTS_PIE will be useless as we can easily activate PIE on eve= ry places > > For i386 we would propably prefer cherry picking the what we want to se= e built > > with PIE. Don't know for other arches. > >=20 > > So here is what I do propose: > > if MK_PIE=3Dno: no PIE at all > > if MK_PIE=3Dyes: > > - on amd64/(platforms without performance penalty): build everything wi= th PIE > > from libs to prog > See below. >=20 > > - on i386/(platforms with performance penalty): build with PIE if WANTS= _PIE > > is defined. > >=20 > > So the difference with the previous approach are: > > - No way to opt out PIE for a single binary either totally disable or e= nable (I > > have encountered no binary so far in the base system which fails with= PIE > > enabled - again only tested on amd64) > > - Activate PIE for both binaries and libraries (no reason not to include > > libraries) > What does it mean 'PIE for library' ? There is simply no such thing. Sorry I badly explained, I was meaning PIC for libs PIE for binaries. >=20 > Also, I strongly oppose compiling everything with PIC, even on amd64. > I described somewhere else that using PIC code changes symbol lookup > rules for binaries. So despite not having performance impact, the > thing does impact runtime behaviour in subtle ways. The most affected > programs are those which support dynamic modules. >=20 > Also, what is the state of static binaries + PIE ? Do our binutils > support this at all ? The csu is definitely not ready for 'everything > PIE'. Only dynamic binaries will receive PIE support (and in case of using an INTERNALLIB will link to the libbla_pic.a) static ones will remain non PIE. regards, Bapt --wj9ZLJVQDRFjGSdK Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlRaDqcACgkQ8kTtMUmk6ExcHgCfSOml3H4uNy+jydng/YK8vPvb fJkAn0JRULihj9CuCOIcFwdkSCiw0wwt =nVSZ -----END PGP SIGNATURE----- --wj9ZLJVQDRFjGSdK-- From owner-freebsd-arch@FreeBSD.ORG Wed Nov 5 12:26:19 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E1B729DE; Wed, 5 Nov 2014 12:26:18 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 83D90FB9; Wed, 5 Nov 2014 12:26:18 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id sA5CQC8k065390 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 5 Nov 2014 14:26:12 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua sA5CQC8k065390 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id sA5CQCrQ065389; Wed, 5 Nov 2014 14:26:12 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 5 Nov 2014 14:26:12 +0200 From: Konstantin Belousov To: Baptiste Daroussin Subject: Re: PIE/PIC support on base Message-ID: <20141105122612.GC53947@kib.kiev.ua> References: <315B4DC5-0E04-4F6B-BBB0-477D049025BF@bsdimp.com> <20141105090215.GF10388@ivaldir.etoilebsd.net> <20141105092614.GB53947@kib.kiev.ua> <20141105114855.GH10388@ivaldir.etoilebsd.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20141105114855.GH10388@ivaldir.etoilebsd.net> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: PaX Team , FreeBSD Arch , Shawn Webb X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 12:26:19 -0000 On Wed, Nov 05, 2014 at 12:48:55PM +0100, Baptiste Daroussin wrote: > On Wed, Nov 05, 2014 at 11:26:14AM +0200, Konstantin Belousov wrote: > > On Wed, Nov 05, 2014 at 10:02:15AM +0100, Baptiste Daroussin wrote: > > > On amd64 WANTS_PIE will be useless as we can easily activate PIE on every places > > > For i386 we would propably prefer cherry picking the what we want to see built > > > with PIE. Don't know for other arches. > > > > > > So here is what I do propose: > > > if MK_PIE=no: no PIE at all > > > if MK_PIE=yes: > > > - on amd64/(platforms without performance penalty): build everything with PIE > > > from libs to prog > > See below. > > > > > - on i386/(platforms with performance penalty): build with PIE if WANTS_PIE > > > is defined. > > > > > > So the difference with the previous approach are: > > > - No way to opt out PIE for a single binary either totally disable or enable (I > > > have encountered no binary so far in the base system which fails with PIE > > > enabled - again only tested on amd64) > > > - Activate PIE for both binaries and libraries (no reason not to include > > > libraries) > > What does it mean 'PIE for library' ? There is simply no such thing. > > Sorry I badly explained, I was meaning PIC for libs PIE for binaries. > > > > Also, I strongly oppose compiling everything with PIC, even on amd64. > > I described somewhere else that using PIC code changes symbol lookup > > rules for binaries. So despite not having performance impact, the > > thing does impact runtime behaviour in subtle ways. The most affected > > programs are those which support dynamic modules. Please do not ignore this ^^^^^^ issue. > > > > Also, what is the state of static binaries + PIE ? Do our binutils > > support this at all ? The csu is definitely not ready for 'everything > > PIE'. > > Only dynamic binaries will receive PIE support (and in case of using an > INTERNALLIB will link to the libbla_pic.a) static ones will remain non PIE. And what about libX.a libraries, required by those static binaries ? It is wrong to compile the .o files for those static libraries in pic mode. More, take look at things which are done with -DPIC, e.g. in the lib/libc/sys/stack_protector*.c. There, it is critical for correctness. From owner-freebsd-arch@FreeBSD.ORG Wed Nov 5 12:44:32 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 53E0E157 for ; Wed, 5 Nov 2014 12:44:32 +0000 (UTC) Received: from mail-wg0-x22a.google.com (mail-wg0-x22a.google.com [IPv6:2a00:1450:400c:c00::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D52EA2A9 for ; Wed, 5 Nov 2014 12:44:31 +0000 (UTC) Received: by mail-wg0-f42.google.com with SMTP id k14so819326wgh.1 for ; Wed, 05 Nov 2014 04:44:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=ducp8V4151aeMY1vAAIoX0VIFd/G0Lu/IbVXwgeqodk=; b=P7jcxylvEEwWXD+nPc6j4gpJw2Bh9JcfouaagDrHkNsI85R0O2bpL7qolP20ENe182 nUuADqtI6xixSrsIhW6TPguoKJFZk537WpPTYKwkyTwoZKxYR8np0V2pqow963Egl+Sr H8fAC7/sFUf/ENsGd0oXW+GXCy9seiZxrMXvn0SQ/rM7hmn4AtjpwhOYrpkBr281z2X+ 2Wn3YIfvSPpN/ISR0ziFMEzcPshPdEV2jyuAnAInTrjMKLS9sZawm0lGiH4F3ESMyuC8 9NJ49PjMzb6MK7pXUGdpngF+OJ9F13i8Sq5w07QOs7Gg3fCqTSUbkcZTARnN92mh1MOg YhbA== X-Received: by 10.194.250.41 with SMTP id yz9mr17492234wjc.34.1415191470081; Wed, 05 Nov 2014 04:44:30 -0800 (PST) Received: from ivaldir.etoilebsd.net ([2001:41d0:8:db4c::1]) by mx.google.com with ESMTPSA id bj7sm3956995wjc.33.2014.11.05.04.44.28 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Nov 2014 04:44:28 -0800 (PST) Sender: Baptiste Daroussin Date: Wed, 5 Nov 2014 13:44:26 +0100 From: Baptiste Daroussin To: Konstantin Belousov Subject: Re: PIE/PIC support on base Message-ID: <20141105124426.GI10388@ivaldir.etoilebsd.net> References: <315B4DC5-0E04-4F6B-BBB0-477D049025BF@bsdimp.com> <20141105090215.GF10388@ivaldir.etoilebsd.net> <20141105092614.GB53947@kib.kiev.ua> <20141105114855.GH10388@ivaldir.etoilebsd.net> <20141105122612.GC53947@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="6b3yLyRKT1M6kiA0" Content-Disposition: inline In-Reply-To: <20141105122612.GC53947@kib.kiev.ua> User-Agent: Mutt/1.5.23 (2014-03-12) Cc: PaX Team , FreeBSD Arch , Shawn Webb X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 12:44:32 -0000 --6b3yLyRKT1M6kiA0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Nov 05, 2014 at 02:26:12PM +0200, Konstantin Belousov wrote: > On Wed, Nov 05, 2014 at 12:48:55PM +0100, Baptiste Daroussin wrote: > > On Wed, Nov 05, 2014 at 11:26:14AM +0200, Konstantin Belousov wrote: > > > On Wed, Nov 05, 2014 at 10:02:15AM +0100, Baptiste Daroussin wrote: > > > > On amd64 WANTS_PIE will be useless as we can easily activate PIE on= every places > > > > For i386 we would propably prefer cherry picking the what we want t= o see built > > > > with PIE. Don't know for other arches. > > > >=20 > > > > So here is what I do propose: > > > > if MK_PIE=3Dno: no PIE at all > > > > if MK_PIE=3Dyes: > > > > - on amd64/(platforms without performance penalty): build everythin= g with PIE > > > > from libs to prog > > > See below. > > >=20 > > > > - on i386/(platforms with performance penalty): build with PIE if W= ANTS_PIE > > > > is defined. > > > >=20 > > > > So the difference with the previous approach are: > > > > - No way to opt out PIE for a single binary either totally disable = or enable (I > > > > have encountered no binary so far in the base system which fails = with PIE > > > > enabled - again only tested on amd64) > > > > - Activate PIE for both binaries and libraries (no reason not to in= clude > > > > libraries) > > > What does it mean 'PIE for library' ? There is simply no such thing. > >=20 > > Sorry I badly explained, I was meaning PIC for libs PIE for binaries. > > >=20 > > > Also, I strongly oppose compiling everything with PIC, even on amd64. > > > I described somewhere else that using PIC code changes symbol lookup > > > rules for binaries. So despite not having performance impact, the > > > thing does impact runtime behaviour in subtle ways. The most affected > > > programs are those which support dynamic modules. > Please do not ignore this ^^^^^^ issue. I was not aware of issues here, I'll investigate but will not ignore for us= re :) >=20 > > >=20 > > > Also, what is the state of static binaries + PIE ? Do our binutils > > > support this at all ? The csu is definitely not ready for 'everything > > > PIE'. > >=20 > > Only dynamic binaries will receive PIE support (and in case of using an > > INTERNALLIB will link to the libbla_pic.a) static ones will remain non = PIE. >=20 > And what about libX.a libraries, required by those static binaries ? > It is wrong to compile the .o files for those static libraries in > pic mode. I was not planning to build .a files with PIC, static binaries at all >=20 > More, take look at things which are done with -DPIC, e.g. in the > lib/libc/sys/stack_protector*.c. There, it is critical for correctness. >=20 >=20 I'll have a look thanks for the pointer! regards, Bapt --6b3yLyRKT1M6kiA0 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlRaG6oACgkQ8kTtMUmk6Ew/DACdGq8qSKEjj04H4ImCimoqRcSb vnkAn3dzT/TUZo4EJiouy3fjZISyOtzC =MHgS -----END PGP SIGNATURE----- --6b3yLyRKT1M6kiA0-- From owner-freebsd-arch@FreeBSD.ORG Wed Nov 5 12:54:39 2014 Return-Path: Delivered-To: arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9D4BD362; Wed, 5 Nov 2014 12:54:39 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0F9A23CE; Wed, 5 Nov 2014 12:54:38 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id sA5CsVnx070997 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 5 Nov 2014 14:54:31 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua sA5CsVnx070997 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id sA5CsV2r070996; Wed, 5 Nov 2014 14:54:31 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 5 Nov 2014 14:54:31 +0200 From: Konstantin Belousov To: Baptiste Daroussin Subject: Re: Overlinking in base Message-ID: <20141105125431.GD53947@kib.kiev.ua> References: <20141105113839.GG10388@ivaldir.etoilebsd.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20141105113839.GG10388@ivaldir.etoilebsd.net> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: arch@FreeBSD.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 12:54:39 -0000 On Wed, Nov 05, 2014 at 12:38:39PM +0100, Baptiste Daroussin wrote: > Hi all, > > We do overlink a lot in the base system. > > I wrote a script to check overlinking on the final binaries and running it on > /usr/bin files I got the following results: > https://wiki.freebsd.org/Overlinking > > Most of the overlink are due to LDADD having all the recursive libraries which > for dynamic linking we do not need. > > We might need them for static linking, but I think we should fix those. Fix how ? Static libraries do not record dependencies, and linker must get list of all libraries to satisfy undefined symbols. > > I think anyway a binary Makefile should only declare its direct dependencies and > not the dependencies of its dependencies. No, this is wrong definition of overlinking. Binary X, which depends on liba.so and libb.so, is overlinked, only when there is no symbol from libb.so which is directly referenced from X. Mere fact that liba.so needs libb.so and X records DT_NEEDED for both a and b, is not enough. Could you, please, share the script to see how the overlinking is checked ? > > I was already proposing in the past to import pkgconf in base to solve this as > the library will declare its dependencies in case of dynamic linking and/or in > case of static linking (which can be different) > > This can also be done in pure make(1) > > regards, > Bapt From owner-freebsd-arch@FreeBSD.ORG Wed Nov 5 12:59:26 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B39F34AA; Wed, 5 Nov 2014 12:59:26 +0000 (UTC) Received: from mail-qg0-x22f.google.com (mail-qg0-x22f.google.com [IPv6:2607:f8b0:400d:c04::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5AEC05EA; Wed, 5 Nov 2014 12:59:26 +0000 (UTC) Received: by mail-qg0-f47.google.com with SMTP id j107so11996477qga.6 for ; Wed, 05 Nov 2014 04:59:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=user-agent:in-reply-to:references:mime-version :content-transfer-encoding:content-type:subject:from:date:to:cc :message-id; bh=jhAc7kmMN9FDThei+ojVlHXfnKJSic5jYW4tVm+WWk4=; b=FqWMldUOFGHNvltqSu/wpWYesJt1UkL8NE0mr54jLPX0Xvt6XjUOqevVlTajJJXR5r Ad9H/XAYSI5eux4DV0CXESsWbA7EGA/ORYwFkr791kD1kXbW7OoI99we2FX9sN4RbGBs hU9yDtgec1mKLvYdStnyM8cUDlLQvRWpQh550KMLPg10njFYCzJGrSaNAqvHjVYAhHmR jSFr+sWkiaHqPbVbgnQV3XykuKFe4SG06f/R0LiZ19errzNIVN98aY9vA/V+5pPkSe4y 5eeUGQVfyCYdaWeh0ZWJR4RdwdOPXSRFEEIJEwi8FO2KY0NDhMt726Zlc2eUgJ0kKla2 Xc4A== X-Received: by 10.224.41.142 with SMTP id o14mr87970839qae.100.1415192364889; Wed, 05 Nov 2014 04:59:24 -0800 (PST) Received: from ?IPv6:2601:a:1380:1046:4cde:a114:1388:ec9c? ([2601:a:1380:1046:4cde:a114:1388:ec9c]) by mx.google.com with ESMTPSA id d2sm3056110qab.24.2014.11.05.04.59.24 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Nov 2014 04:59:24 -0800 (PST) User-Agent: K-9 Mail for Android In-Reply-To: <20141105122612.GC53947@kib.kiev.ua> References: <315B4DC5-0E04-4F6B-BBB0-477D049025BF@bsdimp.com> <20141105090215.GF10388@ivaldir.etoilebsd.net> <20141105092614.GB53947@kib.kiev.ua> <20141105114855.GH10388@ivaldir.etoilebsd.net> <20141105122612.GC53947@kib.kiev.ua> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Subject: Re: PIE/PIC support on base From: Shawn Webb Date: Wed, 05 Nov 2014 07:59:21 -0500 To: Konstantin Belousov , Baptiste Daroussin Message-ID: Cc: FreeBSD Arch X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 12:59:26 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On November 5, 2014 7:26:12 AM EST, Konstantin Belousov wrote: >On Wed, Nov 05, 2014 at 12:48:55PM +0100, Baptiste Daroussin wrote: >> On Wed, Nov 05, 2014 at 11:26:14AM +0200, Konstantin Belousov wrote: >> > On Wed, Nov 05, 2014 at 10:02:15AM +0100, Baptiste Daroussin wrote: >> > > On amd64 WANTS_PIE will be useless as we can easily activate PIE >on every places >> > > For i386 we would propably prefer cherry picking the what we want >to see built >> > > with PIE. Don't know for other arches. >> > > >> > > So here is what I do propose: >> > > if MK_PIE=no: no PIE at all >> > > if MK_PIE=yes: >> > > - on amd64/(platforms without performance penalty): build >everything with PIE >> > > from libs to prog >> > See below. >> > >> > > - on i386/(platforms with performance penalty): build with PIE if >WANTS_PIE >> > > is defined. >> > > >> > > So the difference with the previous approach are: >> > > - No way to opt out PIE for a single binary either totally >disable or enable (I >> > > have encountered no binary so far in the base system which >fails with PIE >> > > enabled - again only tested on amd64) >> > > - Activate PIE for both binaries and libraries (no reason not to >include >> > > libraries) >> > What does it mean 'PIE for library' ? There is simply no such >thing. >> >> Sorry I badly explained, I was meaning PIC for libs PIE for binaries. >> > >> > Also, I strongly oppose compiling everything with PIC, even on >amd64. >> > I described somewhere else that using PIC code changes symbol >lookup >> > rules for binaries. So despite not having performance impact, the >> > thing does impact runtime behaviour in subtle ways. The most >affected >> > programs are those which support dynamic modules. >Please do not ignore this ^^^^^^ issue. > Can you go into detail what those changes are? >> > >> > Also, what is the state of static binaries + PIE ? Do our binutils >> > support this at all ? The csu is definitely not ready for >'everything >> > PIE'. >> >> Only dynamic binaries will receive PIE support (and in case of using >an >> INTERNALLIB will link to the libbla_pic.a) static ones will remain >non PIE. > >And what about libX.a libraries, required by those static binaries ? >It is wrong to compile the .o files for those static libraries in >pic mode. > >More, take look at things which are done with -DPIC, e.g. in the >lib/libc/sys/stack_protector*.c. There, it is critical for >correctness. > > >_______________________________________________ >freebsd-arch@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-arch >To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" - -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -----BEGIN PGP SIGNATURE----- Version: APG v1.1.1 iQI8BAEBCgAmBQJUWh8pHxxTaGF3biBXZWJiIDxsYXR0ZXJhQGdtYWlsLmNvbT4A CgkQaoRlj1JFbu7dvA//UsGATM6oO3Wutl6AeD17GyAd3b8LcYnxeF2mRV7EWwn+ o0I2Riact18cjO8VvCKqb3PGDdiARJK9Qi7kcJz2zQeMg6CvNWKfXBZ+W3Fp5KeC 8dhFaw+vzxG/FaAE/ZfsyUJhjB2kDWPrMqqoui1HNWG3AG8P92r7XEphs3XMX5+s tS68r1P9C02q+jI/GgIyyNmReBlRKdAN7g05vW08TfGSPiI2SG5pd6q7zHn/iVXX eHiHZ06mS3ljpm6rTfrNZlUqX8sPH8/wGGGcR9zWlHhxT8fvdY+2jZ59R2ddCcHu pRZ4oZ3RTMFCh7vBCPPC1FjjzWmp6CNnRu51Ud+w2Yau0bt9AyAcplCCEhiO+jDI ZjWTgkLMnZRyarnHqaF7eHtKRifKL29k/D4Uc3eL5mK8UYHtbDIAAlNAYWdDecwt E8lYT4VUWnw7/slGRt6Zv+lbieEk0Q5iHyNaoE28BZnOLC6smLeNd1CAz7bOradb 8c2C5NH7oOWqV9ZarX2oSgBQdFLhRRTLAKUFBbDY22nRBpLdNKCF48MIzds5ZCo8 dlxSM6DjnJQF4dtIM9VL6zr3o9SqCLn1mGqkAABAYvnTo/zBMJ3u21ayzWYMYAs6 NWD9a/+Iu3dYP4uyi2139/Pvwy6C/Riuiag4iVk+/qc9m42FnNAPzsPgHIEtgRY= =HscB -----END PGP SIGNATURE----- From owner-freebsd-arch@FreeBSD.ORG Wed Nov 5 13:00:02 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 49CF055E for ; Wed, 5 Nov 2014 13:00:02 +0000 (UTC) Received: from mail-wi0-x22e.google.com (mail-wi0-x22e.google.com [IPv6:2a00:1450:400c:c05::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D1E925F4 for ; Wed, 5 Nov 2014 13:00:01 +0000 (UTC) Received: by mail-wi0-f174.google.com with SMTP id d1so12321945wiv.7 for ; Wed, 05 Nov 2014 05:00:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=jg0ULKn5FNJrpGQCpMntiaOiV72awPbzxnpD3cXzJAw=; b=cuMNKuHMx3uYRE2tc/2yzjpvjqI7SshmtCpQM1sVO+zMkCFxo+/CosqrUmzM8cDAbz ODx2LTejPg50fUqmzgQwxrZWlZtHaDFxlWbfWDNbYXB8ocWy1YhuEvAxA0bIFhJo3Yzh M5zvaorbJe5DPSEJ+33FoY3RVVINAld/PKsHsmWaPl1mWxqjpvdYgorLEHUEz0p14OuA 8U84aJgBitMdQ1LV4PTZD5UaFmv6o4T+EL2Cna7uygDFq+HjIK2Sdq+WPabZZnALIYWQ 3m/cuuYCaYpHtdf8WiUxHZPfk43hccE1k4u+hUjWWRet7AoZCTDO9z9bMxdzfENvMgWE IF9w== X-Received: by 10.195.12.45 with SMTP id en13mr54714149wjd.8.1415192400094; Wed, 05 Nov 2014 05:00:00 -0800 (PST) Received: from ivaldir.etoilebsd.net ([2001:41d0:8:db4c::1]) by mx.google.com with ESMTPSA id fa16sm15749717wid.5.2014.11.05.04.59.58 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Nov 2014 04:59:59 -0800 (PST) Sender: Baptiste Daroussin Date: Wed, 5 Nov 2014 13:59:57 +0100 From: Baptiste Daroussin To: Konstantin Belousov Subject: Re: Overlinking in base Message-ID: <20141105125931.GJ10388@ivaldir.etoilebsd.net> References: <20141105113839.GG10388@ivaldir.etoilebsd.net> <20141105125431.GD53947@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="EMQjp+MvU6EBGjHc" Content-Disposition: inline In-Reply-To: <20141105125431.GD53947@kib.kiev.ua> User-Agent: Mutt/1.5.23 (2014-03-12) Cc: arch@FreeBSD.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 13:00:02 -0000 --EMQjp+MvU6EBGjHc Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Nov 05, 2014 at 02:54:31PM +0200, Konstantin Belousov wrote: > On Wed, Nov 05, 2014 at 12:38:39PM +0100, Baptiste Daroussin wrote: > > Hi all, > >=20 > > We do overlink a lot in the base system. > >=20 > > I wrote a script to check overlinking on the final binaries and running= it on > > /usr/bin files I got the following results: > > https://wiki.freebsd.org/Overlinking > >=20 > > Most of the overlink are due to LDADD having all the recursive librarie= s which > > for dynamic linking we do not need. > >=20 > > We might need them for static linking, but I think we should fix those. > Fix how ? Static libraries do not record dependencies, and linker > must get list of all libraries to satisfy undefined symbols. >=20 > >=20 > > I think anyway a binary Makefile should only declare its direct depende= ncies and > > not the dependencies of its dependencies. > No, this is wrong definition of overlinking. Binary X, which depends > on liba.so and libb.so, is overlinked, only when there is no symbol from > libb.so which is directly referenced from X. Mere fact that liba.so > needs libb.so and X records DT_NEEDED for both a and b, is not enough. That is what I do check and what I'm referencing, a good example is bsdtar = which links to all compression libs while libarchive itself does not expose any symbols from them. >=20 > Could you, please, share the script to see how the overlinking is > checked ? Here you are: https://people.freebsd.org/~bapt/check-links.sh Beware it is dirty :) Run it as check-links.sh nameofthebinary Best regards, Bapt --EMQjp+MvU6EBGjHc Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlRaH00ACgkQ8kTtMUmk6EwM6ACgsV9NtU61gvNZ1OplOZt0BQKP v70AnAyNK//25wttw1bMxptT9M7t6x0A =EsUl -----END PGP SIGNATURE----- --EMQjp+MvU6EBGjHc-- From owner-freebsd-arch@FreeBSD.ORG Wed Nov 5 13:19:44 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C4E4FA29; Wed, 5 Nov 2014 13:19:44 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 309DC84D; Wed, 5 Nov 2014 13:19:44 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id sA5DJdwc076615 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 5 Nov 2014 15:19:39 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua sA5DJdwc076615 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id sA5DJdkH076614; Wed, 5 Nov 2014 15:19:39 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 5 Nov 2014 15:19:39 +0200 From: Konstantin Belousov To: Shawn Webb Subject: Re: PIE/PIC support on base Message-ID: <20141105131939.GF53947@kib.kiev.ua> References: <315B4DC5-0E04-4F6B-BBB0-477D049025BF@bsdimp.com> <20141105090215.GF10388@ivaldir.etoilebsd.net> <20141105092614.GB53947@kib.kiev.ua> <20141105114855.GH10388@ivaldir.etoilebsd.net> <20141105122612.GC53947@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: Baptiste Daroussin , FreeBSD Arch X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 13:19:44 -0000 On Wed, Nov 05, 2014 at 07:59:21AM -0500, Shawn Webb wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > > > On November 5, 2014 7:26:12 AM EST, Konstantin Belousov wrote: > >On Wed, Nov 05, 2014 at 12:48:55PM +0100, Baptiste Daroussin wrote: > >> > Also, I strongly oppose compiling everything with PIC, even on > >amd64. > >> > I described somewhere else that using PIC code changes symbol > >lookup > >> > rules for binaries. So despite not having performance impact, the > >> > thing does impact runtime behaviour in subtle ways. The most > >affected > >> > programs are those which support dynamic modules. > >Please do not ignore this ^^^^^^ issue. > > > > Can you go into detail what those changes are? PIE binary resolves its symbols through GOT/PLT. In particular, it is possible to interpose the symbols. From owner-freebsd-arch@FreeBSD.ORG Wed Nov 5 13:30:35 2014 Return-Path: Delivered-To: arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A8515AC; Wed, 5 Nov 2014 13:30:35 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 12172A79; Wed, 5 Nov 2014 13:30:34 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id sA5DUTFI079143 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 5 Nov 2014 15:30:29 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua sA5DUTFI079143 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id sA5DUTFw079141; Wed, 5 Nov 2014 15:30:29 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 5 Nov 2014 15:30:29 +0200 From: Konstantin Belousov To: Baptiste Daroussin Subject: Re: Overlinking in base Message-ID: <20141105133029.GH53947@kib.kiev.ua> References: <20141105113839.GG10388@ivaldir.etoilebsd.net> <20141105125431.GD53947@kib.kiev.ua> <20141105125931.GJ10388@ivaldir.etoilebsd.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20141105125931.GJ10388@ivaldir.etoilebsd.net> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: arch@FreeBSD.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 13:30:35 -0000 On Wed, Nov 05, 2014 at 01:59:57PM +0100, Baptiste Daroussin wrote: > On Wed, Nov 05, 2014 at 02:54:31PM +0200, Konstantin Belousov wrote: > > Could you, please, share the script to see how the overlinking is > > checked ? > > Here you are: > https://people.freebsd.org/~bapt/check-links.sh > > Beware it is dirty :) > > Run it as check-links.sh nameofthebinary Ok. It is mostly fine, but you do not account for symbol versions of the looked up symbols. There were weird changes, e.g. isnanf story, which essentially migrated from libc to libm. I suspect such cases are not very important. From owner-freebsd-arch@FreeBSD.ORG Wed Nov 5 13:40:13 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A5465345 for ; Wed, 5 Nov 2014 13:40:13 +0000 (UTC) Received: from mail-wi0-x236.google.com (mail-wi0-x236.google.com [IPv6:2a00:1450:400c:c05::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 38D06B5A for ; Wed, 5 Nov 2014 13:40:13 +0000 (UTC) Received: by mail-wi0-f182.google.com with SMTP id d1so2038051wiv.3 for ; Wed, 05 Nov 2014 05:40:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=eDeih+Y4JcNzYxuXszhLRp2xcjiYkSxWpdfODIOX4gk=; b=JCJDbh0XIpX6pU/OKB8vjsSMMaG1dDgfK3H49sTOE3xrC79gAANg19tHRlh6qnKtWW y9qsbp6ARkNPWt6jtA4EhOiAzKxBtATXNcqVxeGAusJ7pxMsNpQtjNr+P2Q/dsgKeHle 7NAoucZ21QJNAe9TSzzyS8+zF3PgcZ/KkcAZFG5QpCw/lfQqV3ilALF/9wkk+l5YEDVU PuREnXuU1KSvh/Y3JgPAnIcrIW02zHrdbKRZiKkaGg5nOeysZuaQRpBifL5LGAgufFdg kmjTmsXycoeX1Ep/OeSpMILPtmKVn6FK6H383aThbaeHdDD3GRDcPtgdKSMnj36loKMx JdJw== X-Received: by 10.180.36.229 with SMTP id t5mr32155923wij.56.1415194810231; Wed, 05 Nov 2014 05:40:10 -0800 (PST) Received: from ivaldir.etoilebsd.net ([2001:41d0:8:db4c::1]) by mx.google.com with ESMTPSA id n4sm4109182wjb.40.2014.11.05.05.40.08 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Nov 2014 05:40:09 -0800 (PST) Sender: Baptiste Daroussin Date: Wed, 5 Nov 2014 14:40:07 +0100 From: Baptiste Daroussin To: Konstantin Belousov Subject: Re: Overlinking in base Message-ID: <20141105134006.GL10388@ivaldir.etoilebsd.net> References: <20141105113839.GG10388@ivaldir.etoilebsd.net> <20141105125431.GD53947@kib.kiev.ua> <20141105125931.GJ10388@ivaldir.etoilebsd.net> <20141105133029.GH53947@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="g6DVDhPhk1bqxDrC" Content-Disposition: inline In-Reply-To: <20141105133029.GH53947@kib.kiev.ua> User-Agent: Mutt/1.5.23 (2014-03-12) Cc: arch@FreeBSD.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 13:40:13 -0000 --g6DVDhPhk1bqxDrC Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Nov 05, 2014 at 03:30:29PM +0200, Konstantin Belousov wrote: > On Wed, Nov 05, 2014 at 01:59:57PM +0100, Baptiste Daroussin wrote: > > On Wed, Nov 05, 2014 at 02:54:31PM +0200, Konstantin Belousov wrote: > > > Could you, please, share the script to see how the overlinking is > > > checked ? > >=20 > > Here you are: > > https://people.freebsd.org/~bapt/check-links.sh > >=20 > > Beware it is dirty :) > >=20 > > Run it as check-links.sh nameofthebinary >=20 > Ok. It is mostly fine, but you do not account for symbol versions of > the looked up symbols. There were weird changes, e.g. isnanf story, > which essentially migrated from libc to libm. I suspect such cases > are not very important. >=20 My proposal to fix this overlinking while still supporting static linkage Is to change the way we are declaring those dependencies, imho the library should declare what it needs in case of dynamic linking and what it needs in case of static linking, the binary Makefile should only list what it requir= es as a direct dependency and the framework should do all the magic. This can be done via .pc files (and calling pkgconf) or can be done via .mk files in the library directory. In the first case we could have something like: PCADD=3D liba libb libc Which will result in the build system querying though pkgconf: pkgconf --libs (--static if calling static linkage) In the second case we could do it via make(1) LIBADD=3D liba libc libc this will open something like a ${PATHTOTHELIB}/link.mk which will define DYNAMIC_ADD STATIC_ADD And this could be recursive. (note that pkgconf is also recursive as well). regards, Bapt --g6DVDhPhk1bqxDrC Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlRaKLYACgkQ8kTtMUmk6Ewl7ACghU3tcCk8YOj0EzEdlj4HXgaT DNUAn39NjeESZ4xiEAvefDfI141usQxn =86Gt -----END PGP SIGNATURE----- --g6DVDhPhk1bqxDrC-- From owner-freebsd-arch@FreeBSD.ORG Wed Nov 5 13:57:59 2014 Return-Path: Delivered-To: arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4B29378B; Wed, 5 Nov 2014 13:57:59 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B0C57E29; Wed, 5 Nov 2014 13:57:58 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id sA5DvrlJ085550 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 5 Nov 2014 15:57:53 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua sA5DvrlJ085550 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id sA5Dvph1085549; Wed, 5 Nov 2014 15:57:51 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 5 Nov 2014 15:57:51 +0200 From: Konstantin Belousov To: Baptiste Daroussin Subject: Re: Overlinking in base Message-ID: <20141105135751.GI53947@kib.kiev.ua> References: <20141105113839.GG10388@ivaldir.etoilebsd.net> <20141105125431.GD53947@kib.kiev.ua> <20141105125931.GJ10388@ivaldir.etoilebsd.net> <20141105133029.GH53947@kib.kiev.ua> <20141105134006.GL10388@ivaldir.etoilebsd.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20141105134006.GL10388@ivaldir.etoilebsd.net> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: arch@FreeBSD.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 13:57:59 -0000 On Wed, Nov 05, 2014 at 02:40:07PM +0100, Baptiste Daroussin wrote: > On Wed, Nov 05, 2014 at 03:30:29PM +0200, Konstantin Belousov wrote: > > On Wed, Nov 05, 2014 at 01:59:57PM +0100, Baptiste Daroussin wrote: > > > On Wed, Nov 05, 2014 at 02:54:31PM +0200, Konstantin Belousov wrote: > > > > Could you, please, share the script to see how the overlinking is > > > > checked ? > > > > > > Here you are: > > > https://people.freebsd.org/~bapt/check-links.sh > > > > > > Beware it is dirty :) > > > > > > Run it as check-links.sh nameofthebinary > > > > Ok. It is mostly fine, but you do not account for symbol versions of > > the looked up symbols. There were weird changes, e.g. isnanf story, > > which essentially migrated from libc to libm. I suspect such cases > > are not very important. > > > My proposal to fix this overlinking while still supporting static linkage > Is to change the way we are declaring those dependencies, imho the library > should declare what it needs in case of dynamic linking and what it needs in > case of static linking, the binary Makefile should only list what it requires as > a direct dependency and the framework should do all the magic. > > This can be done via .pc files (and calling pkgconf) or can be done via .mk > files in the library directory. > > In the first case we could have something like: > PCADD= liba libb libc > Which will result in the build system querying though pkgconf: > pkgconf --libs (--static if calling static linkage) > > In the second case we could do it via make(1) > LIBADD= liba libc libc > this will open something like a ${PATHTOTHELIB}/link.mk which will define > DYNAMIC_ADD > STATIC_ADD > > And this could be recursive. > > (note that pkgconf is also recursive as well). Well, there is third option, the linker option --as-needed (I think our old binutils are new enough). Personally, I would prefer make-based solution described above, but --as-needed may be used as the the check. Or, we should have an easy escape to --no-as-needed. From owner-freebsd-arch@FreeBSD.ORG Wed Nov 5 22:37:32 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 944EC55E for ; Wed, 5 Nov 2014 22:37:32 +0000 (UTC) Received: from mail-pa0-f41.google.com (mail-pa0-f41.google.com [209.85.220.41]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 642061E9 for ; Wed, 5 Nov 2014 22:37:32 +0000 (UTC) Received: by mail-pa0-f41.google.com with SMTP id rd3so1686004pab.14 for ; Wed, 05 Nov 2014 14:37:26 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:content-type:mime-version:subject:from :in-reply-to:date:cc:message-id:references:to; bh=Fd+1I7xY1sgWlYaPSN+ePsr1pQKoMZTIlmN0TeNO3Eg=; b=GWx16OVf6a1X/z1SBIsrujFkxUhxT8TAowyirD41isLUoMpTojFs2zA+LAULu2+47t f9k+xsemUb16kjZl2p4S8INqUYLECrDSoe5qXAgaXisPRfCmFOYtW/n47XzMVhGwkvZo wJwyyltGkJqEgssdS7dDwUhEBO2goGgfNDu+kPGNn+gVcg6/0Vv6lf8Iscn03dj19W6b edXK9upu7Bp5bQt/K7Wi9FnU+cRgQ5O3qEs94wTQCM2OOAtqBeGYokRFcdi5d6ZGLU33 iZZi71QURyIhD6vKtFZXdUZZ5CRHicdQmt1WhKbyWaMCREGX11omXXHicSPf7DHrP1Q6 GQ8A== X-Gm-Message-State: ALoCoQlVWirenLYpyyEGG+2XTYJTGHU52tTJ5/H7kW8WjqaEAxIdXqZPRnjtAWy+Brp03qVvWSlJ X-Received: by 10.70.129.209 with SMTP id ny17mr49573pdb.163.1415227045929; Wed, 05 Nov 2014 14:37:25 -0800 (PST) Received: from [10.64.26.112] ([69.53.236.236]) by mx.google.com with ESMTPSA id c9sm4054365pdn.81.2014.11.05.14.37.24 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 05 Nov 2014 14:37:25 -0800 (PST) Sender: Warner Losh Content-Type: multipart/signed; boundary="Apple-Mail=_1783D049-BACA-47C2-A2BF-73289A959AB7"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Why do we have @ in modules builds? From: Warner Losh In-Reply-To: <545AA4C5.1060709@cox.net> Date: Wed, 5 Nov 2014 15:37:13 -0700 Message-Id: References: <545AA4C5.1060709@cox.net> To: "John D. Hendrickson" X-Mailer: Apple Mail (2.1878.6) Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 22:37:32 -0000 --Apple-Mail=_1783D049-BACA-47C2-A2BF-73289A959AB7 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Nov 5, 2014, at 3:29 PM, John D. Hendrickson = wrote: > Warner Losh wrote: >> Does anybody recall why we have @ symlink in our module builds? >> I=92m constantly working around issues that this creates. Maybe it is = time >> to eliminate it? >> So I=92ve posted the following review: = https://reviews.freebsd.org/D1100 >> Warner >=20 > could you please post what it is that "does not work=94 Having a dependency that start with @ simply doesn=92t work. > and what is is "you must do" to work around it ? .if exists(@) foo.x: @/dev/foo_bar.y .else foo.x: @ .endif This is needlessly complicated and somewhat annoying when there=92s = already machinery in place that locates the base of the kernel tree. Warner --Apple-Mail=_1783D049-BACA-47C2-A2BF-73289A959AB7 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJUWqaZAAoJEGwc0Sh9sBEA6Q0P+wQUbIbJmbp5iiDby9iKUpq7 nDmVVs6oN3tNvCI0Tp1VEjXfT/gij69ViMhnRWQPJhhjyechSiAB3exREW5zeqxZ hKVBFXLMANzyTy0Q3gw+3IwRp4VYef7XVWLF91kJglMKMwUPcc35dV68MIRc1YVl sZLLsbxfZf9AZmRV2jrq8Fty3f3Svm8hi3EeRy+SB8ncq99fEdJCnJVbLWNX7vp2 A6FVp6co35HbqSHDgMFnP53iKYFmR+Sg/w4J3yqr4ctdAIOwsNnrpen5HunC/vS3 ci9EYo96RP/zRIuXeYEUGzccOyghYAVpc2tdjXQQ4N9DtZWBig1oY940C/aD+A8q d4njyj3mXDKjWSUHg6dUgMSvoMX8Yf4PDb//D7XnRsusFimoJQU0m7Sclrb1YQIs neKXH6xA8TOXv78q6KSsYNyCLvJxnjkqvPnfcYbXV3Ja3XA9/sENNgkzqfJIUYi/ 84DJQLm0xtlH1t/p1So/QIMk1O8AMZxcpzxooDPlPqivE6LuJLpTAZuwGsszChvk 4xWCbajxT4bUvcj4AkwsU9RMpNywY+yYF9K+L6vl+XIG0Y1nBiriFbjEhwSaNMe5 3tPScLkFVXB56O8RNfrqf7AJ8tklYrLpCgjsrfnl+QC+NkRxA5b3O6NEZltWKjHo yrEnCP6ExKlSQbaoozGg =LuIO -----END PGP SIGNATURE----- --Apple-Mail=_1783D049-BACA-47C2-A2BF-73289A959AB7-- From owner-freebsd-arch@FreeBSD.ORG Wed Nov 5 23:47:42 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0EA29319 for ; Wed, 5 Nov 2014 23:47:42 +0000 (UTC) Received: from eastrmfepi101.cox.net (eastrmfepi101.cox.net [68.230.241.197]) by mx1.freebsd.org (Postfix) with ESMTP id 942A7BBC for ; Wed, 5 Nov 2014 23:47:41 +0000 (UTC) Received: from eastrmimpo209 ([68.230.241.224]) by eastrmfepo103.cox.net (InterMail vM.8.01.05.15 201-2260-151-145-20131218) with ESMTP id <20141105222751.DUMR18577.eastrmfepo103.cox.net@eastrmimpo209> for ; Wed, 5 Nov 2014 17:27:51 -0500 Received: from [192.168.3.15] ([72.219.202.186]) by eastrmimpo209 with cox id ByTr1p00N41obj401yTr7d; Wed, 05 Nov 2014 17:27:51 -0500 X-CT-Class: Clean X-CT-Score: 0.00 X-CT-RefID: str=0001.0A020207.545AA467.01E2,ss=1,re=0.001,fgs=0 X-CT-Spam: 0 X-Authority-Analysis: v=2.0 cv=H/cFNZki c=1 sm=1 a=k40gPPfQ5QH6qv5U/EJc3Q==:17 a=f5xKl4ys9bwA:10 a=N659UExz7-8A:10 a=kviXuzpPAAAA:8 a=6I5d2MoRAAAA:8 a=UlqArFMXEW3J8R2ohJQA:9 a=pILNOxqGKmIA:10 a=8lT166Hsb7oA:10 a=mZUcFzKqlCgA:10 a=K-R649jcuWEA:10 a=k40gPPfQ5QH6qv5U/EJc3Q==:117 X-CM-Score: 0.00 Authentication-Results: cox.net; none Message-ID: <545AA4C5.1060709@cox.net> Date: Wed, 05 Nov 2014 17:29:25 -0500 From: "John D. Hendrickson" User-Agent: Thunderbird 2.0.0.24 (X11/20100228) MIME-Version: 1.0 To: Warner Losh Subject: Re: Why do we have @ in modules builds? References: In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 23:47:42 -0000 Warner Losh wrote: > Does anybody recall why we have @ symlink in our module builds? > I’m constantly working around issues that this creates. Maybe it is time > to eliminate it? > > So I’ve posted the following review: https://reviews.freebsd.org/D1100 > > Warner > could you please post what it is that "does not work" and what is is "you must do" to work around it ? From owner-freebsd-arch@FreeBSD.ORG Wed Nov 5 23:47:42 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C1DD531A for ; Wed, 5 Nov 2014 23:47:42 +0000 (UTC) Received: from eastrmfepi101.cox.net (eastrmfepi101.cox.net [68.230.241.197]) by mx1.freebsd.org (Postfix) with ESMTP id 54109BBD for ; Wed, 5 Nov 2014 23:47:42 +0000 (UTC) Received: from eastrmimpo210 ([68.230.241.225]) by eastrmfepo201.cox.net (InterMail vM.8.01.05.15 201-2260-151-145-20131218) with ESMTP id <20141105222650.DCGW20604.eastrmfepo201.cox.net@eastrmimpo210> for ; Wed, 5 Nov 2014 17:26:50 -0500 Received: from [192.168.3.15] ([72.219.202.186]) by eastrmimpo210 with cox id BySq1p00741obj401ySqBB; Wed, 05 Nov 2014 17:26:50 -0500 X-CT-Class: Clean X-CT-Score: 0.00 X-CT-RefID: str=0001.0A020201.545AA42A.01D3,ss=1,re=0.001,fgs=0 X-CT-Spam: 0 X-Authority-Analysis: v=2.0 cv=aZC/a2Ut c=1 sm=1 a=k40gPPfQ5QH6qv5U/EJc3Q==:17 a=f5xKl4ys9bwA:10 a=N659UExz7-8A:10 a=kviXuzpPAAAA:8 a=6I5d2MoRAAAA:8 a=UlqArFMXEW3J8R2ohJQA:9 a=pILNOxqGKmIA:10 a=k40gPPfQ5QH6qv5U/EJc3Q==:117 X-CM-Score: 0.00 Authentication-Results: cox.net; none Message-ID: <545AA487.7020007@cox.net> Date: Wed, 05 Nov 2014 17:28:23 -0500 From: "John D. Hendrickson" User-Agent: Thunderbird 2.0.0.24 (X11/20100228) MIME-Version: 1.0 To: Warner Losh Subject: Re: Why do we have @ in modules builds? References: In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 23:47:42 -0000 Warner Losh wrote: > Does anybody recall why we have @ symlink in our module builds? > I’m constantly working around issues that this creates. Maybe it is time > to eliminate it? > > So I’ve posted the following review: https://reviews.freebsd.org/D1100 > > Warner > don't know. ideas could it be to aid directory sorting thus build order ? or for tar(1) order? (base world is static order right?) could it be used by some script for old bang notation or email notification ? ie, uucp ? From owner-freebsd-arch@FreeBSD.ORG Thu Nov 6 00:25:08 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E2188FF8; Thu, 6 Nov 2014 00:25:07 +0000 (UTC) Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2on0131.outbound.protection.outlook.com [65.55.169.131]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C3831FA2; Thu, 6 Nov 2014 00:25:06 +0000 (UTC) Received: from CO2PR05CA026.namprd05.prod.outlook.com (10.141.241.154) by DM2PR05MB447.namprd05.prod.outlook.com (10.141.104.150) with Microsoft SMTP Server (TLS) id 15.1.6.9; Thu, 6 Nov 2014 00:24:58 +0000 Received: from BL2FFO11FD023.protection.gbl (2a01:111:f400:7c09::121) by CO2PR05CA026.outlook.office365.com (2a01:111:e400:1429::26) with Microsoft SMTP Server (TLS) id 15.1.16.15 via Frontend Transport; Thu, 6 Nov 2014 00:24:57 +0000 Received: from P-EMF01-SAC.jnpr.net (66.129.239.15) by BL2FFO11FD023.mail.protection.outlook.com (10.173.161.102) with Microsoft SMTP Server (TLS) id 15.1.6.13 via Frontend Transport; Thu, 6 Nov 2014 00:24:57 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF01-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Wed, 5 Nov 2014 16:24:55 -0800 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id sA60OtR25330; Wed, 5 Nov 2014 16:24:55 -0800 (PST) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id 03ABC580A3; Wed, 5 Nov 2014 16:24:55 -0800 (PST) To: Baptiste Daroussin Subject: Re: Overlinking in base In-Reply-To: <20141105134006.GL10388@ivaldir.etoilebsd.net> References: <20141105113839.GG10388@ivaldir.etoilebsd.net> <20141105125431.GD53947@kib.kiev.ua> <20141105125931.GJ10388@ivaldir.etoilebsd.net> <20141105133029.GH53947@kib.kiev.ua> <20141105134006.GL10388@ivaldir.etoilebsd.net> Comments: In-reply-to: Baptiste Daroussin message dated "Wed, 05 Nov 2014 14:40:07 +0100." From: "Simon J. Gerraty" X-Mailer: MH-E 8.0.3; nmh 1.3; GNU Emacs 22.3.1 Date: Wed, 5 Nov 2014 16:24:54 -0800 Message-ID: <3912.1415233494@chaos> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.15; CTRY:US; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(51704005)(199003)(24454002)(189002)(93916002)(120916001)(106466001)(50986999)(92566001)(19580395003)(88136002)(95666004)(102836001)(57986006)(89996001)(50466002)(93886004)(104166001)(97736003)(84676001)(81156004)(47776003)(50226001)(99396003)(105596002)(21056001)(86362001)(107046002)(64706001)(87936001)(76176999)(44976005)(46102003)(76506005)(33716001)(92726001)(69596002)(4396001)(48376002)(20776003)(6806004)(110136001)(68736004)(77156002)(87286001)(19580405001)(62966003)(117636001)(31966008)(42262002)(62816006); DIR:OUT; SFP:1102; SCL:1; SRVR:DM2PR05MB447; H:P-EMF01-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; A:1; MX:1; LANG:en; X-Microsoft-Antispam: UriScan:; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:;SRVR:DM2PR05MB447; X-Exchange-Antispam-Report-Test: UriScan:; X-Forefront-PRVS: 0387D64A71 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.15 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.15) smtp.mailfrom=sjg@juniper.net; X-OriginatorOrg: juniper.net Cc: Konstantin Belousov , arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Nov 2014 00:25:08 -0000 Baptiste Daroussin wrote: > In the second case we could do it via make(1) > LIBADD= liba libc libc > this will open something like a ${PATHTOTHELIB}/link.mk which will define > DYNAMIC_ADD > STATIC_ADD > > And this could be recursive. We do something like that in the Junos build prog makefile might have DPLIBS+= ${LIBFOO} which is exactly equivalent to LDADD+= -lfoo DPADD+= ${LIBFOO} but ensures that they stay in sync (not so important now with meta mode). bsd.libnames.mk can then have DPLIBS_libfoo += ${LIBGOO} DPLIBS_libgoo += ${LIBZOO} All of which is processed by dpadd.mk which you can find in contrib/bmake/mk Though dpadd.mk ignores DPLIBS_libgoo += ${LIBZOO} if LIBZOO has already been added. From owner-freebsd-arch@FreeBSD.ORG Thu Nov 6 13:06:01 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C4162216 for ; Thu, 6 Nov 2014 13:06:01 +0000 (UTC) Received: from mail-wi0-x235.google.com (mail-wi0-x235.google.com [IPv6:2a00:1450:400c:c05::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D1D4C0D for ; Thu, 6 Nov 2014 13:06:01 +0000 (UTC) Received: by mail-wi0-f181.google.com with SMTP id n3so1420422wiv.8 for ; Thu, 06 Nov 2014 05:05:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=8sH3oOOfQOt7wfV/PR6CD8kg0G1NOghTxqog4FvTzKc=; b=lOnvwAPmH3CTjzIDao8g2QQ+msvHcnLYu9O+rViEMJx5fcmK8HW+KsNKbRghDROrpl JNnZwu6ZYhm/Dw/Ck+aN2DYDTagLKquC5ymSbKfqkaNagj9QEOmvGaAAky3FV8QrOESS y0nYwWA5Hz+rhBiUz7XG4VC30ZuD0NBlSNwU3HXeFx8zjhGEyNIFNtblc6qvCBdpQP6N SyV2tRdUiJtIxwzARkkHhGYAkQysBU/bN6bH8eiZ/WK8wzHMYqGhKNW3rhu/4dyJbGYx eNb/cu88Ql1ZFtWE7ieiEX9yGU8iPQHz/XBQzIr8XwaIVdqjd9HnDDD5JQDzhsQeFeW1 A/Sw== X-Received: by 10.194.240.68 with SMTP id vy4mr5625425wjc.36.1415279159641; Thu, 06 Nov 2014 05:05:59 -0800 (PST) Received: from ivaldir.etoilebsd.net ([2001:41d0:8:db4c::1]) by mx.google.com with ESMTPSA id mc4sm8247663wic.6.2014.11.06.05.05.58 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Nov 2014 05:05:58 -0800 (PST) Sender: Baptiste Daroussin Date: Thu, 6 Nov 2014 14:05:55 +0100 From: Baptiste Daroussin To: "Simon J. Gerraty" Subject: Re: Overlinking in base Message-ID: <20141106130555.GP10388@ivaldir.etoilebsd.net> References: <20141105113839.GG10388@ivaldir.etoilebsd.net> <20141105125431.GD53947@kib.kiev.ua> <20141105125931.GJ10388@ivaldir.etoilebsd.net> <20141105133029.GH53947@kib.kiev.ua> <20141105134006.GL10388@ivaldir.etoilebsd.net> <3912.1415233494@chaos> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ZgGN478A9hzzvyZc" Content-Disposition: inline In-Reply-To: <3912.1415233494@chaos> User-Agent: Mutt/1.5.23 (2014-03-12) Cc: Konstantin Belousov , arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Nov 2014 13:06:02 -0000 --ZgGN478A9hzzvyZc Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Nov 05, 2014 at 04:24:54PM -0800, Simon J. Gerraty wrote: > Baptiste Daroussin wrote: > > In the second case we could do it via make(1) > > LIBADD=3D liba libc libc > > this will open something like a ${PATHTOTHELIB}/link.mk which will defi= ne > > DYNAMIC_ADD > > STATIC_ADD > >=20 > > And this could be recursive. >=20 > We do something like that in the Junos build >=20 > prog makefile might have DPLIBS+=3D ${LIBFOO} > which is exactly equivalent to >=20 > LDADD+=3D -lfoo > DPADD+=3D ${LIBFOO} >=20 > but ensures that they stay in sync (not so important now with meta > mode). >=20 > bsd.libnames.mk can then have >=20 > DPLIBS_libfoo +=3D ${LIBGOO} > DPLIBS_libgoo +=3D ${LIBZOO} >=20 > All of which is processed by dpadd.mk which you can find in > contrib/bmake/mk > Though dpadd.mk ignores DPLIBS_libgoo +=3D ${LIBZOO} if LIBZOO has already > been added. >=20 I'am about to add something based on the following principle: https://people.freebsd.org/~bapt/plop.diff With a bit more changes The version I have now (a bit different from the patch now :)) allows multi= ple things: 1/ simplify the Makefile for users: LIBADD=3D m archive util instead of DPADD=3D ${LIBM} ${LIBARCHIVE} ${LIBUTIL} LDADD=3D -lm -larchive -lutil 2/ ensure dependencies are automatically tracked For example -lucl needs -lm adding LIBADD=3D ucl does the magic by itself 3/ allow to build any single binary statically so far I'm able to build everything is bin sbin usr.bin and usr.sbin statically (which wasn't doable before) 3/ hides the private/internal lib from the final user Do more need to say USEPRIVATELIB because I do use a libunbound it is autom= atic regards, Bapt --ZgGN478A9hzzvyZc Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlRbcjMACgkQ8kTtMUmk6EzVjQCeP5y4K5uJZH0WKPqEFAJweEji dH8AnRCe52hvTFnhSuOCR18qbi8HKx4y =IUji -----END PGP SIGNATURE----- --ZgGN478A9hzzvyZc-- From owner-freebsd-arch@FreeBSD.ORG Thu Nov 6 14:35:38 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A41616BF for ; Thu, 6 Nov 2014 14:35:38 +0000 (UTC) Received: from mail-pd0-f172.google.com (mail-pd0-f172.google.com [209.85.192.172]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 70A087EC for ; Thu, 6 Nov 2014 14:35:38 +0000 (UTC) Received: by mail-pd0-f172.google.com with SMTP id r10so1261548pdi.3 for ; Thu, 06 Nov 2014 06:35:32 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:content-type:mime-version:subject:from :in-reply-to:date:cc:message-id:references:to; bh=Q5E8kSv4kMKb8R1aR5SprZ2zBC9JKUxAhbHdVdQoCOo=; b=CGapkxdHRa1Kww06zk/pK3r+hEqBGzqURMC7gwlnY6Z3z4mYcxRk050Im0zbG22DpC 8Gh+abkKynfeHFswo458zfIChcLIodyohC2nS5l7rzHbbo1pTagyLlqyxP84BZNr6nPS SXqMYipH9FnHdCoVE0AJiZstkIS/D68aValF/rULNzAvfrTz8yU4KjknModpTLMb7rnb GlFHHmnxbSDJ5/YgpfyIPGUzoGIz2F8wxxxboyKa63MwMl5sBDe9nyqQzYMZlRVZ3rmX uNzmLbLpoi/v3wMoCtZM/2mlA2a8yzVAgKpe0eMtHXgwbTMuIMRqx3AMPz2PnX8K/xDr 2Fyg== X-Gm-Message-State: ALoCoQnRKBI4DWFS87pzlnzmT42VZrIisvtuD2ksuLfPi6iv6pCQ6WPMd4mV58gewrgH+U/A0Jbf X-Received: by 10.66.141.165 with SMTP id rp5mr4669197pab.121.1415284532053; Thu, 06 Nov 2014 06:35:32 -0800 (PST) Received: from [10.64.26.112] ([69.53.236.236]) by mx.google.com with ESMTPSA id i10sm6164846pdr.21.2014.11.06.06.35.30 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 06 Nov 2014 06:35:31 -0800 (PST) Sender: Warner Losh Content-Type: multipart/signed; boundary="Apple-Mail=_212DDC0F-7FF5-4146-A5C7-DC090AE62788"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Overlinking in base From: Warner Losh In-Reply-To: <20141106130555.GP10388@ivaldir.etoilebsd.net> Date: Thu, 6 Nov 2014 07:35:16 -0700 Message-Id: <0A3B2AAE-91A7-4F79-BC3C-2463E3AF6C68@bsdimp.com> References: <20141105113839.GG10388@ivaldir.etoilebsd.net> <20141105125431.GD53947@kib.kiev.ua> <20141105125931.GJ10388@ivaldir.etoilebsd.net> <20141105133029.GH53947@kib.kiev.ua> <20141105134006.GL10388@ivaldir.etoilebsd.net> <3912.1415233494@chaos> <20141106130555.GP10388@ivaldir.etoilebsd.net> To: Baptiste Daroussin X-Mailer: Apple Mail (2.1878.6) Cc: Konstantin Belousov , arch@freebsd.org, "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Nov 2014 14:35:38 -0000 --Apple-Mail=_212DDC0F-7FF5-4146-A5C7-DC090AE62788 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On Nov 6, 2014, at 6:05 AM, Baptiste Daroussin wrote: > On Wed, Nov 05, 2014 at 04:24:54PM -0800, Simon J. Gerraty wrote: >> Baptiste Daroussin wrote: >>> In the second case we could do it via make(1) >>> LIBADD=3D liba libc libc >>> this will open something like a ${PATHTOTHELIB}/link.mk which will = define >>> DYNAMIC_ADD >>> STATIC_ADD >>>=20 >>> And this could be recursive. >>=20 >> We do something like that in the Junos build >>=20 >> prog makefile might have DPLIBS+=3D ${LIBFOO} >> which is exactly equivalent to >>=20 >> LDADD+=3D -lfoo >> DPADD+=3D ${LIBFOO} >>=20 >> but ensures that they stay in sync (not so important now with meta >> mode). >>=20 >> bsd.libnames.mk can then have >>=20 >> DPLIBS_libfoo +=3D ${LIBGOO} >> DPLIBS_libgoo +=3D ${LIBZOO} >>=20 >> All of which is processed by dpadd.mk which you can find in >> contrib/bmake/mk >> Though dpadd.mk ignores DPLIBS_libgoo +=3D ${LIBZOO} if LIBZOO has = already >> been added. >>=20 > I'am about to add something based on the following principle: > https://people.freebsd.org/~bapt/plop.diff >=20 > With a bit more changes >=20 > The version I have now (a bit different from the patch now :)) allows = multiple > things: >=20 > 1/ simplify the Makefile for users: > LIBADD=3D m archive util > instead of > DPADD=3D ${LIBM} ${LIBARCHIVE} ${LIBUTIL} > LDADD=3D -lm -larchive -lutil >=20 > 2/ ensure dependencies are automatically tracked > For example -lucl needs -lm adding LIBADD=3D ucl does the magic by = itself >=20 > 3/ allow to build any single binary statically so far I'm able to = build > everything is bin sbin usr.bin and usr.sbin statically (which wasn't = doable > before) >=20 > 3/ hides the private/internal lib from the final user > Do more need to say USEPRIVATELIB because I do use a libunbound it is = automatic Generally I like this. One issue though about dependencies: Is there a tool to manage them? How = do we know if things are wrong? Will a simple buildworld always detect that, or do we need to do a = buildworld with static linking? Warner --Apple-Mail=_212DDC0F-7FF5-4146-A5C7-DC090AE62788 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJUW4ckAAoJEGwc0Sh9sBEA2ngP/Rq8V09YhmYXAsUkQxal+3k1 nb3Ci3WELWatZtfJNMbQkheQGylmiJBf1PDBJkR8Cqmr7rX3/3nPrNpf/YQAX8E/ 34BmHCJ6F1QImrdZnEbjdgh8mDhwdSPsMlOuGepAqeeXZMU/ws3H38FcjfdAXu23 er2AAAHcATEFuO6fZKafug5ILae2mWQ75NvE1bOY9fUNm3ESJ7mT9bEF15//1SGl G8df1ae/Y2NSTZIpEY6PxkA0iMilP8b8L5gX7eqlSnuraCv0/HSUuJLqIc5r1Yfb dIyUtoADrzu42PeE9jTR7qwO00nLL8+hfA2mlGZaV4T/0d+errxjWaql6q6FS9J8 XaJRdIZfT0w5pG665LzTL3WEyNjv3ro8kx9EN7uqbTlz+Z9uPjGfTCuWs76d3u8R oujxo5wuTID+JltqRzLB/gGQ2HpU/FKjKXOnjC9dIJrzuUqNuUOhE7kdS/2Jyoi3 vUmjnQ9IVACdxx/lJqaTqt4GyiDgu7dtIpnFdpyZeSI7e7wmP3TSrZuRbrQ4mcAi QeOwy8n28qWqxB/bldxM8DJXEdKUn5xNzGXRV0ogK6OBAvIV4kGKMU6GU15w4ofh 5IBB+AUOG0l3GRr+YznNwHf+1MrX5pwCp19XAA0nvCJGPqmZhTPkIMqx5+iqFVJX X+2+ebG9IP30c9iomYdV =ZAJ8 -----END PGP SIGNATURE----- --Apple-Mail=_212DDC0F-7FF5-4146-A5C7-DC090AE62788-- From owner-freebsd-arch@FreeBSD.ORG Thu Nov 6 15:33:48 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6ED5C725; Thu, 6 Nov 2014 15:33:48 +0000 (UTC) Received: from spindle.one-eyed-alien.net (spindle.one-eyed-alien.net [199.48.129.229]) by mx1.freebsd.org (Postfix) with ESMTP id 46E24E11; Thu, 6 Nov 2014 15:33:47 +0000 (UTC) Received: by spindle.one-eyed-alien.net (Postfix, from userid 3001) id EBFFF5A9F0B; Thu, 6 Nov 2014 15:33:40 +0000 (UTC) Date: Thu, 6 Nov 2014 15:33:40 +0000 From: Brooks Davis To: Baptiste Daroussin Subject: Re: Overlinking in base Message-ID: <20141106153340.GB76675@spindle.one-eyed-alien.net> References: <20141105113839.GG10388@ivaldir.etoilebsd.net> <20141105125431.GD53947@kib.kiev.ua> <20141105125931.GJ10388@ivaldir.etoilebsd.net> <20141105133029.GH53947@kib.kiev.ua> <20141105134006.GL10388@ivaldir.etoilebsd.net> <3912.1415233494@chaos> <20141106130555.GP10388@ivaldir.etoilebsd.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ZPt4rx8FFjLCG7dd" Content-Disposition: inline In-Reply-To: <20141106130555.GP10388@ivaldir.etoilebsd.net> User-Agent: Mutt/1.5.23 (2014-03-12) Cc: Konstantin Belousov , arch@freebsd.org, "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Nov 2014 15:33:48 -0000 --ZPt4rx8FFjLCG7dd Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Nov 06, 2014 at 02:05:55PM +0100, Baptiste Daroussin wrote: > On Wed, Nov 05, 2014 at 04:24:54PM -0800, Simon J. Gerraty wrote: > > Baptiste Daroussin wrote: > > > In the second case we could do it via make(1) > > > LIBADD=3D liba libc libc > > > this will open something like a ${PATHTOTHELIB}/link.mk which will de= fine > > > DYNAMIC_ADD > > > STATIC_ADD > > >=20 > > > And this could be recursive. > >=20 > > We do something like that in the Junos build > >=20 > > prog makefile might have DPLIBS+=3D ${LIBFOO} > > which is exactly equivalent to > >=20 > > LDADD+=3D -lfoo > > DPADD+=3D ${LIBFOO} > >=20 > > but ensures that they stay in sync (not so important now with meta > > mode). > >=20 > > bsd.libnames.mk can then have > >=20 > > DPLIBS_libfoo +=3D ${LIBGOO} > > DPLIBS_libgoo +=3D ${LIBZOO} > >=20 > > All of which is processed by dpadd.mk which you can find in > > contrib/bmake/mk > > Though dpadd.mk ignores DPLIBS_libgoo +=3D ${LIBZOO} if LIBZOO has alre= ady > > been added. > >=20 > I'am about to add something based on the following principle: > https://people.freebsd.org/~bapt/plop.diff >=20 > With a bit more changes >=20 > The version I have now (a bit different from the patch now :)) allows mul= tiple > things: >=20 > 1/ simplify the Makefile for users: > LIBADD=3D m archive util > instead of > DPADD=3D ${LIBM} ${LIBARCHIVE} ${LIBUTIL} > LDADD=3D -lm -larchive -lutil For the list, the reason I proposed LIBADD=3Dfoo initially vs the DPLIBS model is that I need the library name is a format I can use for other things such as linking programs to a combination of an archive containing native code and a collection of llvm bitcode. In the tesla branch in perforce I'm using this to perform whole program analysis. I use this snipit to add entries to both llvm-link command line and the final ld comamnd. +.if ${MK_LLVM_INSTRUMENTED} !=3D "no" +LLVM_LINK_ADD+=3D ${LIBADD:@L@${LLVM_IR_FILE_${L:tu}:U"No LLVM_IR_F= ILE_${L:tu} variable defined"}@} +LLVM_LDADD+=3D ${LIBADD:@L@${LLVM_IR_FILE_${L:tu}:U"No LLVM_NATIVE_FILE_= ${L:tu} variable defined"}@} +.endif Thinking about this in the context of overlinking. There's an argument we should use LIBADD to produce an _LIBADD which is a pure list of actual libraries to be linked so uses like this can use it. That would also provide a place to do sorting hacks if required. -- Brooks --ZPt4rx8FFjLCG7dd Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlRblNQACgkQXY6L6fI4GtRBTACgyw/ih1IJNtIPj+WrbkvUfx9J PAsAoMGipViAAobjES31kZUxZaAuzoks =MlnA -----END PGP SIGNATURE----- --ZPt4rx8FFjLCG7dd-- From owner-freebsd-arch@FreeBSD.ORG Fri Nov 7 05:52:07 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 077C38A8; Fri, 7 Nov 2014 05:52:07 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1on0110.outbound.protection.outlook.com [157.56.110.110]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E257FC39; Fri, 7 Nov 2014 05:52:05 +0000 (UTC) Received: from BL2PR05CA0038.namprd05.prod.outlook.com (10.255.226.38) by BN1PR05MB437.namprd05.prod.outlook.com (10.141.58.11) with Microsoft SMTP Server (TLS) id 15.1.16.15; Fri, 7 Nov 2014 05:52:04 +0000 Received: from BN1AFFO11FD051.protection.gbl (2a01:111:f400:7c10::188) by BL2PR05CA0038.outlook.office365.com (2a01:111:e400:c04::38) with Microsoft SMTP Server (TLS) id 15.1.16.15 via Frontend Transport; Fri, 7 Nov 2014 05:52:03 +0000 Received: from P-EMF01-SAC.jnpr.net (66.129.239.15) by BN1AFFO11FD051.mail.protection.outlook.com (10.58.53.66) with Microsoft SMTP Server (TLS) id 15.1.6.13 via Frontend Transport; Fri, 7 Nov 2014 05:52:03 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF01-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Thu, 6 Nov 2014 21:52:02 -0800 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id sA75q0R04875; Thu, 6 Nov 2014 21:52:01 -0800 (PST) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id 9BE0B580A3; Thu, 6 Nov 2014 21:52:00 -0800 (PST) To: Baptiste Daroussin Subject: Re: Overlinking in base In-Reply-To: <20141106130555.GP10388@ivaldir.etoilebsd.net> References: <20141105113839.GG10388@ivaldir.etoilebsd.net> <20141105125431.GD53947@kib.kiev.ua> <20141105125931.GJ10388@ivaldir.etoilebsd.net> <20141105133029.GH53947@kib.kiev.ua> <20141105134006.GL10388@ivaldir.etoilebsd.net> <3912.1415233494@chaos> <20141106130555.GP10388@ivaldir.etoilebsd.net> Comments: In-reply-to: Baptiste Daroussin message dated "Thu, 06 Nov 2014 14:05:55 +0100." From: "Simon J. Gerraty" X-Mailer: MH-E 8.0.3; nmh 1.3; GNU Emacs 22.3.1 Date: Thu, 6 Nov 2014 21:52:00 -0800 Message-ID: <20938.1415339520@chaos> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.15; CTRY:US; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(189002)(24454002)(199003)(68736004)(19580405001)(19580395003)(6806004)(76506005)(99396003)(57986006)(50986999)(86362001)(76176999)(120916001)(44976005)(69596002)(84676001)(87286001)(87936001)(88136002)(89996001)(117636001)(93886004)(97736003)(46102003)(104166001)(95666004)(92726001)(92566001)(62966003)(77156002)(4396001)(107046002)(31966008)(21056001)(50226001)(105596002)(106466001)(15975445006)(81156004)(64706001)(48376002)(50466002)(33716001)(110136001)(102836001)(47776003)(20776003)(42262002)(62816006); DIR:OUT; SFP:1102; SCL:1; SRVR:BN1PR05MB437; H:P-EMF01-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; A:1; LANG:en; X-Microsoft-Antispam: UriScan:; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:;SRVR:BN1PR05MB437; X-Exchange-Antispam-Report-Test: UriScan:; X-Forefront-PRVS: 03883BD916 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.15 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.15) smtp.mailfrom=sjg@juniper.net; X-OriginatorOrg: juniper.net Cc: Konstantin Belousov , arch@freebsd.org, sjg@juniper.net X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Nov 2014 05:52:07 -0000 Baptiste Daroussin wrote: > I'am about to add something based on the following principle: > https://people.freebsd.org/~bapt/plop.diff This does much of what dpadd.mk does but with more overhead. Once you have a variable like DPADD that has all the libs you need in the right order, you can easily derrive the needed -l's (and even -L's if the libs are in non-standard locations). From owner-freebsd-arch@FreeBSD.ORG Fri Nov 7 15:39:19 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 82288E42 for ; Fri, 7 Nov 2014 15:39:19 +0000 (UTC) Received: from hydra.pix.net (hydra.pix.net [IPv6:2001:470:e254::4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5D22A650 for ; Fri, 7 Nov 2014 15:39:19 +0000 (UTC) Received: from torb.pix.net (torb.pix.net [IPv6:2001:470:e254:10:12dd:b1ff:febf:eca9]) (authenticated bits=0) by hydra.pix.net (8.14.9/8.14.9) with ESMTP id sA7FdH2O022034; Fri, 7 Nov 2014 10:39:17 -0500 (EST) (envelope-from lidl@pix.net) Message-ID: <545CE7A4.8060804@pix.net> Date: Fri, 07 Nov 2014 10:39:16 -0500 From: Kurt Lidl User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: freebsd-arch@freebsd.org Subject: Re: Overlinking in base References: <3912.1415233494@chaos> In-Reply-To: <3912.1415233494@chaos> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Nov 2014 15:39:19 -0000 Simon J Gerrart wrote: > Baptiste Daroussin wrote: >> In the second case we could do it via make(1) >> LIBADD= liba libc libc >> this will open something like a ${PATHTOTHELIB}/link.mk which will define >> DYNAMIC_ADD >> STATIC_ADD >> >> And this could be recursive. > > We do something like that in the Junos build > > prog makefile might have DPLIBS+= ${LIBFOO} > which is exactly equivalent to > > LDADD+= -lfoo > DPADD+= ${LIBFOO} > > but ensures that they stay in sync (not so important now with meta > mode). > > bsd.libnames.mk can then have > > DPLIBS_libfoo += ${LIBGOO} > DPLIBS_libgoo += ${LIBZOO} > > All of which is processed by dpadd.mk which you can find in > contrib/bmake/mk > Though dpadd.mk ignores DPLIBS_libgoo += ${LIBZOO} if LIBZOO has already > been added. Wow, I wish I had know that this existed. I ended up writing an simpler version of this for some work last year. This was mostly so we didn't have to track the same library twice, for both LDADD and DPADD. I'm totally in favor of something that exposes this to wider usage, like having it in /usr/share/mk rather than hidden away in the source tree only. -Kurt