From owner-soc-status@freebsd.org Wed Aug 21 20:56:18 2019 Return-Path: Delivered-To: soc-status@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3B9B6D3F38; Wed, 21 Aug 2019 20:56:18 +0000 (UTC) (envelope-from therontarigo@gmail.com) Received: from mail-pg1-f174.google.com (mail-pg1-f174.google.com [209.85.215.174]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46DKhK19bSz4XVM; Wed, 21 Aug 2019 20:56:16 +0000 (UTC) (envelope-from therontarigo@gmail.com) Received: by mail-pg1-f174.google.com with SMTP id n4so2053192pgv.2; Wed, 21 Aug 2019 13:56:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:subject:to:cc:message-id:date:user-agent :mime-version:content-transfer-encoding:content-language; bh=1OfV25YDlo/eZgzCzyB35o7pxcP0Xi+wRIMycIxpi8U=; b=nKrMsolIq6dypMlnaCpc9zg22zGsCgH4zN5MmIo4XGS63XClCbyoljW//jBAgzWI7+ BD96lTtmEp50fU3duIA2zZEPEUEj+6COyThdNRGDIZIBHIMTS6dtnhBxLhOZUDjYuRif F5aDfr+ZPoQsEEcmFVJTycE+WKFiO+IEWipX+M0poU71zEo3c8e4OeRZ5N/ZJRHqaEoF +oEs62d3vje42UEGrA942TbMa3WKm5rjk4pumNRgysZOyRwnD1LF+x3rLsk+BDveHZSX jnaLAWaNvSvOQHkkzglTsDZdRffICcHKp8BmRfxgMdRaP7d/ZLXVs7MpUcyRcpSJYgDz AqUg== X-Gm-Message-State: APjAAAXVDH9ndVzf5404G3PhVHwrSD8p9+zmFEyjlE7WUY5cwjFH7xhh 9r6uSZ6LMCOhK1dIeQHcPUH1Jv4b8WI= X-Google-Smtp-Source: APXvYqzeruajs3Smf/ayLNaIXiodBRrfX3qhJgVCx7e1x4Rix2bjhAr5UcUa9/RNnq2EqDlOiTYS7A== X-Received: by 2002:a17:90a:9cf:: with SMTP id 73mr1908612pjo.90.1566420975437; Wed, 21 Aug 2019 13:56:15 -0700 (PDT) Received: from [192.168.1.21] (c-73-170-47-221.hsd1.ca.comcast.net. [73.170.47.221]) by smtp.gmail.com with ESMTPSA id s67sm679345pjb.8.2019.08.21.13.56.14 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 21 Aug 2019 13:56:15 -0700 (PDT) From: Theron Tarigo Subject: Building ports without needing to installing dependencies into /usr/local To: freebsd-ports@freebsd.org Cc: soc-status@freebsd.org, Bakul Shah Message-ID: Date: Wed, 21 Aug 2019 13:56:14 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 46DKhK19bSz4XVM X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of therontarigo@gmail.com designates 209.85.215.174 as permitted sender) smtp.mailfrom=therontarigo@gmail.com X-Spamd-Result: default: False [-6.12 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[freebsd.org]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.988,0]; RCVD_IN_DNSWL_NONE(0.00)[174.215.85.209.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; IP_SCORE(-3.13)[ip: (-9.89), ipnet: 209.85.128.0/17(-3.36), asn: 15169(-2.35), country: US(-0.05)]; FORGED_SENDER(0.30)[theron@freebsd.org,therontarigo@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[221.47.170.73.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[theron@freebsd.org,therontarigo@gmail.com]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: soc-status@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Summer of Code Status Reports and Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Aug 2019 20:56:18 -0000 Hello all, I'd like to share with you the following summary of my progress in adapting the FreeBSD ports framework to gain the capability to build packages (including dependencies) in isolation of the local system configuration in /usr/local, whereas the existing behavior is to require dependencies installed there.  This work is the result of my participation in FreeBSD's Google Summer of Code. Due to the widespread assumption by ports that dependencies are installed at build time into their final deployed locations, this has been a non-trivial undertaking.  Whereas Poudriere remedies this by running the ports framework entirely within Jails, this project attempts to patch all build-time scripts and tooling to access required files from an environment-controlled location.  In most cases this is accomplished by a userspace library to catch and rewrite file paths in POSIX API calls, which has also been developed as part of this project: https://github.com/therontarigo/freebsd-user-namespace/ The project was motivated by the observation that the ports framework as-is (without external tools) fundamentally lacks the capability of completing all build work before installation is performed, which is found in nearly all single-project build systems including FreeBSD base.  This is different from tools such as Poudriere or Synth as it is designed as a core dependency-handling mechanism of the ports build system rather than as an external management or automation tool. The need to touch some core parts of the FreeBSD ports framework means that readiness of the project for adoption may be a long time from now.  That said, I feel confident in declaring it a successful experiment, after testing a limited number of ports under the new scheme (in which ${LOCALBASE} i.e. /usr/local is never touched): Of 5638 ports known (1979 ports selected randomly from ports tree, and their dependencies), 75% were built successfully, since 23% were skipped due to failed dependencies. Of 4230 ports that could be tested (all dependencies were satisfied), 97% succeeded. I would have liked to have tested the entire ports tree, but haven't had access to sufficient machine-time. I've discussed these success rates with my mentor, Bakul Shah, and we agreed that this demonstrates the usefulness of the method. Of course the project is not ready for adoption as the default way of building ports until 100% success here is achieved, but merging of the code on an earlier schedule should be feasible since it is implemented as an option which can remain turned off by default without changing existing behaviors. From a user's perspective, the project currently provides a lighter-weight alternative to one of the core features of Poudriere: A port and all its dependencies are buildable in isolation (like Jail) to create one or more packages to be later installed on one or more systems. It is light-weight because: - Creation of virtual environments for dependencies is done purely in userspace, so Jail and Nullfs are not used. - The feature is usable through the standard Make targets; no top-down separate script or program is used to start the builds. - No preparation other than checking out the (modified, to be eventually merged) ports tree is required. - Direct changes to existing files in the ports framework are kept to a minimum. Other points of potential interest: - Superuser is not required at any part of the process (other than the installation of the resulting packages). It should not be misunderstood as trying to be a potential replacement for Poudriere: It does not perform jailed testing, incremental rebuilds, utilization of ZFS, or many other advanced features, nor are such features planned, nor even appropriate for direct inclusion into /usr/ports/Mk. The scheme currently works only on amd64, due to a small piece of the userspace virtual environment implementation involving machine code manipulation, but this can be extended to support other architectures. For those interested in helping to test, the work is available on Github as follows.  The ports are synchronized to quarterly 2019Q3 as of Aug 15. An example of how it can be tested (no need for superuser): (note that /usr/lib/debug/libexec/ld-elf.so.1.debug (from base-dbg or from installworld) must exist) $ git clone https://github.com/therontarigo/freebsd-ports -b separated --depth 1 $ cd freebsd-ports $ make PORTS_SEPARATED_BUILD=1 PORTSDIR=$PWD PORTBLDBASE=$HOME/ports -C devel/gmake config-recursive package-recursive To be extra sure it is not relying on /usr/local: try instead with LOCALBASE=/usr/nlocal (of course then packages won't install to the default local prefix). Resulting packages will be in $HOME/ports/packages/All/ This modified ports tree is intended to behave exactly as the official one when PORTS_SEPARATED_BUILD=1 is NOT used.  Anywhere that this is violated is a bug that I must fix. Much cleanup and far more thorough testing beyond what I have accomplished in the time of this project are needed before this should be used in production or considered for merging into official ports tree. Your feedback would be much appreciated and will help me to prepare a report on the project's successes, shortcomings, and future directions with respect to the community's needs. Theron