Date: Wed, 21 Aug 2019 13:56:14 -0700 From: Theron Tarigo <theron@freebsd.org> To: freebsd-ports@freebsd.org Cc: soc-status@freebsd.org, Bakul Shah <bakul@bitblocks.com> Subject: Building ports without needing to installing dependencies into /usr/local Message-ID: <bd94f2f2-b074-0ea6-fa8d-27c1f202c422@freebsd.org>
next in thread | raw e-mail | index | archive | help
Hello all, I'd like to share with you the following summary of my progress in adapting the FreeBSD ports framework to gain the capability to build packages (including dependencies) in isolation of the local system configuration in /usr/local, whereas the existing behavior is to require dependencies installed there. This work is the result of my participation in FreeBSD's Google Summer of Code. Due to the widespread assumption by ports that dependencies are installed at build time into their final deployed locations, this has been a non-trivial undertaking. Whereas Poudriere remedies this by running the ports framework entirely within Jails, this project attempts to patch all build-time scripts and tooling to access required files from an environment-controlled location. In most cases this is accomplished by a userspace library to catch and rewrite file paths in POSIX API calls, which has also been developed as part of this project: https://github.com/therontarigo/freebsd-user-namespace/ The project was motivated by the observation that the ports framework as-is (without external tools) fundamentally lacks the capability of completing all build work before installation is performed, which is found in nearly all single-project build systems including FreeBSD base. This is different from tools such as Poudriere or Synth as it is designed as a core dependency-handling mechanism of the ports build system rather than as an external management or automation tool. The need to touch some core parts of the FreeBSD ports framework means that readiness of the project for adoption may be a long time from now. That said, I feel confident in declaring it a successful experiment, after testing a limited number of ports under the new scheme (in which ${LOCALBASE} i.e. /usr/local is never touched): Of 5638 ports known (1979 ports selected randomly from ports tree, and their dependencies), 75% were built successfully, since 23% were skipped due to failed dependencies. Of 4230 ports that could be tested (all dependencies were satisfied), 97% succeeded. I would have liked to have tested the entire ports tree, but haven't had access to sufficient machine-time. I've discussed these success rates with my mentor, Bakul Shah, and we agreed that this demonstrates the usefulness of the method. Of course the project is not ready for adoption as the default way of building ports until 100% success here is achieved, but merging of the code on an earlier schedule should be feasible since it is implemented as an option which can remain turned off by default without changing existing behaviors. From a user's perspective, the project currently provides a lighter-weight alternative to one of the core features of Poudriere: A port and all its dependencies are buildable in isolation (like Jail) to create one or more packages to be later installed on one or more systems. It is light-weight because: - Creation of virtual environments for dependencies is done purely in userspace, so Jail and Nullfs are not used. - The feature is usable through the standard Make targets; no top-down separate script or program is used to start the builds. - No preparation other than checking out the (modified, to be eventually merged) ports tree is required. - Direct changes to existing files in the ports framework are kept to a minimum. Other points of potential interest: - Superuser is not required at any part of the process (other than the installation of the resulting packages). It should not be misunderstood as trying to be a potential replacement for Poudriere: It does not perform jailed testing, incremental rebuilds, utilization of ZFS, or many other advanced features, nor are such features planned, nor even appropriate for direct inclusion into /usr/ports/Mk. The scheme currently works only on amd64, due to a small piece of the userspace virtual environment implementation involving machine code manipulation, but this can be extended to support other architectures. For those interested in helping to test, the work is available on Github as follows. The ports are synchronized to quarterly 2019Q3 as of Aug 15. An example of how it can be tested (no need for superuser): (note that /usr/lib/debug/libexec/ld-elf.so.1.debug (from base-dbg or from installworld) must exist) $ git clone https://github.com/therontarigo/freebsd-ports -b separated --depth 1 $ cd freebsd-ports $ make PORTS_SEPARATED_BUILD=1 PORTSDIR=$PWD PORTBLDBASE=$HOME/ports -C devel/gmake config-recursive package-recursive To be extra sure it is not relying on /usr/local: try instead with LOCALBASE=/usr/nlocal (of course then packages won't install to the default local prefix). Resulting packages will be in $HOME/ports/packages/All/ This modified ports tree is intended to behave exactly as the official one when PORTS_SEPARATED_BUILD=1 is NOT used. Anywhere that this is violated is a bug that I must fix. Much cleanup and far more thorough testing beyond what I have accomplished in the time of this project are needed before this should be used in production or considered for merging into official ports tree. Your feedback would be much appreciated and will help me to prepare a report on the project's successes, shortcomings, and future directions with respect to the community's needs. Theron
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bd94f2f2-b074-0ea6-fa8d-27c1f202c422>