From owner-svn-src-user@freebsd.org Wed Mar 22 22:49:12 2017 Return-Path: Delivered-To: svn-src-user@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B9D48D184B3 for ; Wed, 22 Mar 2017 22:49:12 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 91F919EE; Wed, 22 Mar 2017 22:49:12 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v2MMnBJk057119; Wed, 22 Mar 2017 22:49:11 GMT (envelope-from gjb@FreeBSD.org) Received: (from gjb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v2MMnBd7057118; Wed, 22 Mar 2017 22:49:11 GMT (envelope-from gjb@FreeBSD.org) Message-Id: <201703222249.v2MMnBd7057118@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: gjb set sender to gjb@FreeBSD.org using -f From: Glen Barber Date: Wed, 22 Mar 2017 22:49:11 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org Subject: svn commit: r315758 - user/gjb/thermite X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Mar 2017 22:49:12 -0000 Author: gjb Date: Wed Mar 22 22:49:11 2017 New Revision: 315758 URL: https://svnweb.freebsd.org/changeset/base/315758 Log: Add a README.txt explaining the high-level overview of how thermite.sh and related bits work. Sponsored by: The FreeBSD Foundation Added: user/gjb/thermite/README.txt (contents, props changed) Added: user/gjb/thermite/README.txt ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ user/gjb/thermite/README.txt Wed Mar 22 22:49:11 2017 (r315758) @@ -0,0 +1,88 @@ +# +# $FreeBSD$ +# + +1) Configuration Files: + + - Each architecture and individual kernel have their own configuration file + used by release.sh. Each branch has its own defaults-X.conf file which + contains entries common throughout each architecture, where overrides or + special variables are set and/or overridden in the per-build files. + + The per-build configuration file naming scheme is in the form of: + + ${revision}-${TARGET_ARCH}-${KERNCONF}-${type}.conf + + where the uppercase variables are equivalent to what make(1) uses in the + build system, and lowercase variables are set within the configuration + files, mapping to the major version of the respective branch. + + - Each branch also has its own builds-X.conf file, which is used by + thermite.sh. The thermite.sh script iterates through each ${revision}, + ${TARGET_ARCH}, ${KERNCONF}, and ${type} value, creating a master list + of what to build. However, a given combination from the list will only + be built if the respective configuration file exists, which is where the + naming convention above is relevant. + + - There are two paths of file sourcing: + + - builds-X.conf -> main.conf: + This controls thermite.sh behavior. + + - X-arch-KERNCONF-type.conf -> defaults-X.conf -> main.conf + This controls release/release.sh behavior. + +2) Filesystem Layout: + + - The official release build machines have a specific filesystem layout, + which using ZFS, thermite.sh takes heavy advantage of with clones, + snapshots, etc., ensuring a pristine build environment. + + - The build scripts reside in /releng/scripts-snapshot/scripts or + /releng/scripts-release/scripts respectfully, to avoid collisions between + an RC build from a releng branch versus a STABLE snapshot from the + respective stable branch. + + - A separate dataset exists for the final build images, /snap/ftp. This + directory contains both snapshots and releases directories. They are + only used if the EVERYTHINGISFINE variable is defined in main.conf. + + - As thermite.sh iterates through the master list of combinations and + locates the per-build configuration file, a zfs dataset is created under + the /releng directory, such as /releng/12-amd64-GENERIC-snap. The src, + ports, and doc trees are checked out to separate zfs datasets, such as + /releng/12-src-snap, which are then cloned into the respective build + datasets. This is done to avoid checking out a given tree more than + once. + +3) Helper Scripts: + + - To avoid repetition and possible human error, a few scripts were written + to help keep things as automated as possible: + + - zfs-setup.sh: + Destroys and creates pristine zfs datasets for each build. + + - setrev.sh: + Retrieves the 'Last Changed Revision' from the target branch and writes + the version to a 'svnrev_src' file, and outputs the date in YYYYMMDD + format to a builddate file. + + - get-checksums.sh: + When all builds have completed, this script will iterate through and + generate a list of sha512 and sha256 checksums for all builds. + + - generate-email.pl: + This script generates the snapshot announcement email text. + +4) Example Usage: + + root@builder:~ # mkdir -p /releng/scripts-snapshot/scripts + root@builder:~ # cd /releng/scripts-snapshot/scripts + root@builder:~ # svn co svn://svn.freebsd.org/base/user/gjb/thermite . + root@builder:~ # ./zfs-setup.sh -c ./builds-12.conf + root@builder:~ # ./setrev.sh -b head + root@builder:~ # ./thermite.sh -c ./builds-12.conf + root@builder:~ # ./get-checksums.sh -c ./builds-12.conf | ./generate-email.pl \ + > 12-snap-mail +