Date: Mon, 22 Jul 2013 23:08:36 +0000 (UTC) From: Nathan Whitehorn <nwhitehorn@FreeBSD.org> To: src-committers@freebsd.org, svn-src-user@freebsd.org Subject: svn commit: r253557 - user/nwhitehorn/condorports Message-ID: <201307222308.r6MN8aSg056744@svn.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: nwhitehorn Date: Mon Jul 22 23:08:36 2013 New Revision: 253557 URL: http://svnweb.freebsd.org/changeset/base/253557 Log: Remove comment and add documentation. Added: user/nwhitehorn/condorports/README Modified: user/nwhitehorn/condorports/buildportsdag.sh Added: user/nwhitehorn/condorports/README ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ user/nwhitehorn/condorports/README Mon Jul 22 23:08:36 2013 (r253557) @@ -0,0 +1,90 @@ +condorports +----------- + +This is a version of portbuild designed to exploit the features of the HTCondor +batch scheduler instead of a homegrown scheduler, in particular the abilities +of the Condor DAG manager. + +Features: +- Heterogeneous clusters: support builds for multiple architectures using a + mixed architecture cluster +- No shared file system required: distfiles and packages are transferred via + condor file transfer to the build node, requiring only static read-only copies + of the base system and the ports tree[s] on build nodes +- Flocking support: if you have multiple Condor clusters, jobs submitted on one + can transparently "flock" to another, letting the package builds exploit the + resources of geographically distributed computing clusters +- Many builds for many architectures, OS releases, and ports trees can + continue in parallel +- Uses standard HPC software for scheduling with high performance, reliability, + and security with only a light layer on top (155 lines of shell scripts) + +Submit node prerequisites: +-------------------------- +- sysutils/condor, configured to allow job submission +- Copies of the relevant ports trees, at the same paths as they are on the + build nodes + +Build node prerequisites: +------------------------- +- sysutils/condor, configured using condor_config.local in this directory + (adapted for the system in question) +- security/sudo, with sudoers file in this directory +- A lot of local disk space in /scratch +- A copy of all relavent worlds in /releases/<releasenames> +- The prepbuildjail and reapbuildjail scripts installed in /pkgscripts + +Using: +------ + +The basic procedure is: +cd condorports +./buildportsdag.sh +condor_submit_dag -maxpre 10 ports.dag +<Eat one to several meals depending on cluster size> + +This will configure a Condor DAG for the ports tree in /usr/ports (the default, +see the options section below), the current system's architecture +(again see options), and release (again), placing the packages in the current +directory (once more). Submitting the DAG to the scheduler will begin the build. + +NOTE: The -maxpre option is *very* important, as this controls the number of +simultaneous make fetch operations to run on the submit node. If you don't +set it, you will get thousands of fetches at once. + +As the build runs, log files will appear in logs and built packages in a +directory by default in condorports. The build queue can be inspected using +condor_q. + +Options to buildportsdag.sh are specified using environment variables: +- PORTSDIR: Name of ports directory to build in. This *MUST* occur on the same + path on both the build and submit nodes. This directory may be read only. + Default: /usr/ports +- PKGSDIR: Name of directory on submit node in which to place built packages. + Need not exist on the build nodes and can be on a local disk. + Default: $PORTSDIR/packages +- DISTHORDE: Name of directory on submit node in which to store and find + distfiles. Can be shared across ports tree. Need not exist on the build nodes + and can be on a local disk. This directory must be writeable by the user + running the job submission. + Default: $PORTSDIR/distfiles +- ARCH: Value of uname -p for which to build + Default: Value of uname -p on submit node +- RELEASE: Name of FreeBSD release to use. This directory must exist on the + build nodes (it need not be accessible from the submit node) in + /releases/$RELEASE + Default: $ARCH/`uname -r` +- STAGEDIR: Name of directory in which to stage files for transfer for this + particular build. Must be unique to this particular package build run and + will be deleted at completion. + Default: $(pwd)/stage-$ARCH/$PKGSDIR + +TODO: +---- +Not all features are fully implemented yet. In particular: +- Need to configure networking in the build jails +- Need to advertise release and ports directory availability in the machine + classad (via STARTD_CRON) so that jobs are scheduled only on hosts that + have acquired the appropriate resources. This is especially important in + the flocking case. + Modified: user/nwhitehorn/condorports/buildportsdag.sh ============================================================================== --- user/nwhitehorn/condorports/buildportsdag.sh Mon Jul 22 22:16:47 2013 (r253556) +++ user/nwhitehorn/condorports/buildportsdag.sh Mon Jul 22 23:08:36 2013 (r253557) @@ -6,8 +6,6 @@ : ${RELEASE=$ARCH/`uname -r`} : ${STAGEDIR=$(pwd)/stage-$ARCH/$PKGSDIR} -#ports="/usr/ports/games/sl /usr/ports/ports-mgmt/pkg /usr/ports/japanese/nkf" - mkdir -p $STAGEDIR dagjobs=$(pwd)/ports.dagjobs
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201307222308.r6MN8aSg056744>