Date: Sat, 26 Jul 2008 14:49:26 +0000 (UTC) From: Kris Kennaway <kris@FreeBSD.org> To: ports-committers@FreeBSD.org, cvs-ports@FreeBSD.org, cvs-all@FreeBSD.org Subject: cvs commit: ports/Tools/portbuild/scripts dopackages Message-ID: <200807261449.m6QEnQok001408@repoman.freebsd.org>
next in thread | raw e-mail | index | archive | help
kris 2008-07-26 14:49:26 UTC
FreeBSD ports repository
Modified files:
Tools/portbuild/scripts dopackages
Log:
* Cleanup
* Catch up to build ID directory changes
* Improve usage()
* Fix a variety of small bugs
* Remove support for -ftp builds: we have not supported direct
uploading for many years due to the desire to manually inspect build
output for quality
* All data associated to a build is now localized in its own directory
named according to a build ID:
/var/portbuild/${arch}/${branch}/builds/${buildid}, where ${buildid}
is the creation time. These are actually ZFS filesystems.
* Tasks such as cloning a new build, updating a ZFS snapshot, and
cleaning up a build are exported to the "build" script, which can be
used independently.
* Creating a new build is done by ZFS cloning and takes a couple of
seconds since it is copy-on-write (i.e. no data needs to be copied).
* Ports and source trees are also cloned from pre-updated ZFS images
(updated regularly from the "updatesnap" cron job). In most cases
we do not care if we are building a ports tree that is an hour or so
old since it will become outdated almost immediately anyway, so no
matter what we do there will be times when a port has been fixed by
the time the build error is generated by a client.
* In case an up-to-the-second tree is desired, the -portscvs and
-srccvs switches update the existing ports tree via CVS.
* -noports and -nosrc can be used to prevent any automatic changes to
the ports tree. This is useful for dealing with local
modifications (e.g. for -exp builds), since the default when
creating a new build is to replace the previous trees with fresh,
pristine trees. If you forget to use this then any local changes
that are not also present in other trees will be lost.
* By default we keep two builds for each arch/branch pair. These
build IDs also may be referred to via "latest" and "previous"
symlinks. When creating a new build, the old "previous" build is
destroyed by default, unless it was originally created using the
-keep switch. This prevents the build from being destroyed
automatically.
* By default when a build finishes all of the clients are completely
cleaned up (i.e. all build data such as ports trees, tarballs,
client chroots, etc are deleted). This is needed to save space on
the clients. If you expect to *immediately* perform further builds
after this one completes, the -nocleanup switch prevents this step.
Otherwise they will just be set up again if further builds are
scheduled.
* Try to parallelize build pre-processing as much as possible, by
running jobs in the background wherever possible. In several places
we operate on the same parts of the filesystem from multiple jobs,
so we can make good use of caching to improve performance
* Clients no longer need to be set up explicitly at the start of the
build, they will be set up on-demand when the first job is
dispatched to them. This allows fast clients or those that already
have been set up to begin building ports as soon as possible, while
slow clients are set up in the background. It also improves
robustness of client recovery, e.g. if the client was offline at the
time of build startup but later brought back online.
* Optimize copying back in the previous set of restricted packages by
hardlinking instead of copying.
TODO: The record of failed ports is arch/branch-global still. This is
the only thing preventing us from running concurrent builds of the
same arch/branch (e.g. while one is stuck building openoffice, the
next build can start to keep the cluster busy). The difficulty is
that one build from a later ports tree may signal that a build was
successful, then a phase 2 build from an earlier ports tree may
indicate that it was broken. The solution is probably to migrate this
to a real database instead of a flat file, and query it for the set of
broken ports as of a certain ports tree date.
Revision Changes Path
1.48 +586 -501 ports/Tools/portbuild/scripts/dopackages
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200807261449.m6QEnQok001408>