Date: Mon, 28 Jan 2013 14:51:49 -0600 From: Stephen Montgomery-Smith <stephen@missouri.edu> To: ctm-users@freebsd.org Subject: Re: CTM index on servers Message-ID: <5106E4E5.5060905@missouri.edu> In-Reply-To: <1359401402-8482577.26305758.fr0SJTdBX014869@rs149.luxsci.com> References: <1359401402-8482577.26305758.fr0SJTdBX014869@rs149.luxsci.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 01/28/2013 01:29 PM, Isaac (.ike) Levy wrote: > Hi Stephen, ctm-users, > > New to this list, coming from a lively thread on freebsd-stable@ > > For me and a lot of admins/users, fetching sources just got a lot harder for a *lot* of admins/users. A page was raised to try to help cull facts out of all the confusing details, here: > > https://wiki.freebsd.org/UsersFetchingSource > > At this point the page may be more confusing itself :) > > -- > Enter: CTM > > I'm delighted to learn about ctm(1) usage, and quite interested in using it to fetch REL and RELENG sources, and keep it up to date (to redistribute internally across a cluster of servers). > > I know a *ton* of *BSD admins who love c[v]sup, but I don't know any who use ctm(1) any more, so I'm sorry if my questions are naive. > > On Jan 28, 2013, at 1:53 PM, Stephen Montgomery-Smith wrote: >>> On the FTP server, is there an index of which CTM delta files are on the FTP server? I'd like to automate fetching just the new deltas... >>> >>> If there was perhaps 1 file with a consistent name, I could fetch that on a nightly basis and fetch the other CTM files. >> >> I can easily put in something like this. Maybe the output of ls in each directory? >> >> If you have more ideas on this, sign up to the ctm-users mailing list, and we can discuss exactly what is wanted. > > I'm very interested in seeing some kind of 'index' file put in place, a simple file which lists the ctm delta (and xEmpty) files. > > This way, users could do something in a shell/cron job, like: > > fetch ftp://ftp.freebsd.org/pub/FreeBSD/development/CTM/src-9/index > # > # diff against the last ctm file, or a directory listing containing the delta files > # download/apply any ctm delta files which have not yet been applied > > -- > It's an open question weather or not the entire CTM dir should have the index, or the individual releases (or both?) > > e.g. > /pub/FreeBSD/development/CTM/src-9/index > - vs - > /pub/FreeBSD/development/CTM/index > > (I assume it would be appropriate also to provide sig files for the index, as well.) > > -- > The index could be as simple as: > > $ echo "`date +%s`" ; find . -type f -name 'src*' | sort | sed 's/.*\///' > 1359382951 > src-9.0400xEmpty.gz > src-9.0401.gz > src-9.0401.gz.sig > src-9.0402.gz > src-9.0402.gz.sig > src-9.0403.gz > src-9.0403.gz.sig > $ > > Could get fancier as necessary later, (perhaps date modified for each file, etc…) > > -- > With some sort of index, the ease-of-use functionality of c[v]sup could be trivially replicated in sh(1). > Even just as a prototype, this is a task I will enthusiastically do, even if only for my own use- (I started something already which culls FTP tarballs, a less than ideal solution). > > What do yall' think? The index idea would be very easy for me to implement. Unless anyone has a serious objection, I will probably go ahead and implement it as above. Except I will not include the time stamp in the file, as the file already has a modification time on it. I will probably sign the index file as well, but I don't think it is necessary, as if that information is corrupted it is unlikely to do any damage. But ... I use a script like this: I store all the ctm files in /usr/home/stephen/ctm or subdirectories thereof. This script sees what is already there, and then tries to fetch any future ctm deltas. #!/bin/sh cd /usr/home/stephen/ctm/incoming bases=`find /usr/home/stephen/ctm -type f | sed -n -E -e 's+.*/++' -e 's/\.[[:digit:]]{4}\.gz//p' | sort -u` for b in $bases; do while true; do index=`find /usr/home/stephen/ctm -type f -name "$b*" | sed -n -E 's/.*([[:digit:]]{4}).*/\1/p' | sort -n | tail -1 | xargs expr 1 + | xargs printf "%04d"` fetch ftp://ftp.freebsd.org/pub/CTM/$b/$b.$index.gz || break done done echo ls
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5106E4E5.5060905>