From owner-freebsd-ports@FreeBSD.ORG Thu Mar 16 20:03:55 2006 Return-Path: X-Original-To: freebsd-ports@freebsd.org Delivered-To: freebsd-ports@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6761A16A41F for ; Thu, 16 Mar 2006 20:03:55 +0000 (UTC) (envelope-from vadim_nuclight@mail.ru) Received: from mx3.mail.ru (mx3.mail.ru [194.67.23.149]) by mx1.FreeBSD.org (Postfix) with ESMTP id 03C8743D45 for ; Thu, 16 Mar 2006 20:03:54 +0000 (GMT) (envelope-from vadim_nuclight@mail.ru) Received: from [83.172.2.47] (port=60655 helo=[83.172.2.47]) by mx3.mail.ru with esmtp id 1FJyhN-0009qy-00 for freebsd-ports@freebsd.org; Thu, 16 Mar 2006 23:03:53 +0300 Date: Fri, 17 Mar 2006 02:02:44 +0600 From: Vadim Goncharov X-Mailer: The Bat! (v2.10.01) Personal Organization: Home X-Priority: 3 (Normal) Message-ID: <1965362278.20060317020244@mail.ru> To: freebsd-ports@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Subject: Getting distfiles for entire tree X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Vadim Goncharov List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Mar 2006 20:03:55 -0000 Hello freebsd-ports, I need to maintain a local distfiles mirror (all N-ty gigabytes) for almost entire ports tree (some lang categories excluded). Also, I have limited access to Internet - not all distfiles could be fetched (however eventually they'll do). So, as I can't use plain ``cd /usr/ports && make fetch'', I've made a script, which, for speeding up, fetches each category in parallel. Script was simple, so it hasn't got all, mainly because of interactive ports, etc. So I searched a little and, after reading some tips, modified script - commented ``find'' and inserted ``daemon''. Here it is: #!/bin/sh CATEGORIES="accessibility archivers astro audio benchmarks biology cad comms converters databases deskutils devel dns editors emulators finance ftp games graphics irc japanese java lang mail math mbone misc multimedia net net-im net-mgmt net-p2p news palm print russian science security shells sysutils textproc ukrainian www x11 x11-clocks x11-fm x11-fonts x11-servers x11-themes x11-toolkits x11-wm" LOG=/data/FreeBSD/fetchlog # vars for fetch export HTTP_TIMEOUT=30 export FTP_TIMEOUT=30 for i in $CATEGORIES ; do cd /usr/ports/$i || exit 1 # find . -type d -mindepth 1 -maxdepth 1 -exec sh -c "cd {} && make fetch" \; > ${LOG}-$i 2>&1 & daemon sh -c "BATCH=yes NO_IGNORE=yes PACKAGE_BUILDING=yes make -k checksum" > ${LOG}-$i 2>&1 done First version (find) forked 300 processes at beginning. Second (daemon) forked 400, and, as it was going, I saw strange things, e.g. error messages in logs like that: ===> abclock Abort trap *** Error code 134 (continuing) or ===> xtu => MD5 Checksum OK for xtu-1.2.tar.gz. => SHA256 Checksum OK for xtu-1.2.tar.gz. *** Signal 6 (continuing) `checksum' not remade because of errors. Why did it happen ? Lack of resources? But I ran it as root, limits were soft, and at least that 400 processes forked! But then some anothe shit happened - system began to build and install packages without checking if they were actually installed, e.g. it made mess installing expat2 over expat-1.95 (I had to deal with it manually later), so I was forced to interrupt the script. So, what happened (why it began to install, why errors?) and how should I create local distfiles mirror ? -- Best regards, Vadim Goncharov ICQ UIN 166852181 mailto:vadim_nuclight@mail.ru