From owner-freebsd-ports@FreeBSD.ORG Thu Aug 20 02:33:25 2009 Return-Path: Delivered-To: freebsd-ports@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5ABBD1065691; Thu, 20 Aug 2009 02:33:25 +0000 (UTC) (envelope-from amdmi3@amdmi3.ru) Received: from smtp.timeweb.ru (smtp.timeweb.ru [217.170.79.85]) by mx1.freebsd.org (Postfix) with ESMTP id E155B8FC16; Thu, 20 Aug 2009 02:33:24 +0000 (UTC) Received: from [213.148.20.85] (helo=hive.panopticon) by smtp.timeweb.ru with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.69) (envelope-from ) id 1MdxSu-0004gm-26; Thu, 20 Aug 2009 06:33:24 +0400 Received: from hades.panopticon (hades.panopticon [192.168.0.32]) by hive.panopticon (Postfix) with ESMTP id 6410DB860; Thu, 20 Aug 2009 06:33:22 +0400 (MSD) Received: by hades.panopticon (Postfix, from userid 1000) id 13E7E108842; Thu, 20 Aug 2009 06:33:15 +0400 (MSD) Date: Thu, 20 Aug 2009 06:33:15 +0400 From: Dmitry Marakasov To: freebsd-ports@FreeBSD.org Message-ID: <20090820023314.GF1295@hades.panopticon> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline User-Agent: Mutt/1.5.20 (2009-06-14) Cc: portmgr@FreeBSD.org Subject: Migration to new SourceForge url scheme now inevitable, solution X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Aug 2009 02:33:25 -0000 Hi! I've just discovered that even old ports hosted on SourceForge were switched to the new file distribution scheme (i.e. MASTER_SITE_SOURCEFORGE_PROJECT). Fetching from old urls result in redirects, which means that these ports can now only be fetched from our mirror. So we should switch to SOURECFORGE_PROJECT immediately. I've written a simple Perl script ([1]) which uses curl to try to fetch distfiles for each SourceForge-hosted port from old location, parses curl output to discover redirect target, gets SUBDIR out of it and enhances it by substituding ${PORTNAME} and ${PORTVERSION}. The script doesn't really download distfiles (--max-filesize 1000 option for curl), so it doesn't eat much traffic. However, it eats some (much!) CPU time due to many make calls. It also runs in 10 threads by default. The test run results are here: [2]. Except for 45 ports for which new URL could not be determined (that basically means that the don't fetch at all and files were likely removed), this provides useable MASTER_SITES_SUBDIRs (curl also produces escaped urls so there's no problem with spaces and other weird characters) for all SF ports, so we can generate a patch semi-automatically. I'd like to do it :) Also note that top 3 subdirs happen to be ([3]): 1018 ${PORTNAME}/${PORTNAME}/${PORTVERSION} 259 ${PORTNAME}/${PORTNAME}/${PORTNAME}-${PORTVERSION} 72 ${PORTNAME}/OldFiles thus I'd also change default MASTER_SITE_SUBDIR for SFP to ${PORTNAME}/${PORTNAME}/${PORTVERSION}. This will save us trouble of caring about subdir for ~30% of SF ports. Finally, since MASTER_SITE_SOURCEFORGE will no longer be of use, I'd remove MASTER_SITE_SOURCEFORGE_PROJECT and just use MASTER_SITE_SOURCEFORGE for the new scheme. [1] http://people.freebsd.org/~amdmi3/sf.pl.txt [2] http://people.freebsd.org/~amdmi3/sourceforge-subdirs.txt [3] http://people.freebsd.org/~amdmi3/sourceforge-subdirs-top.txt -- Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D amdmi3@amdmi3.ru ..: jabber: amdmi3@jabber.ru http://www.amdmi3.ru