From owner-freebsd-ports@FreeBSD.ORG Sun Apr 1 02:16:56 2012 Return-Path: Delivered-To: ports@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C5D32106564A for ; Sun, 1 Apr 2012 02:16:56 +0000 (UTC) (envelope-from anthonybryan@gmail.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id 48D298FC18 for ; Sun, 1 Apr 2012 02:16:55 +0000 (UTC) Received: by yhgm50 with SMTP id m50so999587yhg.13 for ; Sat, 31 Mar 2012 19:16:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=EMb7Uu70eWxBIoILZHnH7kw9ec3kiwLLJH0bdkl0538=; b=thv2Xe+OtTz/EN81PbJQJP1U2iLUUiZyScZLU3e3J7gSIdaX+q/Q0ndsX+IBzgW413 MUWgB5wqzSbIVEHKKDkjV94oMJDy2Ctw1mI0Ae7wVuzSlBkHDyoilUQmqCE4NIduWlG5 2fD/oP6b7pw9fY8ITF05+7NbJcIdqaVtKYuWMlCdipESHysl16wOJCqTXLKFIfznUYMt gb5KmpFluPCYdP2Q7eL44Hs6nTcZ6Hu922zz5OAU5L7guOuyGmy+wzHO6hk/zqYFWS9q FCbknp9BH3apDJ6htWs57wBGv+IwzPhcBqk7BQCcPPX5tMdmw/qxFXPwnvf6zmE2vhXX SXng== MIME-Version: 1.0 Received: by 10.236.78.2 with SMTP id f2mr2947354yhe.118.1333246608140; Sat, 31 Mar 2012 19:16:48 -0700 (PDT) Received: by 10.146.95.15 with HTTP; Sat, 31 Mar 2012 19:16:48 -0700 (PDT) Date: Sat, 31 Mar 2012 22:16:48 -0400 Message-ID: From: Anthony Bryan To: ports@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Subject: Re: Support for RfC 6249 - aka Metalinks - in ports? / Google Summer of Code X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Apr 2012 02:16:56 -0000 hi, I came across the following about metalink (a way to list mirrors & checksums)/ports: http://lists.freebsd.org/pipermail/freebsd-ports/2011-September/070607.html (follows after my sig) along with http://wiki.freebsd.org/Metalink we'd be interested in using one of our Google Summer of Code slots for this, if someone from your end was interested in mentoring. it's up on our ideas page, but we haven't had any nibbles yet :) http://www.metalinker.org/ideas.html --=20 (( Anthony Bryan ... Metalink [ http://www.metalinker.org ] =A0 )) Easier, More Reliable, Self Healing Downloads ---- Raphael Eiselstein rabe at uugrn.org Hi, I wonder if there will be support for RfC 6249 / Metalink in the FreeBSD Ports infrastructure or someone is working / thinking about implementing this. >From http://wiki.freebsd.org/Metalink ------------------------------------------------------------- Metalink, an Open Standard, makes downloading easier by bundling the various ways (FTP/HTTP/P2P) to get files into one format for easier downloads. Simpler. Faster. More Reliable. Better. Metalink is a great concept which might be added to the ports infrastructure. Metalink files can be preprocessed in order to retain big mirror lists in bsd.sites.mk. Or an XML guru can convert bsd.sites.mk to an XML version and include it into metalinks in some way. Stub metalinks can be easily created for each port with a simple script. At this time only www/aria2 has some Metalink support. Aria2 has a few dependencies, but it's completely unnecessary to cram it into the base system. FETCH_DEPENDS would work very nice. Metalink support can be implemented simultaneously with P2P support, which is long overdue. Aria2 fully supports Metalink 3.0 and other downloaders will soon follow. Metalink (last edited 2008-06-17 21:37:49 by localhost) ------------------------------------------------------------- My approach would be some kind of http proxy which knows based on a table of URLs from the ports all the mirror information (if there are mirror sites) for given files or "content". We already have MD5 and SHA256 checksums and sizes in distinfo, so we might enhance the fetch routines not to request a specific URL and filename but for a given checksum and writing the downloaded content to a known filename. Instead of requesting ftp://ftp.freebsd.org/pub/FreeBSD/ports/distfiles/aria2-1.10.9.tar.bz2 (or mirror sites) the ports might fetch http://my-proxy.example.com/sha256/51319be3006545c442fbbedb476fa7ba2fd562e0= 468d9c573e7f5261ea610af5 and my-proxy.example.com would resolve this to a list of mirror URLs using an implementation of RfC 6249 and maybe could itself be an "dynamic ftp mirror" through some local caches (using squid-proxy or similar caches). +------------------+ | FreeBSD Ports | Our ports (already have distinfo) +----+-------+-----+ | fetch | using fetch(1) +-------+ | v +--------------------------+ | http://localhost:8080/ | requesting checksums | sha256/51319be300...0af5 | from a local RfC 6249 +--------------------------+ proxy/cache which might | | | serve this from local v v v cache or just will request +------+ +-----+ +------+ the file from multiple | A | | B | | C | sources which have a copy +------+ +-----+ +------+ (of a copy (of a copy)) \ | / of the requested file \ | / originally obtained from \ | / the master site or another +---------+ "primary source". | primary | | source | +---------+ All we need to change in ports is a little enhancement which will make fetch (optionally) requesting for the sha256 checksums instead of given filenames, enabled in /etc/make.conf So, basically this would be a endoscopic surgery (=3Dminimal impact) to the freebsd ports supporting optional requesting checksums instead of static URLs from a list. Requesting static files parallel from multiple proxy caches might be implemented by parallel requesting byte ranges from different sources using 64kByte..256kByte blocks for example. Beside this, this approach wouldn't be limited to distfiles but may work for every kind of static files (tarballs, packages, ... ) by any open source project which is distributed through ftp mirrors. And all you need for this is * a filename (we already have) * a sha256 checksum (we have) * a rfc6249 proxy server (3rd party component) * a traditional HTTP/1.0 or HTTP/1.1 compliant tool (like fetch(1)) Does anyone like this approach? Does anyone work on (knows about) a concept or is developing something implementing rfc6249? Regards Raphael Eiselstein