Date: Mon, 31 Dec 2007 13:00:04 -0500 (EST) From: "Mikhail T." <mi@aldan.algebra.com> To: FreeBSD-gnats-submit@FreeBSD.org Subject: bin/119196: [wish] parallel download from multiple servers by fetch Message-ID: <200712311800.lBVI046T004917@aldan.algebra.com> Resent-Message-ID: <200712311830.lBVIU2Sj034751@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 119196 >Category: bin >Synopsis: [wish] parallel download from multiple servers by fetch >Confidential: no >Severity: non-critical >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: change-request >Submitter-Id: current-users >Arrival-Date: Mon Dec 31 18:30:02 UTC 2007 >Closed-Date: >Last-Modified: >Originator: Mikhail T. >Release: FreeBSD 6.3-PRERELEASE amd64 >Organization: Virtual Estates, Inc. >Environment: FreeBSD >Description: In most cases, when fetch(1) is invoked (or libfetch called into), it is to either download sources for a port, or binaries for a package. In all of those cases multiple servers hosting the same files are known to the caller. Unfortunately, the caller must use the servers /in sequence/, because fetch does not allow downloading different ranges of the same file from different sources /in parallel/. This is, of course, suboptimal as some servers (such as ftp.freebsd.org) are hit by much higher loads. The users may also have a (much) wider download bandwidth, than an upload bandwidth, that a particular server has (or would allow a single client to use). It should be possible to implement parallel downloads (over different protocols -- http, ftp, etc.). The implementation should also be adaptive: if the download from one of the servers finishes faster, the program should proceed to automatically fetch a not-yet-downloaded sub-range of the range, that is being downloaded from another server. Various "p2p" downloading programs (torrent, gnuttella) implement this feature already -- out of necessity. We should do it, because we can. No, I don't have a patch. Not yet, anyway... Somebody looking for a cool project is most welcome to this idea. >How-To-Repeat: >Fix: >Release-Note: >Audit-Trail: >Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200712311800.lBVI046T004917>