From owner-svn-ports-head@FreeBSD.ORG Sat Aug 31 13:22:02 2013 Return-Path: Delivered-To: svn-ports-head@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 650539F8; Sat, 31 Aug 2013 13:22:02 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5226A2168; Sat, 31 Aug 2013 13:22:02 +0000 (UTC) Received: from svn.freebsd.org ([127.0.1.70]) by svn.freebsd.org (8.14.7/8.14.7) with ESMTP id r7VDM2uo076582; Sat, 31 Aug 2013 13:22:02 GMT (envelope-from bdrewery@svn.freebsd.org) Received: (from bdrewery@localhost) by svn.freebsd.org (8.14.7/8.14.5/Submit) id r7VDM2wa076581; Sat, 31 Aug 2013 13:22:02 GMT (envelope-from bdrewery@svn.freebsd.org) Message-Id: <201308311322.r7VDM2wa076581@svn.freebsd.org> From: Bryan Drewery Date: Sat, 31 Aug 2013 13:22:02 +0000 (UTC) To: ports-committers@freebsd.org, svn-ports-all@freebsd.org, svn-ports-head@freebsd.org Subject: svn commit: r325805 - head/Mk X-SVN-Group: ports-head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-ports-head@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SVN commit messages for the ports tree for head List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Aug 2013 13:22:02 -0000 Author: bdrewery Date: Sat Aug 31 13:22:01 2013 New Revision: 325805 URL: http://svnweb.freebsd.org/changeset/ports/325805 Log: - make fetch/checksum: If a fetched file does not match the expected size, delete it and try the next site, if there is one to try. Normally fetch(1) will detect that the size is wrong, fail, and the 'fetch' target will move on to the next site: => Attempting to fetch http://www.chiark.greenend.org.uk/~sgtatham/agedu/agedu-r9723.tar.gz fetch: http://www.chiark.greenend.org.uk/~sgtatham/agedu/agedu-r9723.tar.gz: size mismatch: expected 155321, actual 163319 => Attempting to fetch http://ftp.FreeBSD.org/pub/FreeBSD/ports/local-distfiles/sunpoet/agedu-r9723.tar.gz agedu-r9723.tar.gz 100% of 151 kB 259 kBps However, if the remote server does not properly advertise the size of the file, fetch(1) will still download the file, and succeed. The 'fetch' target then stops, assuming it has a proper file. This is returned to 'checksum', which fails, and then tries a refetch, but on the same broken site. In this example, the size is expected to be 214984, but a 4830 size is fetched and checksummed. => Attempting to fetch http://kornelix.squarespace.com/storage/downloads/picpuz-2.1.1.tar.gz fetch: http://kornelix.squarespace.com/storage/downloads/picpuz-2.1.1.tar.gz: size unknown fetch: http://kornelix.squarespace.com/storage/downloads/picpuz-2.1.1.tar.gz: size of remote file is not known picpuz-2.1.1.tar.gz 4830 B 48 kBps ===> Fetching all distfiles required by picpuz-2.1.1_5 for building => SHA256 Checksum mismatch for picpuz-2.1.1.tar.gz. Now the 'fetch' target will verify the size is proper before returning to 'checksum': => Attempting to fetch http://kornelix.squarespace.com/storage/downloads/picpuz-2.1.1.tar.gz fetch: http://kornelix.squarespace.com/storage/downloads/picpuz-2.1.1.tar.gz: size unknown fetch: http://kornelix.squarespace.com/storage/downloads/picpuz-2.1.1.tar.gz: size of remote file is not known picpuz-2.1.1.tar.gz 4830 B 130 kBps => Fetched file size mismatch (expected 214984 actual 4830) => Trying next site => Attempting to fetch http://www.stasyan.com/devel/distfiles/picpuz-2.1.1.tar.gz picpuz-2.1.1.tar.gz 100% of 209 kB 20 kBps 00m00s ===> Fetching all distfiles required by picpuz-2.1.1_5 for building => SHA256 Checksum OK for picpuz-2.1.1.tar.gz. Reviewed by: bapt With hat: portmgr Modified: head/Mk/bsd.port.mk Modified: head/Mk/bsd.port.mk ============================================================================== --- head/Mk/bsd.port.mk Sat Aug 31 13:14:24 2013 (r325804) +++ head/Mk/bsd.port.mk Sat Aug 31 13:22:01 2013 (r325805) @@ -3413,7 +3413,13 @@ do-fetch: else \ SORTED_MASTER_SITES_CMD_TMP="${SORTED_MASTER_SITES_DEFAULT_CMD}" ; \ fi; \ - for site in `eval $$SORTED_MASTER_SITES_CMD_TMP ${_RANDOMIZE_SITES}`; do \ + sites_remaining=0; \ + sites="`eval $$SORTED_MASTER_SITES_CMD_TMP ${_RANDOMIZE_SITES}`"; \ + for site in $${sites}; do \ + sites_remaining=$$(($${sites_remaining} + 1)); \ + done; \ + for site in $${sites}; do \ + sites_remaining=$$(($${sites_remaining} - 1)); \ ${ECHO_MSG} "=> Attempting to fetch $${site}$${file}"; \ CKSIZE=`alg=SIZE; ${DISTINFO_DATA}`; \ case $${file} in \ @@ -3422,7 +3428,16 @@ do-fetch: *) args=$${site}$${file};; \ esac; \ if ${SETENV} ${FETCH_ENV} ${FETCH_CMD} ${FETCH_BEFORE_ARGS} $${args} ${FETCH_AFTER_ARGS}; then \ - continue 2; \ + actual_size=`stat -f %z "$${file}"`; \ + if [ $${actual_size} -eq $${CKSIZE} ]; then \ + continue 2; \ + else \ + ${ECHO_MSG} "=> Fetched file size mismatch (expected $${CKSIZE}, actual $${actual_size})"; \ + if [ $${sites_remaining} -gt 1 ]; then \ + ${ECHO_MSG} "=> Trying next site"; \ + ${RM} -f $${file}; \ + fi; \ + fi; \ fi; \ done; \ ${ECHO_MSG} "=> Couldn't fetch it - please try to retrieve this";\