From owner-freebsd-ports Mon Mar 18 20:10:14 2002 Delivered-To: freebsd-ports@hub.freebsd.org Received: from freefall.freebsd.org (freefall.FreeBSD.org [216.136.204.21]) by hub.freebsd.org (Postfix) with ESMTP id D9A6B37B404 for ; Mon, 18 Mar 2002 20:10:01 -0800 (PST) Received: (from gnats@localhost) by freefall.freebsd.org (8.11.6/8.11.6) id g2J4A1a74681; Mon, 18 Mar 2002 20:10:01 -0800 (PST) (envelope-from gnats) Received: from heaven.gigo.com (gigo.com [207.173.11.186]) by hub.freebsd.org (Postfix) with ESMTP id B102F37B404 for ; Mon, 18 Mar 2002 20:05:09 -0800 (PST) Received: from bsace7005.dsl.telebrasilia.net.br (unknown [200.163.6.195]) by heaven.gigo.com (Postfix) with ESMTP id AA391B70F for ; Mon, 18 Mar 2002 20:05:02 -0800 (PST) Received: (qmail 52401 invoked by uid 1001); 19 Mar 2002 04:03:40 -0000 Message-Id: <20020319040340.52400.qmail@exxodus.fedaykin.here> Date: 19 Mar 2002 04:03:40 -0000 From: Mario Sergio Fujikawa Ferreira Reply-To: Mario Sergio Fujikawa Ferreira To: FreeBSD-gnats-submit@FreeBSD.org X-Send-Pr-Version: 3.113 Subject: ports/36078: Fix MASTER_SITES_NN recursive bug Sender: owner-freebsd-ports@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org >Number: 36078 >Category: ports >Synopsis: Fix MASTER_SITES_NN recursive bug >Confidential: no >Severity: non-critical >Priority: low >Responsible: freebsd-ports >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Mon Mar 18 20:10:01 PST 2002 >Closed-Date: >Last-Modified: >Originator: Mario Sergio Fujikawa Ferreira >Release: FreeBSD 4.5-STABLE i386 >Organization: >Environment: System: FreeBSD exxodus.fedaykin.here 4.5-STABLE FreeBSD 4.5-STABLE #2: Thu Mar 7 23:13:19 BRT 2002 root@exxodus.fedaykin.here:/usr/obj/usr/src/sys/LIOUX i386 Up to date ports tree CVSup updated as of 2002/03/18 against cvs.freebsd.org previous to all testing. >Description: Fix a fetch bug introduced in bsd.port.mk rev 1.396 by my patch MASTER_SITES_NN Pointy hat goes to: lioux Also, "correct" historical ordering behavior of a fetch list constructed by {MASTER,PATCH}_SITES_SUBDIR >How-To-Repeat: Hi, Just fixing some bugs. And, a little enhancement. :-) This is all regarding the MASTER_SITES_NN patch introduced in bsd.port.mk rev 1.396 A list of the patches: 1) patch-bsd.port.mk-fix 2) patch-bsd.port.mk-enhance 3) patch-bsd.port.mk-strict A list of the test Makefiles (packed in a uuencode tgz due to size) All Makefile tests are run as 'make DISTDIR=/tmp -f fill_appropriate_makefile fetch-list' A) Makefile-qmail B) Makefile-ghostscript C) Makefile-enhance All these files are coming in a follow email since send-pr is not exactly good with files. (I) First, and foremost, lets begin fixing a bug introduced by me. If one tries Makefile either (A) or (B) with an up to date ports tree, one will notice that several screens will scroll before it finishes. For example, $ make DISTDIR=. -f Makefile-ghostscript fetch-list | tr "||" "\012" | grep "tp://" | sort will show one the fetch commands while $ make DISTDIR=. -f Makefile-ghostscript fetch-list | tr "||" "\012" | grep "tp://" | sort | wc should return one the number of fetch commands: 995; which is absurd. If ones inspects the line without '| wc', one will notice duplicated lines. Lets remove the duplicated lines $ make DISTDIR=. -f Makefile-ghostscript fetch-list | tr "||" "\012" | grep "tp://" | sort | uniq and $ make DISTDIR=. -f Makefile-ghostscript fetch-list | tr "||" "\012" | grep "tp://" | sort | uniq | wc which returns a reasonable value of 104 fetch commands. Similarly, $ make DISTDIR=. -f Makefile-qmail fetch-list | tr "||" "\012" | grep "tp://" | sort | wc returns 206 while $ make DISTDIR=. -f Makefile-qmail fetch-list | tr "||" "\012" | grep "tp://" | sort | uniq | wc returns 45. Thus, one can conclude that either a variable is being used without being cleaned (initialiazed with no content) or the algorithm is doing something utterly wrong. Close inspection shows that it's a simple bug, a variable was being re-used without being cleaned. Design had it that it was cleaned but the version sent to portmgr for commit had this bug (either confusion on my part or simple mistake). Anyway, patch (1) fixes this. This patch levels us with previous behavior of fetch-* targets. (II) Well, patch (1) brought us to the expected behavior of bsd.port.mk rev 1.395 previous to MASTER_SITES_NN patch. However, previous behavior had a little glitch. All following steps assume that patch (1) has been applied. Makefile (C) is excellent to exhibit the glitch. MASTER_SITES= http://www.gns.com.br/%SUBDIR%/:GIMP_PRINT,HPINKJET,RENO \ http://www.dod.com/:RENO http://2/%SUBDIR%/ \ http://something/%SUBDIR%/ http://another/ MASTER_SITE_SUBDIR= ${PORTNAME} gimp-print/:GIMP_PRINT \ hpinkjet/:HPINKJET heilo/:DEFAULT DISTFILES= file-1:GIMP_PRINT file-2 $ make DISTDIR=. -f Makefile-enhance fetch-list | tr "||" "\012" | grep "tp://" /usr/bin/env /usr/bin/fetch -A http://www.gns.com.br/gimp-print/file-1 /usr/bin/env /usr/bin/fetch -A ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/file-1 /usr/bin/env /usr/bin/fetch -A http://prdownloads.sourceforge.net/ghostscript/file-2 /usr/bin/env /usr/bin/fetch -A ftp://ftp3.sourceforge.net/pub/sourceforge/ghostscript/file-2 /usr/bin/env /usr/bin/fetch -A http://ftp3.sourceforge.net/ghostscript/file-2 /usr/bin/env /usr/bin/fetch -A ftp://ftp1.sourceforge.net/pub/sourceforge/ghostscript/file-2 /usr/bin/env /usr/bin/fetch -A http://ftp1.sourceforge.net/ghostscript/file-2 /usr/bin/env /usr/bin/fetch -A ftp://ftp2.sourceforge.net/pub/sourceforge/ghostscript/file-2 /usr/bin/env /usr/bin/fetch -A http://ftp2.sourceforge.net/ghostscript/file-2 /usr/bin/env /usr/bin/fetch -A ftp://ftp.kddlabs.co.jp/sourceforge/ghostscript/file-2 /usr/bin/env /usr/bin/fetch -A http://telia.dl.sourceforge.net/ghostscript/file-2 /usr/bin/env /usr/bin/fetch -A ftp://ftp.chg.ru/pub/sourceforge/ghostscript/file-2 /usr/bin/env /usr/bin/fetch -A ftp://mirror.cs.wisc.edu/pub/mirrors/ghost/AFPL/gs703/file-2 /usr/bin/env /usr/bin/fetch -A ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/ghostscript/file-2 As you can see, both http://2/%SUBDIR%/ and http://something/%SUBDIR%/ belong to the default group $ make -f Makefile-enhance master-sites-default returns http://2/ghostscript/ http://something/ghostscript/ http://another/ http://2/heilo/ http://something/heilo/ http://another/ ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/ which is not what one would expect. What do I mean by that? The DEFAULT group is composed of http://2/%SUBDIR%/, http://something/%SUBDIR%/ and http://another/; in this particular order. The DEFAULT SUBDIR is composed of ${PORTNAME} and heilo. It is iterating SUBDIR against SITE (for subdir.... for site...). One would expect the ordering chosen by maintainer to be upheld since sites are listed in order of preference. That is, http://2/ghostscript/ http://2/heilo/ http://something/ghostscript/ http://something/heilo/ http://another/ ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/ it parses http://2/ using all SUBDIRs, then proceed to the 2nd entry doing the same to the 3rd... Iterating SITE against SUBDIR (for site .... for subdir...). Which most maintainers believed they were doing when they wrote their Makefiles (or they should have :). This is covered by patch (2). It is incremental, apply (1) then (2). (III) bsd.port.mk does not enforce the syntax of %SUBDIR%. All {MASTER,PATCH}_SITES entries are supposed to be terminated by a trailing /. However, this is not enforced by .mk rules. I wrote a little patch that enforces that. It is a simple modification to the %SUBDIR% regexp making it stricter. It should be applied incrementally, apply (1), (2) then (3). My advice, test trial patch (1) in bento 4-exp ASAP as it fixes a current error. After a test run, apply it. Try patch (2) as well but for a bit longer (though it SHOULD be correct); a few runs. I will submit more agressive MASTER_SITES_NN patches for some ports (ghostscript springs to mind) so that the trial runs in 4-exp are effective tests. Patch (2) is an excellent addition. Patch (3) should be applied at portmgr discretion. I believe it is "the right thing to have (TM)" something like that, though there could be a warning message when the user does not syntatically writes the correct thing. However, it can be argued that this sort of warning message belongs in portlint. Furthermore, if the maintainer/committer/porter did the "Right Thing(TM), he should have tested the port with fetch-list noticing that %SUBDIR% was not being replaced. Of course, I will be writing an entry to Porters Handbook. I have been holding off on that because of the bug fixed by patch (1). I was hoping that people wouldn't notice MASTER_SITES_NN before that. :-) >Fix: Check description and follow-up email. >Release-Note: >Audit-Trail: >Unformatted: To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-ports" in the body of the message