Date: 19 Mar 2002 04:03:40 -0000 From: Mario Sergio Fujikawa Ferreira <lioux@FreeBSD.org> To: FreeBSD-gnats-submit@FreeBSD.org Subject: ports/36078: Fix MASTER_SITES_NN recursive bug Message-ID: <20020319040340.52400.qmail@exxodus.fedaykin.here>
next in thread | raw e-mail | index | archive | help
>Number: 36078
>Category: ports
>Synopsis: Fix MASTER_SITES_NN recursive bug
>Confidential: no
>Severity: non-critical
>Priority: low
>Responsible: freebsd-ports
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Mon Mar 18 20:10:01 PST 2002
>Closed-Date:
>Last-Modified:
>Originator: Mario Sergio Fujikawa Ferreira
>Release: FreeBSD 4.5-STABLE i386
>Organization:
>Environment:
System: FreeBSD exxodus.fedaykin.here 4.5-STABLE FreeBSD 4.5-STABLE #2: Thu Mar 7 23:13:19 BRT 2002 root@exxodus.fedaykin.here:/usr/obj/usr/src/sys/LIOUX i386
Up to date ports tree CVSup updated as of 2002/03/18 against
cvs.freebsd.org previous to all testing.
>Description:
Fix a fetch bug introduced in bsd.port.mk
rev 1.396 by my patch MASTER_SITES_NN
Pointy hat goes to: lioux
Also, "correct" historical ordering behavior
of a fetch list constructed by {MASTER,PATCH}_SITES_SUBDIR
>How-To-Repeat:
Hi,
Just fixing some bugs. And, a little enhancement. :-)
This is all regarding the MASTER_SITES_NN patch introduced in
bsd.port.mk rev 1.396
A list of the patches:
1) patch-bsd.port.mk-fix
2) patch-bsd.port.mk-enhance
3) patch-bsd.port.mk-strict
A list of the test Makefiles (packed in a uuencode tgz due
to size) All Makefile tests are run as 'make DISTDIR=/tmp -f
fill_appropriate_makefile fetch-list'
A) Makefile-qmail
B) Makefile-ghostscript
C) Makefile-enhance
All these files are coming in a follow email since send-pr
is not exactly good with files.
(I) First, and foremost, lets begin fixing a bug introduced
by me.
If one tries Makefile either (A) or (B) with an up to date
ports tree, one will notice that several screens will scroll before
it finishes. For example,
$ make DISTDIR=. -f Makefile-ghostscript fetch-list | tr "||" "\012" | grep "tp://" | sort
will show one the fetch commands while
$ make DISTDIR=. -f Makefile-ghostscript fetch-list | tr "||" "\012" | grep "tp://" | sort | wc
should return one the number of fetch commands: 995; which is absurd.
If ones inspects the line without '| wc', one will notice duplicated
lines. Lets remove the duplicated lines
$ make DISTDIR=. -f Makefile-ghostscript fetch-list | tr "||" "\012" | grep "tp://" | sort | uniq
and
$ make DISTDIR=. -f Makefile-ghostscript fetch-list | tr "||" "\012" | grep "tp://" | sort | uniq | wc
which returns a reasonable value of 104 fetch commands.
Similarly,
$ make DISTDIR=. -f Makefile-qmail fetch-list | tr "||" "\012" | grep "tp://" | sort | wc
returns 206 while
$ make DISTDIR=. -f Makefile-qmail fetch-list | tr "||" "\012" | grep "tp://" | sort | uniq | wc
returns 45.
Thus, one can conclude that either a variable is being used
without being cleaned (initialiazed with no content) or the algorithm
is doing something utterly wrong.
Close inspection shows that it's a simple bug, a variable
was being re-used without being cleaned. Design had it that it was
cleaned but the version sent to portmgr for commit had this bug
(either confusion on my part or simple mistake). Anyway, patch (1)
fixes this.
This patch levels us with previous behavior of fetch-* targets.
(II) Well, patch (1) brought us to the expected behavior of
bsd.port.mk rev 1.395 previous to MASTER_SITES_NN patch. However,
previous behavior had a little glitch. All following steps assume
that patch (1) has been applied.
Makefile (C) is excellent to exhibit the glitch.
MASTER_SITES= http://www.gns.com.br/%SUBDIR%/:GIMP_PRINT,HPINKJET,RENO \
http://www.dod.com/:RENO http://2/%SUBDIR%/ \
http://something/%SUBDIR%/ http://another/
MASTER_SITE_SUBDIR= ${PORTNAME} gimp-print/:GIMP_PRINT \
hpinkjet/:HPINKJET heilo/:DEFAULT
DISTFILES= file-1:GIMP_PRINT file-2
$ make DISTDIR=. -f Makefile-enhance fetch-list | tr "||" "\012" | grep "tp://"
/usr/bin/env /usr/bin/fetch -A http://www.gns.com.br/gimp-print/file-1
/usr/bin/env /usr/bin/fetch -A ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/file-1
/usr/bin/env /usr/bin/fetch -A http://prdownloads.sourceforge.net/ghostscript/file-2
/usr/bin/env /usr/bin/fetch -A ftp://ftp3.sourceforge.net/pub/sourceforge/ghostscript/file-2
/usr/bin/env /usr/bin/fetch -A http://ftp3.sourceforge.net/ghostscript/file-2
/usr/bin/env /usr/bin/fetch -A ftp://ftp1.sourceforge.net/pub/sourceforge/ghostscript/file-2
/usr/bin/env /usr/bin/fetch -A http://ftp1.sourceforge.net/ghostscript/file-2
/usr/bin/env /usr/bin/fetch -A ftp://ftp2.sourceforge.net/pub/sourceforge/ghostscript/file-2
/usr/bin/env /usr/bin/fetch -A http://ftp2.sourceforge.net/ghostscript/file-2
/usr/bin/env /usr/bin/fetch -A ftp://ftp.kddlabs.co.jp/sourceforge/ghostscript/file-2
/usr/bin/env /usr/bin/fetch -A http://telia.dl.sourceforge.net/ghostscript/file-2
/usr/bin/env /usr/bin/fetch -A ftp://ftp.chg.ru/pub/sourceforge/ghostscript/file-2
/usr/bin/env /usr/bin/fetch -A ftp://mirror.cs.wisc.edu/pub/mirrors/ghost/AFPL/gs703/file-2
/usr/bin/env /usr/bin/fetch -A ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/ghostscript/file-2
As you can see, both http://2/%SUBDIR%/ and http://something/%SUBDIR%/ belong
to the default group
$ make -f Makefile-enhance master-sites-default
returns
http://2/ghostscript/ http://something/ghostscript/ http://another/ http://2/heilo/ http://something/heilo/ http://another/ ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/
which is not what one would expect. What do I mean by that?
The DEFAULT group is composed of http://2/%SUBDIR%/,
http://something/%SUBDIR%/ and http://another/; in this particular
order. The DEFAULT SUBDIR is composed of ${PORTNAME} and heilo. It
is iterating SUBDIR against SITE (for subdir.... for site...). One
would expect the ordering chosen by maintainer to be upheld since
sites are listed in order of preference. That is,
http://2/ghostscript/ http://2/heilo/ http://something/ghostscript/ http://something/heilo/ http://another/ ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/
it parses http://2/ using all SUBDIRs, then proceed to the 2nd entry
doing the same to the 3rd... Iterating SITE against SUBDIR (for
site .... for subdir...). Which most maintainers believed they were
doing when they wrote their Makefiles (or they should have :).
This is covered by patch (2). It is incremental, apply (1) then (2).
(III) bsd.port.mk does not enforce the syntax of %SUBDIR%. All
{MASTER,PATCH}_SITES entries are supposed to be terminated by a
trailing /. However, this is not enforced by .mk rules. I wrote a
little patch that enforces that. It is a simple modification to
the %SUBDIR% regexp making it stricter.
It should be applied incrementally, apply (1), (2) then
(3).
My advice, test trial patch (1) in bento 4-exp ASAP as it
fixes a current error. After a test run, apply it. Try patch (2)
as well but for a bit longer (though it SHOULD be correct); a few
runs. I will submit more agressive MASTER_SITES_NN patches for some
ports (ghostscript springs to mind) so that the trial runs in 4-exp
are effective tests. Patch (2) is an excellent addition.
Patch (3) should be applied at portmgr discretion. I believe
it is "the right thing to have (TM)" something like that, though
there could be a warning message when the user does not syntatically
writes the correct thing. However, it can be argued that this sort
of warning message belongs in portlint. Furthermore, if the
maintainer/committer/porter did the "Right Thing(TM), he should
have tested the port with fetch-list noticing that %SUBDIR% was not
being replaced.
Of course, I will be writing an entry to Porters Handbook.
I have been holding off on that because of the bug fixed by patch
(1). I was hoping that people wouldn't notice MASTER_SITES_NN before
that. :-)
>Fix:
Check description and follow-up email.
>Release-Note:
>Audit-Trail:
>Unformatted:
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-ports" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20020319040340.52400.qmail>