Date: Sun, 23 Sep 2001 15:25:05 +0400 From: Maxim Sobolev <sobomax@mail.ru> To: lioux@uol.com.br, FreeBSD-ports@FreeBSD.ORG Cc: portmgr@FreeBSD.ORG, pat@databits.net, fenner@FreeBSD.ORG, kris@FreeBSD.ORG, green@FreeBSD.ORG, knu@FreeBSD.ORG, julian@FreeBSD.ORG, petef@FreeBSD.ORG, cwasser@v-wave.com, sjh-cl@horan.net.au, john_m_cooper@yahoo.com, matt@ipperformance.com Subject: Re: review plz MASTER_SITES_n (ala OpenBSD) patch for bsd.port.mk Message-ID: <E15l7Nf-00072f-00@mx10.port.ru> In-Reply-To: <20010923055224.A93855@exxodus.fedaykin.here>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Mario, Thank you for a great work, it is really step in the right direction. I have not reviewed the implementation yet, but the writeup is really interesting and well thought out. At the same time, I am not very happy with the resulting Makefile bloat, and have a quick idea, which may or may not be a better way out: Why not in addition to postfixes in DISTFILES introduce the same postfixes in MASTER_SITES and MASTER_SITE_SUBDIR? This would allow to avoid introducing MASTER_SITES_foo, turning your example below into: MASTER_SITES=3D=09http://master7/:n \ =09=09http://master1/:t http://master2/:t http://master3/%SUBDIR%/:t \ =09=09http://master4/:y http://master5/:y http://master6/%SUBDIR%/:y \ =09=09http://master8/:p DISTFILES=3D=09file1:t file2:y file3:n file4:o:p MASTER_SITE_SUBDIR=3D=09subdir:t subdir_test:y In the case when you have to use some of MASTER_SITE_* from bsd.sites.mk you can use ${MASTER_SITE_FOO:S/$/:yourpostfix} to add appropriate postfixe= s into all members of MASTER_SITE_FOO. I would like to hear what do you and others think about it. -Maxim On Sun, 23 Sep 2001 05:52:24 -0300, Mario Sergio Fujikawa Ferreira wrote: > Hi, > = > =09I've worked this week on this little patch. A little backgroup > first. The patch can be found in = > = > http://people.freebsd.org/~lioux/patch-bsd.port.mk.gz > = > A complete file for those having problems with the patch can be found > in > = > http://people.freebsd.org/~lioux/bsd.port.mk.gz > = > =09OpenBSD has a neat feature, inside the DISTFILES area, files > can be postfixed with :n where n can be [0-9] and denotes a group. Fo > example: > = > DISTFILES=3D=09file1:0 file2:1 > = > =09Distfile file1 will be associated with variable MASTER_SITES0 instead > of our common MASTER_SITES and files2 with MASTER_SITES1 > =09This is a very neat feature which can decrease that endless > search for the correct download site. > =09Just picture 2 DISTFILES and 20 MASTER_SITES, the sites > are slow as hell and (of course); while file2 is carried by all > MASTER_SITES, file1 is only carried by MASTER_SITE number 20. Seems > a waste to check all of them if the maintainer knew beforehand, > does not it? Not a good start for that lovely weekend. :-) > =09Now that you got the picture, just imagine more distfiles > and more master_sites. I am sure that our "distfiles survey meister" > would appreciate the network strain relieve this would bring. :) > = > =09How this works for us? I took the OpenBSD concept and > expanded it a bit further (the FreeBSD way (TM))... > = > =091) {DIST,PATCH}FILES elements can be postfixed with :n > =09 where n is [^:,]+ n can be any alphanumerical string > =09 but we will limit it to [0-9a-zA-Z_]+ for now. We are > =09 not limited to a single numerical [0-9] as OpenBSD is. > =09 Also, n shouldn't be the word all (since it is being used > =09 internally in (b***) below) > = > =092) there can be {MASTER,PATCH}_SITES_n where n has the same > =09 definition as in (1) > = > =093) any non-postfixed {DIST,PATCH}FILES element will > =09 use the default {MASTER,PATCH}_SITES > = > =094) if there is a postfixed {DIST,PATCH}FILES element with > =09 :n but there is no defined corresponding {MASTER,PATCH}_SITES_n, > =09 :n is considered part of the element. Also, aforementioned > =09 element will use default {MASTER,PATCH}_SITES as in (3) > = > =095) any {MASTER,PATCH}_SITES_n which do not have a matching > =09 postfixed {DIST,PATCH}FILES element will be just ignored > = > =096) {MASTER,PATCH}_SITE_SUBDIR_n applies to the respective > =09 {MASTER,PATCH}_SITES_n > = > =097) make -V {DIST,PATCH}FILES will return the definition > =09 inside the Makefile without any changes, i.e., the :n > =09 postfixed will be there whether it is part of the > =09 element's name or a postfix > = > =098) all {MASTER,PATCH}_SITES_n are sorted using the same > =09 MASTER_SITE_REGEX feature as regular {MASTER,PATCH}_SITES > =09 entries > = > =099) all {MASTER,PATCH}_SITES_n behave as regular > =09 {MASTER,PATCH}_SITES regarding MASTER_SITE_BACKUP and > =09 MASTER_SITE_OVERRIDE, i.e., they respect them > = > = > =09What changes for current working ports? What doesn't? > = > =09a) All current ports remain the same. The feature code is > =09 only activated if there are {DIST,PATCH}FILES elements > =09 postfixed with :n with matching {MASTER,PATCH}_SITES_n. > =09 Therefore, even if there are postfixed elements, without > =09 matching {MASTER,PATCH}_SITES_n, the code considers the > =09 :n as part of the filename as refered by (4). This is > =09 important if one needs a {dist,patch}file with an ending > =09 :j, just do not define a {MASTER,PATCH}_SITES_j > = > =09b) The port targets remain the same: checksum, makesum, > =09 patch, configure, build, ... All of them, with the > =09 obvious exceptions of do-fetch, fetch-list, master-sites > =09 and patch-sites > = > =09=09- do-fetch: deploys the new grouping postfixed > =09=09{DIST,PATCH}FILES with their matching > =09=09{MASTER,PATCH}_SITES_n which use matching > =09=09{MASTER,PATCH}_SITE_SUBDIR_n (just like regular > =09=09{MASTER,PATCH}_SITES) > = > =09=09- fetch-list: works like old fetch-list with the > =09=09exception that it groups just like do-fetch > = > =09=09- {master,patch}-sites: (for compatibility with older > =09=09version) returns all {MASTER,PATCH}_SITES_n with > =09=09the caveat that it lists as many both MASTER_SITE_BACKUP > =09=09and MASTER_SITE_OVERRIDE as there are groups defined > =09=09in {DIST,PATCH}FILES > = > =09=09* using target {master,patch}-sites is preferred to > =09=09directly checking {MASTER,PATCH}_SITES or any of > =09=09{MASTER,PATCH}_SITES_n. Also, directly checking is not > =09=09guaranteed to work in any future versions, the targets > =09=09are > = > =09=09** (NEW) there are {master,patch}-sites-n targets > =09=09which will list the respective {MASTER,PATCH}_SITES_n > =09=09if there are groups defined in {DIST,PATCH}FILES, > =09=09instead of listing all of them together > = > =09=09*** (NEW) there are new targets {master,patch}-sites-all > =09=09which do the work of {master,patch}-sites. Actually, > =09=09{master,patch}-sites only call their respective > =09=09{master,patch}-sites-all > = > = > =09Doubts on the design of the patch: > = > =09d1) Should we use {MASTER,PATCH}_SITES_n or > =09 {MASTER,PATCH}_SITESn as does OpenBSD (green we should > =09 use _n cleaner syntax) ? > = > =09d2) Should {MASTER,PATCH}_SITE_SUBDIR_n or > =09 {MASTER,PATCH}_SITE_n_SUBDIR? I prefer the former, > =09 easier to spot > = > =09d3) Should {master,patch}-sites do what is cited in (b) ? > =09 Or, should they only list the contents of {MASTER,PATCH}_SITES > =09 and leave the full listing for {master,patch}-sites-all ? > =09 I prefer the later, since it is uniform. I do not > =09 know how (if any) much impact will have this behavior > =09 change > = > =09d4) Should make -V {DIST,PATCH}FILES hide the postfix :n ? I'd > =09 rather not, since this is an interesting information > =09 which cannot be accessed any other way by external > =09 means (unless we place it in another well-known variable) > = > =09d5) Others doubts I am sure will be brought by reviewers of > =09 the code :) > = > = > =09Ideas for improving the code: > = > =09i1) As n can be [0-9a-zA-Z_]+ , what if the variables inside > =09 bsd.sites.mk were of the form MASTER_SITES_.* instead > =09 of MASTER_SITE_.*? For example, we could use > =09 MASTER_SITES_SOURCEFORGE by simply having n be SOURCEFORGE > =09 for example, very interesting. Or, I could add code to > =09 check for {MASTER,PATCH}_SITE_n as well. Just a thought. > = > =09i2) This one is tempting, what if we could have several > =09 groups postfixed to a single {dist,patch}file? For > =09 instance, :m,n,o meaning it uses {MASTER,PATCH}_SITE_m, > =09 {MASTER,PATCH}_SITE_n and {MASTER,PATCH}_SITE_o. A very > =09 tempting thought. > = > = > =09Here follows an example of the patch: > = > file1 can be found in http://master1/ http://master2/ http://master3/subd= ir/ > file2 can be found in http://master4/ http://master5/ http://master6/subd= ir_test/ > file3:n can be found in http://master7/ > file4:o can be found in http://master8/ > = > MASTER_SITES=3D=09http://master7/ > MASTER_SITES_today=3D=09http://master1/ http://master2/ http://master3/%S= UBDIR%/ > MASTER_SITES_yesterday=3D=09http://master4/ http://master5/ http://master= 6/%SUBDIR%/ > MASTER_SITES_p=3D=09http://master8/ > DISTFILES=3D=09file1:today file2:yesterday file3:n file4:o:p > MASTER_SITE_SUBDIR_today=3D=09subdir > MASTER_SITE_SUBDIR_yesterday=3D=09subdir_test > = > =09This is MUCH MUCH MUCH more efficient then old: > = > MASTER_SITES=3D=09http://master1/ http://master2/ \ > =09=09http://master3/%SUBDIR%/ http://master4/ \ > =09=09http://master5/ http://master6/%SUBDIR%/ \ > =09=09http://master7/ http://master8/ > DISTFILES=3D=09file1 file2 file3:n file4:o > MASTER_SITE_SUBDIR=3D=09subdir subdir_test > = > =09Of course, most ppl will say that the later is much easier > on the eye then the former, yet the former carries intrinsic > information about the port. And that cannot be dismissed quite > easily. :) Not to mention the computational complexity decrease. > =09As most things, this is optional and for those willing to use > this new options, there can written much more powerful examples. > =09I hope this gets added to our ports infrastructure. And, I > hope that people have as much fun using it as I had writing it despite > of the 5 badly slept days. Heheheheh > = > =09Regards, > = > --- > = > =09Credits: > = > =09fenner -> "Oh, please do it", And probably the one who will > =09provide the most feedback given his ubermeister "portsurveyor > =09status" > = > =09green, kris -> discussed some stuff on early design such as = > =09{MASTER,PATCH}_SITE_n instead of {MASTER,PATCH}_SITEn > = > =09knu -> incentive: "MASTER_SITES stuff really needs a revamp. > =09It's too lame that it tries nonexisting combinations over > =09again.." > = > =09patli -> testing > = > =09petef -> support and bugging me on this one (as well as > =09threatening to write it first, healthy competition is a > =09nice incentive :) > = > =09Chris "TDF" Wasser <cwasser@v-wave.com> -> patience, code > =09review, as well as providing the INVALUABLE sh expansion > =09which replicates the following make expansion > = > =09=09make -> SORTED_MASTER_SITES_CMD_TMP=3D${SORTED_MASTER_SITES_${selec= t:S/^://}_CMD} > =09=09sh -> eval SORTED_MASTER_SITES_CMD_TMP=3D\$${SORTED_MASTER_SITES_$$= {select#:}_CMD} > = > =09Steve "wuf" Horan <sjh-cl@horan.net.au> -> for providing this > =09little code which was the last bug on the patch. This was also > =09submitted later by > =09=09julian > =09=09John Merryweather Cooper <john_m_cooper@yahoo.com>=09 > =09=09Matt Finlay <matt@ipperformance.com> > = > producing what I wanted > = > sh -c 'TDF=3D"echo a && echo b";for i in `eval ${TDF}`; do echo "i:$i"; > done' > i:a > i:b > = > instead of > = > sh -c 'TDF=3D"echo a && echo b";for i in `${TDF}`; do echo "i:$i"; done' > i:a > i:&& > i:echo > i:b > = > =09Anyone forgotten -> thanks too > = > -- = > Mario S F Ferreira - UnB - Brazil - "I guess this is a signature." > lioux at ( freebsd dot org | linf dot unb dot br ) > flames to beloved devnull@someotherworldbeloworabove.org > feature, n: a documented bug | bug, n: an undocumented feature > = > = > = To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-ports" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?E15l7Nf-00072f-00>