Date: Sun, 23 Sep 2001 05:52:24 -0300 From: "Mario Sergio Fujikawa Ferreira" <lioux@uol.com.br> To: FreeBSD-ports@FreeBSD.org Cc: portmgr@FreeBSD.org, Patrick Li <pat@databits.net>, fenner@FreeBSD.org, kris@FreeBSD.org, green@FreeBSD.org, knu@FreeBSD.org, julian@FreeBSD.org, petef@FreeBSD.org, cwasser@v-wave.com, sjh-cl@horan.net.au, john_m_cooper@yahoo.com, matt@ipperformance.com Subject: review plz MASTER_SITES_n (ala OpenBSD) patch for bsd.port.mk Message-ID: <20010923055224.A93855@exxodus.fedaykin.here>
next in thread | raw e-mail | index | archive | help
Hi, I've worked this week on this little patch. A little backgroup first. The patch can be found in http://people.freebsd.org/~lioux/patch-bsd.port.mk.gz A complete file for those having problems with the patch can be found in http://people.freebsd.org/~lioux/bsd.port.mk.gz OpenBSD has a neat feature, inside the DISTFILES area, files can be postfixed with :n where n can be [0-9] and denotes a group. Fo example: DISTFILES= file1:0 file2:1 Distfile file1 will be associated with variable MASTER_SITES0 instead of our common MASTER_SITES and files2 with MASTER_SITES1 This is a very neat feature which can decrease that endless search for the correct download site. Just picture 2 DISTFILES and 20 MASTER_SITES, the sites are slow as hell and (of course); while file2 is carried by all MASTER_SITES, file1 is only carried by MASTER_SITE number 20. Seems a waste to check all of them if the maintainer knew beforehand, does not it? Not a good start for that lovely weekend. :-) Now that you got the picture, just imagine more distfiles and more master_sites. I am sure that our "distfiles survey meister" would appreciate the network strain relieve this would bring. :) How this works for us? I took the OpenBSD concept and expanded it a bit further (the FreeBSD way (TM))... 1) {DIST,PATCH}FILES elements can be postfixed with :n where n is [^:,]+ n can be any alphanumerical string but we will limit it to [0-9a-zA-Z_]+ for now. We are not limited to a single numerical [0-9] as OpenBSD is. Also, n shouldn't be the word all (since it is being used internally in (b***) below) 2) there can be {MASTER,PATCH}_SITES_n where n has the same definition as in (1) 3) any non-postfixed {DIST,PATCH}FILES element will use the default {MASTER,PATCH}_SITES 4) if there is a postfixed {DIST,PATCH}FILES element with :n but there is no defined corresponding {MASTER,PATCH}_SITES_n, :n is considered part of the element. Also, aforementioned element will use default {MASTER,PATCH}_SITES as in (3) 5) any {MASTER,PATCH}_SITES_n which do not have a matching postfixed {DIST,PATCH}FILES element will be just ignored 6) {MASTER,PATCH}_SITE_SUBDIR_n applies to the respective {MASTER,PATCH}_SITES_n 7) make -V {DIST,PATCH}FILES will return the definition inside the Makefile without any changes, i.e., the :n postfixed will be there whether it is part of the element's name or a postfix 8) all {MASTER,PATCH}_SITES_n are sorted using the same MASTER_SITE_REGEX feature as regular {MASTER,PATCH}_SITES entries 9) all {MASTER,PATCH}_SITES_n behave as regular {MASTER,PATCH}_SITES regarding MASTER_SITE_BACKUP and MASTER_SITE_OVERRIDE, i.e., they respect them What changes for current working ports? What doesn't? a) All current ports remain the same. The feature code is only activated if there are {DIST,PATCH}FILES elements postfixed with :n with matching {MASTER,PATCH}_SITES_n. Therefore, even if there are postfixed elements, without matching {MASTER,PATCH}_SITES_n, the code considers the :n as part of the filename as refered by (4). This is important if one needs a {dist,patch}file with an ending :j, just do not define a {MASTER,PATCH}_SITES_j b) The port targets remain the same: checksum, makesum, patch, configure, build, ... All of them, with the obvious exceptions of do-fetch, fetch-list, master-sites and patch-sites - do-fetch: deploys the new grouping postfixed {DIST,PATCH}FILES with their matching {MASTER,PATCH}_SITES_n which use matching {MASTER,PATCH}_SITE_SUBDIR_n (just like regular {MASTER,PATCH}_SITES) - fetch-list: works like old fetch-list with the exception that it groups just like do-fetch - {master,patch}-sites: (for compatibility with older version) returns all {MASTER,PATCH}_SITES_n with the caveat that it lists as many both MASTER_SITE_BACKUP and MASTER_SITE_OVERRIDE as there are groups defined in {DIST,PATCH}FILES * using target {master,patch}-sites is preferred to directly checking {MASTER,PATCH}_SITES or any of {MASTER,PATCH}_SITES_n. Also, directly checking is not guaranteed to work in any future versions, the targets are ** (NEW) there are {master,patch}-sites-n targets which will list the respective {MASTER,PATCH}_SITES_n if there are groups defined in {DIST,PATCH}FILES, instead of listing all of them together *** (NEW) there are new targets {master,patch}-sites-all which do the work of {master,patch}-sites. Actually, {master,patch}-sites only call their respective {master,patch}-sites-all Doubts on the design of the patch: d1) Should we use {MASTER,PATCH}_SITES_n or {MASTER,PATCH}_SITESn as does OpenBSD (green we should use _n cleaner syntax) ? d2) Should {MASTER,PATCH}_SITE_SUBDIR_n or {MASTER,PATCH}_SITE_n_SUBDIR? I prefer the former, easier to spot d3) Should {master,patch}-sites do what is cited in (b) ? Or, should they only list the contents of {MASTER,PATCH}_SITES and leave the full listing for {master,patch}-sites-all ? I prefer the later, since it is uniform. I do not know how (if any) much impact will have this behavior change d4) Should make -V {DIST,PATCH}FILES hide the postfix :n ? I'd rather not, since this is an interesting information which cannot be accessed any other way by external means (unless we place it in another well-known variable) d5) Others doubts I am sure will be brought by reviewers of the code :) Ideas for improving the code: i1) As n can be [0-9a-zA-Z_]+ , what if the variables inside bsd.sites.mk were of the form MASTER_SITES_.* instead of MASTER_SITE_.*? For example, we could use MASTER_SITES_SOURCEFORGE by simply having n be SOURCEFORGE for example, very interesting. Or, I could add code to check for {MASTER,PATCH}_SITE_n as well. Just a thought. i2) This one is tempting, what if we could have several groups postfixed to a single {dist,patch}file? For instance, :m,n,o meaning it uses {MASTER,PATCH}_SITE_m, {MASTER,PATCH}_SITE_n and {MASTER,PATCH}_SITE_o. A very tempting thought. Here follows an example of the patch: file1 can be found in http://master1/ http://master2/ http://master3/subdir/ file2 can be found in http://master4/ http://master5/ http://master6/subdir_test/ file3:n can be found in http://master7/ file4:o can be found in http://master8/ MASTER_SITES= http://master7/ MASTER_SITES_today= http://master1/ http://master2/ http://master3/%SUBDIR%/ MASTER_SITES_yesterday= http://master4/ http://master5/ http://master6/%SUBDIR%/ MASTER_SITES_p= http://master8/ DISTFILES= file1:today file2:yesterday file3:n file4:o:p MASTER_SITE_SUBDIR_today= subdir MASTER_SITE_SUBDIR_yesterday= subdir_test This is MUCH MUCH MUCH more efficient then old: MASTER_SITES= http://master1/ http://master2/ \ http://master3/%SUBDIR%/ http://master4/ \ http://master5/ http://master6/%SUBDIR%/ \ http://master7/ http://master8/ DISTFILES= file1 file2 file3:n file4:o MASTER_SITE_SUBDIR= subdir subdir_test Of course, most ppl will say that the later is much easier on the eye then the former, yet the former carries intrinsic information about the port. And that cannot be dismissed quite easily. :) Not to mention the computational complexity decrease. As most things, this is optional and for those willing to use this new options, there can written much more powerful examples. I hope this gets added to our ports infrastructure. And, I hope that people have as much fun using it as I had writing it despite of the 5 badly slept days. Heheheheh Regards, --- Credits: fenner -> "Oh, please do it", And probably the one who will provide the most feedback given his ubermeister "portsurveyor status" green, kris -> discussed some stuff on early design such as {MASTER,PATCH}_SITE_n instead of {MASTER,PATCH}_SITEn knu -> incentive: "MASTER_SITES stuff really needs a revamp. It's too lame that it tries nonexisting combinations over again.." patli -> testing petef -> support and bugging me on this one (as well as threatening to write it first, healthy competition is a nice incentive :) Chris "TDF" Wasser <cwasser@v-wave.com> -> patience, code review, as well as providing the INVALUABLE sh expansion which replicates the following make expansion make -> SORTED_MASTER_SITES_CMD_TMP=${SORTED_MASTER_SITES_${select:S/^://}_CMD} sh -> eval SORTED_MASTER_SITES_CMD_TMP=\$${SORTED_MASTER_SITES_$${select#:}_CMD} Steve "wuf" Horan <sjh-cl@horan.net.au> -> for providing this little code which was the last bug on the patch. This was also submitted later by julian John Merryweather Cooper <john_m_cooper@yahoo.com> Matt Finlay <matt@ipperformance.com> producing what I wanted sh -c 'TDF="echo a && echo b";for i in `eval ${TDF}`; do echo "i:$i"; done' i:a i:b instead of sh -c 'TDF="echo a && echo b";for i in `${TDF}`; do echo "i:$i"; done' i:a i:&& i:echo i:b Anyone forgotten -> thanks too -- Mario S F Ferreira - UnB - Brazil - "I guess this is a signature." lioux at ( freebsd dot org | linf dot unb dot br ) flames to beloved devnull@someotherworldbeloworabove.org feature, n: a documented bug | bug, n: an undocumented feature To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-ports" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20010923055224.A93855>