Date: Sun, 20 May 2007 23:56:13 -0500 From: Stephen Montgomery-Smith <stephen@math.missouri.edu> To: Doug Barton <dougb@freebsd.org> Cc: ports@freebsd.org, Alexander Leidinger <Alexander@Leidinger.net> Subject: Re: Speeding up pkg_version and perhaps other port utilities Message-ID: <4651266D.8020802@math.missouri.edu> In-Reply-To: <20070520204725.S52261@math.missouri.edu> References: <464FD334.9040705@math.missouri.edu> <20070520082931.47fe8db9@deskjail> <4650C9C0.3080706@FreeBSD.org> <20070520204725.S52261@math.missouri.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
Stephen Montgomery-Smith wrote: > > > On Sun, 20 May 2007, Doug Barton wrote: > >> Alexander Leidinger wrote: >>> Quoting Stephen Montgomery-Smith <stephen@math.missouri.edu> (Sat, 19 >>> May 2007 23:48:52 -0500): >>> >>>> On my system, the program pkg_version can double its speed simply by >>>> replacing "make -V PKGNAME" by "make BEFOREPORTMK=yes -V PKGNAME" >> >> This is very cool! I've already added it to my devel version of >> portmaster in several places. I call make a LOT in portmaster, so I >> went through every invocation and added this wherever it worked (i.e., >> didn't cause a difference in (or absence of) output. >> > > However just because it works in a majority of cases doesn't mean that > it will work all the time. You just don't know what important thing > might be in AFTERPORTMK, which 99% of the time you never need. > > So I see two approaches: > > 1. Add more .if's to bsd.port.mk so that there is a more efficient > "make VARIABLEONLY=yes -V variable-name" that does all the variables, > but doesn't calculate any of the targets - this might make things a bit > faster, or it might not; I don't think this will speed things up. I hacked into the code for make, putting conditionals so that when the "-V" flag was set, that make wouldn't bother to calculate targets or dependencies or creations, but only variables. > 2. Profile bsd make and see if there are any bottlenecks. I bet make > was never designed for speed in these kinds of situations. But this > would be a long term project, albeit definitely worth doing. It looks to me like the variables are stored as a linear list in the "make" program. Thus if you have something like 500 variables (e.g. try a "make -d g1 -V XX | grep = | wc -l" in a port) it is going to take quite some time to search through all the variables to find one. This is especially a problem for variable assignments using "?=" which first has to search to see if the variable is already defined. And there are a lot of "?=" in bsd.port.mk, 298 of them. I haven't done any profiling yet, but I bet that this is what is taking up all the time. Probably the way to solve this is to rewrite src/usr.bin/make/var.c so that it uses some more sophisticated way of storing the variables - maybe Berkeley databases is the way to go. I almost get the feeling that if make were completely rewritten in perl that it would go faster!
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4651266D.8020802>