Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 29 May 2007 20:02:56 +1000
From:      Peter Jeremy <peterjeremy@optushome.com.au>
To:        Stephen Montgomery-Smith <stephen@math.missouri.edu>
Cc:        ports@freebsd.org, hackers@freebsd.org
Subject:   Re: Looking for speed increases in "make index" and pkg_version for ports
Message-ID:  <20070529100255.GA70055@turion.vk2pj.dyndns.org>
In-Reply-To: <4659EF80.70100@math.missouri.edu>
References:  <4659EF80.70100@math.missouri.edu>

next in thread | previous in thread | raw e-mail | index | archive | help

--+HP7ph2BbKc20aGI
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2007-May-27 15:52:16 -0500, Stephen Montgomery-Smith <stephen@math.misso=
uri.edu> wrote:
> I have been thinking a lot about looking for speed increases for "make=20
> index" and pkg_version and things like that.  So for example, in=20
> pkg_version, it calls "make -V PKGNAME" for every installed package. Now=
=20
> "make -V PKGNAME" should be a speedy operation, but the make has to load =
in=20
> and analyze bsd.port.mk, a quite complicated file with about 200,000=20
> characters in it, when all it is needing to do is to figure out the value=
 of=20
> the variable PKGNAME.

This would be trivial if some packages didn't change names depending
on options and what was installed.  I agree that parsing a 210KB file
17,000 times is not going to be fast.  Especially since some ports
include bsd.ports.mk multiple times...

> I suggest rewriting "make" so that variables are only evaluated on a "nee=
d=20
> to know" basis.

This sounds like a good idea but I suspect it's not going to be
feasible.  The biggest problem I see is that the make language allows
variables to be expanded either when they are assigned or when they
are referenced.  If a variable expansion is delayed from the
assignment to the first use then the expansion must be performed using
the state of make as it was when the variable was assigned.  The cost
of keeping this state probably exceeds the cost of actually evaluating
the variable.

>  So, for example, if all we need to know is PKGNAME, there=20
> is no need to evaluate, for example, _RUN_LIB_DEPENDS, unless the writer =
of=20
> that particular port has done something like having PORTNAME depend on th=
e=20
> value of _RUN_LIB_DEPENDS.

Rather than trying to develop a tool that can quickly expand PKGNAME
irrespective of what convoluted code the author has used, how about a
partial solution?  For most ports, PKGNAME depends solely on 3 or 4
variables that are statically defined in the port Makefile.  The
obvious solution would seem to be to develop a script that can handle
the easy cases itself and punt the difficult cases back to make.  The
definition of 'easy' can be adjusted over time to increase performance.
This approach would seem to have a relatively low bar to entry whilst
offering good effort/performance tradeoff at the low end.

The various depends lists would seem amenable to the same approach -
though the entry level tool will have far lower coverage due to the
extensive use of USE_GNOME=3D... and similar 'macro'-style constructs.

--=20
Peter Jeremy

--+HP7ph2BbKc20aGI
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (FreeBSD)

iD8DBQFGW/pP/opHv/APuIcRArw9AJ9WsRYjwIrgxdypFLGTK4G7JnQKPQCeNJvl
QvVG9B881APC8Ap2MDhjbyA=
=M39M
-----END PGP SIGNATURE-----

--+HP7ph2BbKc20aGI--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070529100255.GA70055>