Date: Mon, 28 May 2007 13:04:23 +0200 From: Hartmut Brandt <hartmut.brandt@dlr.de> To: Matthew Seaman <m.seaman@infracaninophile.co.uk> Cc: ports@freebsd.org, Stephen Montgomery-Smith <stephen@math.missouri.edu>, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports Message-ID: <465AB737.1030708@dlr.de> In-Reply-To: <465A94C0.2080301@infracaninophile.co.uk> References: <4659EF80.70100@math.missouri.edu> <465A94C0.2080301@infracaninophile.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
Matthew Seaman wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Stephen Montgomery-Smith wrote: >> I have been thinking a lot about looking for speed increases for "make >> index" and pkg_version and things like that. So for example, in >> pkg_version, it calls "make -V PKGNAME" for every installed package. Now >> "make -V PKGNAME" should be a speedy operation, but the make has to load >> in and analyze bsd.port.mk, a quite complicated file with about 200,000 >> characters in it, when all it is needing to do is to figure out the >> value of the variable PKGNAME. > > pkg_version is one thing -- but to build the INDEX you need to extract > at least the values of the following variables: > > PKGNAME > .CURDIR > PREFIX > COMMENT > DESCR > MAINTAINER > CATEGORIES > EXTRACT_DEPENDS > PATCH_DEPENDS > FETCH_DEPENDS > BUILD_DEPENDS > RUN_DEPENDS > LIB_DEPENDS > > Plus you need to grep in the referenced pkg-descr file for any WWW > links. I also extract the values of: > > MASTER_PORT > .MAKEFILE_LIST > SUBDIR > > for my FreeBSD::Portindex stuff. > > Trouble is, by the time you've extracted all that lot, you have pretty > much done the same level of variable processing as you would were you > actually going to build the port. > > One thing that would speed up this process would be a make option > to just do parsing of the Makefile and variable expansion, without > calling stat(2) on all the various sources and dependencies involved. > > For instance: > > happy-idiot-talk:...ports/databases/mysql-connector-java:% truss make -V PKGNAME |& grep stat | wc -l > 49 > > It is quite instructive to see what files make(1) touches while doing > that. At least half of them are irrelevant if all make(1) is going to > do is print out the values of some variables. Multiply that by 17,000 > and it adds up to a big waste of effort. Are you sure? A good deal of the stats() is make poking around for the make infrastructure. This should be in the cache. And there are a couple of stats for the *done* files that might be avoided by doing something in the ports infrastructure. But as I already said in my previous mail: numbers, please, no guessing. harti
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?465AB737.1030708>