From owner-freebsd-ports@FreeBSD.ORG Sun May 27 22:30:33 2007 Return-Path: X-Original-To: ports@freebsd.org Delivered-To: freebsd-ports@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0A77616A57E for ; Sun, 27 May 2007 22:30:32 +0000 (UTC) (envelope-from michel@lpthe.jussieu.fr) Received: from shiva.jussieu.fr (shiva.jussieu.fr [134.157.0.129]) by mx1.freebsd.org (Postfix) with ESMTP id 595CD13C4B7 for ; Sun, 27 May 2007 22:30:32 +0000 (UTC) (envelope-from michel@lpthe.jussieu.fr) Received: from parthe.lpthe.jussieu.fr (parthe.lpthe.jussieu.fr [134.157.10.1]) by shiva.jussieu.fr (8.13.8/jtpda-5.4) with ESMTP id l4RMFUHF045047 ; Mon, 28 May 2007 00:15:31 +0200 (CEST) X-Ids: 164 Received: by parthe.lpthe.jussieu.fr (Postfix, from userid 10096) id DA39FBF69A; Mon, 28 May 2007 00:15:29 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on parthe.lpthe.jussieu.fr X-Spam-Level: X-Spam-Status: No, score=-3.2 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.8 Received: from niobe.lpthe.jussieu.fr (niobe.lpthe.jussieu.fr [134.157.10.41]) by parthe.lpthe.jussieu.fr (Postfix) with ESMTP id 105FEBF5A0; Mon, 28 May 2007 00:15:29 +0200 (CEST) Received: by niobe.lpthe.jussieu.fr (Postfix, from userid 2005) id E91BC80; Mon, 28 May 2007 00:15:28 +0200 (CEST) Date: Mon, 28 May 2007 00:15:28 +0200 From: Michel Talon To: freebsd-hackers@freebsd.org Message-ID: <20070527221528.GA19603@lpthe.jussieu.fr> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.2i X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-3.0 (shiva.jussieu.fr [134.157.0.164]); Mon, 28 May 2007 00:15:31 +0200 (CEST) X-Virus-Scanned: ClamAV 0.88.7/3305/Sun May 27 20:19:57 2007 on shiva.jussieu.fr X-Virus-Status: Clean X-Miltered: at shiva.jussieu.fr with ID 465A0302.004 by Joe's j-chkmail (http://j-chkmail.ensmp.fr)! Cc: ports@freebsd.org Subject: Re: Looking for speed increases in "make index" X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 May 2007 22:30:33 -0000 Stephen Montgomery-Smith said: > I suggest rewriting "make" so that variables are only evaluated on a > "need to know" basis. .... > or "I have tried to do this." Of course a lot of people have thinked about it, and quickly realized that it was not going to work. In the bsd.ports.mk, evaluation of one variable may be dependent on some conditional, which may itself be dependant on some other variable, appearing as some target. This constantly happens in bsd.ports.mk. If you think about that, you convince yourself that a "reduced make" needs to understand targets, needs to understand conditionals, and needs to evaluate not only the variable under consideration, but but possibly any other. In other words, you need a full blown make. To gain some performance, a first idea would be to simplify bsd.ports.mk. I am convinced that a substantial part of the 4000 lines are historical crap which serve no useful purpose. There are tons of variables who have probably purely anecdotical interest. There are targets which could be happily suppressed. Of course, due to the complexity of bsd.ports.mk, rewriting it is certainly not an easy task. There is a freebsd port whose aim is to rewrite it, i don't know how far they are. The NetBSD people have completely rewritten the system, i have no idea if the new one is faster. The OpenBSD people have also rewritten bsd.ports.mk, with apparently much faster makefile. A second idea would be to multithread make, since modern machines will have a lot of cores. At present make -j forks submakes and waits for their completion. This doesn't help for the problem at hand. However, multithreading the execution of a single makefile is certainly not trivial due to the interdependencies. Anyways, one of the things which cannot be a big factor is reading bsd.ports.mk from hard disk. It is certainly cached in memory when you run make index. On the other hand it is so big that it probably doesn't fit in cache, or probably only fits in caches of machines generously endowed. I have remarked that the difference of execution speed of make index between machines of similar speed but very different cache size (i am thinking Pentium 4 versus Core 2 Duo) is striking. It is less than 10 minutes on the big cache machine versus 30 minutes on the small cache one. If the cache effect is indeed dominant, then reducing the size of bsd.ports.mk by all possible means would be very beneficial. By the way an alternative system would be to use something other than make to do the job. This is what the Gentoo people are doing, using first a python based system, and now a C++ one (paludis). Here also i don't have any idea if it is faster. -- Michel TALON