From owner-freebsd-ports@FreeBSD.ORG Wed May 16 14:29:48 2007 Return-Path: X-Original-To: freebsd-ports@freebsd.org Delivered-To: freebsd-ports@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 25BB016A403; Wed, 16 May 2007 14:29:48 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from sccmmhc91.asp.att.net (sccmmhc91.asp.att.net [204.127.203.211]) by mx1.freebsd.org (Postfix) with ESMTP id BA22F13C480; Wed, 16 May 2007 14:29:47 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from laptop2.montlan (12-216-253-187.client.mchsi.com[12.216.253.187]) by sccmmhc91.asp.att.net (sccmmhc91) with ESMTP id <20070516142946m9100rqlu5e>; Wed, 16 May 2007 14:29:47 +0000 Message-ID: <464B155A.1080105@math.missouri.edu> Date: Wed, 16 May 2007 09:29:46 -0500 From: Stephen Montgomery-Smith User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.8.1.2) Gecko/20070505 SeaMonkey/1.1.1 MIME-Version: 1.0 To: "[LoN]Kamikaze" References: <464597C6.3030406@gmx.de> <20070512174011.GA22526@xor.obsecurity.org> <4645FF71.60100@gmx.de> <20070512175824.GA23103@xor.obsecurity.org> <20070512133054.B5588@math.missouri.edu> <4646193E.5040503@gmx.de> <464665D5.1090509@math.missouri.edu> <464676CC.2000203@math.missouri.edu> <46469BF5.9030603@math.missouri.edu> <1179058696.13237.3.camel@ikaros.oook.cz> <20070513101332.L82202@math.missouri.edu> <46479215.1080401@math.missouri.edu> <4647C4FB.9050607@math.missouri.edu> <4647F182.6000402@gmx.de> <46487491.6020806@math.missouri.edu> <46488168.3030300@math.missouri.edu> In-Reply-To: <46488168.3030300@math.missouri.edu> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: Pav Lucistnik , freebsd-ports@freebsd.org, Kris Kennaway Subject: Re: Time to abandon recursive pulling of dependencies? X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 May 2007 14:29:48 -0000 Stephen Montgomery-Smith wrote: > Stephen Montgomery-Smith wrote: >> [LoN]Kamikaze wrote: >>> Stephen Montgomery-Smith wrote: >>>> Basically I think we are stuck on making "make package-depends" go any >>>> faster. >>>> >>>> However I do think that the modifications I made to pkg_create go a >>>> very >>>> significant way to solving the problem of registration taking so >>>> very long. >>>> >>>> Stephen >>>> >>> >>> You are right about invoking make being the problem. However, I was >>> able to >>> increase the performance of package-depends by 20 to 30 percent, by >>> avoiding >>> recursive make calls and moving the recursion into the stack of the >>> shell >>> invocation. >>> >>> With my changes there's less than 10% overhead over the sum of make >>> calls. I.e. >>> for 56 seconds of make calls (x11/xorg on my system) there are only >>> 4 more >>> seconds of processing, instead of 15 to 20 seconds. >>> >>> The original make package-depends does some things I don't >>> understand. Once >>> I've figured them out I will post my little patch here. >>> >>> >> >> >> Someone pointed out that what I was proposing in +DEPENDENCIES is >> already to be found in +CONTENTS. So here is a proof of concept patch >> to /usr/ports/Mk/bsd.port.mk (proof of concept because no error >> checking, and things like that). >> >> For me it makes registration about 3 times faster. > > And also, the only reason it goes slow is because it has to do > (cd $$dir; make -V PKGNAME) > for every dir in _LIB_RUN_DEPENDS. But if instead we kept a file in > /var/db/pkg called something like +PACKAGE_NAMES, where as each port is > created we add in a one line entry with this mapping of origin to > package name, then registration would take less than a second. It also > has the advantage that the created +REQUIRED_BY really would be an > accurate reflection of the true state of affairs rather than what it > should be (meaning that if you cvsup ports and then build on top of old > dependencies, +REQUIRED_BY would still be accurate). > > Stephen Another option is to create this +PACKAGE_NAMES file on the fly, something like as follows: @rm -f /tmp/tempfile; \ for pkgname in /var/db/pkg/*; do \ echo `basename $$pkgname``grep "@comment ORIGIN:" $$pkgname/+CONTENTS | sed "s/@comment ORIGIN//"` >> /tmp/tempfile; \ done; \ This process really is rather quick, maybe a few seconds, and certainly way quicker than the many invocations of make.