From owner-freebsd-ports@FreeBSD.ORG Mon Aug 4 14:43:40 2008 Return-Path: Delivered-To: freebsd-ports@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A5AB7106567B for ; Mon, 4 Aug 2008 14:43:40 +0000 (UTC) (envelope-from chukharev@mail.ru) Received: from mx2.mail.ru (mx2-2.mail.ru [194.67.23.122]) by mx1.freebsd.org (Postfix) with ESMTP id 4155C8FC23 for ; Mon, 4 Aug 2008 14:43:39 +0000 (UTC) (envelope-from chukharev@mail.ru) Received: from [130.230.40.65] (port=60421 helo=localhost) by mx2.mail.ru with asmtp id 1KQ1He-000N5H-00; Mon, 04 Aug 2008 18:43:38 +0400 Date: Mon, 04 Aug 2008 17:45:27 +0300 To: "Doug Barton" From: "V.Chukharev" Content-Type: text/plain; charset=koi8-r MIME-Version: 1.0 References: <08GlBzRclM@dmeyer.dinoex.sub.org> <488A0B2E.9070403@FreeBSD.org> <488F79F8.1030105@FreeBSD.org> <48969F94.2060100@FreeBSD.org> Content-Transfer-Encoding: 7bit Message-ID: In-Reply-To: <48969F94.2060100@FreeBSD.org> User-Agent: Opera Mail/9.51 (FreeBSD) X-Spam: Not detected X-Mras: OK Cc: Dirk Meyer , freebsd-ports@freebsd.org Subject: Re: Mk/bsd.openssl.mk optimization X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Aug 2008 14:43:40 -0000 On Mon, 04 Aug 2008 09:20:04 +0300, Doug Barton wrote: > Vladimir Chukharev wrote: >> On Tue, 29 Jul 2008 23:13:44 +0300, Doug Barton >> wrote: >> >>> V.Chukharev wrote: >>>> Another patch, just one line. It can be applied independently >>>> from the patch for bsd.port.subdir.mk. >>> I use: grep -l "@comment ORIGIN:${1}$" $pdb/*/+CONTENTS for >>> similar purposes in portmaster with no complaints so far. Is >>> there a reason you need to do the complicated thing? >> >> I am a bit worried by the size of * expansion. > > I'm starting to wonder if you've actually tested and/or benchmarked Not very deeply, you are right. Actually I hoped that when I demonstrate the 10 time reduced time of index generation with a bad patch, maintainer or someone else would jump in and make a real fix for the problem. When that did not happen, I tried to prepare a patch which does not break things. It's not ready yet (repeat: I knew practically nothing about make and bsd.*.mk when starting!). > this stuff. I've run tests of the construction above for 5,000 > directories which is way more ports than a user would ever have > installed. Rerunning this contrived example: Ok, that means that the limit is between 5000 and 6721: $ ls /var/db/pkg/*/+* | wc bash: /bin/ls: Argument list too long 0 0 0 $ time find /var/db/pkg -type f | wc -l 6721 real 0m0.151s user 0m0.013s sys 0m0.091s Is 5000 more then a user would EVER have? So, NEVER more than 4 times what I have now? Hm... It's possible that you are right. But I didn't want to bet on it and used find. > 101$ time grep -l doug /home/dougb/testglob/*/file > /home/dougb/testglob/4785/file > real 0m0.718s > user 0m0.026s > sys 0m0.690s > > time find /home/dougb/testglob/* -type f -name file -exec grep -l doug > {} \; > /home/dougb/testglob/4785/file > real 0m26.344s > user 0m1.706s > sys 0m22.771s > > Piping to xargs instead of using -exec is actually quite a bit faster, > roughly 3.5 seconds wall clock time using the same setup (post caching). Yes, I know this (and since I did not know about \+ instead of \; in find -exec, I used xargs). Now I prefer \+. >> I have about 1380 ports installed, and this number can grow. One >> That's the reason for find. And it is 5-30 times faster then grep >> -r > > I didn't say anything about grep -r, look carefully at what I wrote. Oh, so 'the complicated thing' is not 'find | xargs' but the rest of the command? Then you ask wrong person, I have nothing to do with it. I do not know who wrote it and how it works. I only wanted to get rid of grep -r in this small patch. >> (depending on existence and size of /var/pkg/db/pkgdb.db). > > ... which is why the command I pasted above skips it altogether. ...the same way as any of the tricks with find. I tried to paste your command into the place. It gives errors. >> As to the rest of the command - I do not want to mess with it yet. >> It works. > > I think you need to read des' rules on optimization, especially the > bit about not doing optimization unless you're doing extensive > benchmarking. I believe I read it many years ago. If you give a link I will check it out. (No, I do not want to google it.) > Doug > Thanks! Vladimir -- V. Chukharev