From owner-freebsd-ports@FreeBSD.ORG Mon Mar 31 07:49:52 2008 Return-Path: Delivered-To: freebsd-ports@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6AEA1106566C; Mon, 31 Mar 2008 07:49:52 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from pd2mo1so.prod.shaw.ca (idcmail-mo1so.shaw.ca [24.71.223.10]) by mx1.freebsd.org (Postfix) with ESMTP id 5B2F38FC1F; Mon, 31 Mar 2008 07:49:52 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from pd3mr2so.prod.shaw.ca (pd3mr2so-qfe3.prod.shaw.ca [10.0.141.178]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with ESMTP id <0JYL00DS41N1DXB0@l-daemon>; Mon, 31 Mar 2008 00:49:49 -0600 (MDT) Received: from pn2ml2so.prod.shaw.ca ([10.0.121.146]) by pd3mr2so.prod.shaw.ca (Sun Java System Messaging Server 6.2-7.05 (built Sep 5 2006)) with ESMTP id <0JYL00CO21N0OK80@pd3mr2so.prod.shaw.ca>; Mon, 31 Mar 2008 00:49:50 -0600 (MDT) Received: from soralx ([24.87.3.133]) by l-daemon (Sun Java System Messaging Server 6.2-7.05 (built Sep 5 2006)) with ESMTP id <0JYL00JOV1MYLO00@l-daemon>; Mon, 31 Mar 2008 00:49:48 -0600 (MDT) Date: Sun, 30 Mar 2008 23:49:46 -0700 From: soralx@cydem.org In-reply-to: <20080326201416.GA34807@freebsd.org> To: freebsd-ports@FreeBSD.org Message-id: <20080330234946.7fec88bf@soralx> MIME-version: 1.0 X-Mailer: Claws Mail 3.3.0 (GTK+ 2.12.9; i386-portbld-freebsd7.0) Content-type: text/plain; charset=US-ASCII Content-transfer-encoding: 7bit References: <1206548309.78560.69.camel@pav.hide.vol.cz> <20080326201416.GA34807@freebsd.org> Cc: rdivacky@FreeBSD.org, pav@FreeBSD.org Subject: Re: [patch] pkg_delete(1) speedup X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2008 07:49:52 -0000 > > You might have noticed a thread on the mailing list called "ports system > > woes". The submitter pointed out an inefficiency in pkg_delete routine, > > that parses the whole /var/db/pkg over and over again for every > > dependency of a package being removed. > > > > Attached is a patch by rdivacky that implements the idea of looking up > > all the values in a single pass over /var/db/pkg content. > > I hacked a slightly better patch that coveres a part of pkg_add too.. > > please review/test on: > > www.vlakno.cz/~rdivacky/pkg_tools.patch > > comments, benchmarks more than welcome! All right, I've been gone to the Real World for a while, but I returned %-) First, allow me to note that it's rather impressing to see the level of interest and responses my half-baked email about my little digs into pkg_* tools produced. Before I even finished thinking whether I will have enough time to fix the tools proper, patches started appearing on the horizon (the same day, practically)! This is quite reassuring, as it shows that developers still care about code and algorithm quality, even if things work OK on modern hardware (just lack of developer time, that's all, I suppose). For that I'm grateful -- way to go :) Now, here are the same tests on the same hardware, but with pkg_tools.patch applied: [root@freen0de /usr/ports/x11/rxvt-unicode]# make [root@freen0de /usr/ports/x11/rxvt-unicode]# time make install ===> Generating temporary packing list ===> Checking if x11/rxvt-unicode already installed load: 0.53 cmd: pkg_info 25799 [biord] 0.06u 0.07s 0% 1532k /usr/bin/install -c -o root -g wheel -d /usr/local/bin ===> Documentation installed in /usr/local/share/doc/rxvt-unicode. ===> Compressing manual pages for rxvt-unicode-9.02_1 ===> Registering installation for rxvt-unicode-9.02_1 load: 0.29 cmd: sed 26266 [biord] 0.00u 0.00s 0% 728k load: 0.27 cmd: sh 26568 [runnable] 0.00u 0.00s 0% 164k load: 0.24 cmd: sh 25951 [biord] 0.14u 0.09s 0% 1228k load: 0.22 cmd: grep 27026 [runnable] 0.00u 0.00s 0% 256k ===> SECURITY REPORT: This port has installed the following binaries which execute with real 1m13.885s user 0m3.903s sys 0m4.870s [root@freen0de /usr/ports/x11/rxvt-unicode]# s; sleep 300 && echo -e "\n" [root@freen0de /usr/ports/x11/rxvt-unicode]# time pkg_delete /var/db/pkg/rxvt-unicode-9.02_1/ load: 0.36 cmd: pkg_delete 27480 [biord] 0.35u 0.40s 1% 972k real 0m37.218s user 0m0.448s sys 0m0.509s [root@freen0de /usr/ports/x11/rxvt-unicode]# make reinstall > /dev/null; sync [root@freen0de /usr/ports/x11/rxvt-unicode]# time pkg_delete /var/db/pkg/rxvt-unicode-9.02_1/ real 0m20.261s user 0m0.349s sys 0m0.476s [root@freen0de /usr/ports/x11/rxvt-unicode]# So, the time drops from over 7 minutes to 20 seconds -- sweet! :) Notice pkg_info in ^T output during "Checking if x11/rxvt-unicode already installed" phase. This one takes awhile. The actual command is: `/usr/sbin/pkg_info -q -O x11/rxvt-unicode` real 0m37.697s user 0m0.125s sys 0m0.360s find_pkgs_by_origin() in info/perform.c uses the same matchbyorigin() in lib/match.c. What's interesting here, however, is that simple `time grep ORIGIN /var/db/pkg/*/+CONTENTS` takes ~7 sec (XXX re-test on that same notebook XXX), while find_pkgs_by_origin() incarnation of practically the same functionality takes over 30 sec. To my eye, it doesn't look like matchbyorigin() could be re-implemented to be faster with little effort, but could somebody have a quick look as well? Would doing mmap() instead of scanning file line-by-line be any faster? (though I'm not saying it's a great idea) BTW, I have a feeling that the "Registering installation" should be made more verbose. It takes more time that anything else now, and one's left to wonder what exactly is going on (seems like quite a few different things). Also, I found that during the "Checking if <*> already installed" step, 'mtree' (XXX find out exact command here XXX) is called (from bsd.port.mk?), which can be skipped by setting NO_MTREE. What effect does [not] calling mtree have? And to conclude, here we have a benchmark from my faster machine (Core2 Dual 2.72GHz, 2G RAM, MK2018GAS 4200RPM HDD with 2M buffer), BEFORE patch was applied: [root@soralx /usr/ports/x11/rxvt-unicode]# time make install > /dev/null real 0m23.097s user 0m0.000s sys 0m0.219s [root@soralx /usr/ports/x11/rxvt-unicode]# time pkg_delete /var/db/pkg/rxvt-unicode-9.02/ real 0m2.243s user 0m0.056s sys 0m0.202s [root@soralx /usr/ports/x11/rxvt-unicode]# time make reinstall > /dev/null real 0m26.867s user 0m0.641s sys 0m0.936s [root@soralx /usr/ports/x11/rxvt-unicode]# time pkg_delete /var/db/pkg/rxvt-2.6.4_3/ #very few depends real 0m0.090s user 0m0.009s sys 0m0.001s [root@soralx /usr/ports/x11/rxvt-unicode]# time pkg_delete /var/db/pkg/rxvt-unicode-9.02_1/ real 0m0.521s user 0m0.073s sys 0m0.443s After patching pkg_install/: [root@soralx /usr/ports/x11/rxvt-unicode]# time make reinstall > /dev/null real 0m3.602s user 0m0.534s sys 0m0.905s [root@soralx /usr/ports/x11/rxvt-unicode]# time pkg_delete /var/db/pkg/rxvt-unicode-9.02_1/ real 0m0.079s user 0m0.033s sys 0m0.046s Not bad, we see improvements in both pkg_install and pkg_delete, as expected. > roman [SorAlx] ridin' VN1500-B2