From owner-freebsd-ports@FreeBSD.ORG Wed Jul 23 22:03:38 2008 Return-Path: Delivered-To: freebsd-ports@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9FC0F1065674; Wed, 23 Jul 2008 22:03:38 +0000 (UTC) (envelope-from chukharev@mail.ru) Received: from mx4.mail.ru (fallback.mail.ru [194.67.57.14]) by mx1.freebsd.org (Postfix) with ESMTP id 402758FC0A; Wed, 23 Jul 2008 22:03:37 +0000 (UTC) (envelope-from chukharev@mail.ru) Received: from mx2.mail.ru (mx2-2.mail.ru [194.67.23.122]) by mx4.mail.ru (mPOP.Fallback_MX) with ESMTP id 2FDA710B3E8E; Wed, 23 Jul 2008 10:41:32 +0400 (MSD) Received: from [130.230.40.65] (port=53865 helo=localhost) by mx2.mail.ru with asmtp id 1KLY2U-00038q-00; Wed, 23 Jul 2008 10:41:30 +0400 Date: Wed, 23 Jul 2008 09:43:03 +0300 To: dinoex@freebsd.org From: "V.Chukharev" Content-Type: text/plain; charset=iso-8859-15 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Message-ID: User-Agent: Opera Mail/9.51 (FreeBSD) X-Spam: Not detected X-Mras: OK Cc: "freebsd-ports@freebsd.org" Subject: Mk/bsd.openssl.mk optimization X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jul 2008 22:03:38 -0000 Dear maintainer, I have noticed that most time of 'make index' on my system takes a grep for libssl. The following is done while 'make index' was running. $ ps axww | grep libssl 23119 p1 IN+ 0:00,00 sh -c grep -l -r "^lib/libssl.so." "/var/db/pkg" | while read contents; do sslprefix=`grep "^@cwd " "${contents}" | /usr/bin/head -n 1`; if test "${sslprefix}" = "@cwd /usr/local" ; then echo "${contents}"; break; fi; done 23124 p1 DN+ 0:00,84 grep -l -r ^lib/libssl.so. /var/db/pkg 23125 p1 IN+ 0:00,00 sh -c grep -l -r "^lib/libssl.so." "/var/db/pkg" | while read contents; do sslprefix=`grep "^@cwd " "${contents}" | /usr/bin/head -n 1`; if test "${sslprefix}" = "@cwd /usr/local" ; then echo "${contents}"; break; fi; done 24555 p1 IN+ 0:00,00 sh -c grep -l -r "^lib/libssl.so." "/var/db/pkg" | while read contents; do sslprefix=`grep "^@cwd " "${contents}" | /usr/bin/head -n 1`; if test "${sslprefix}" = "@cwd /usr/local" ; then echo "${contents}"; break; fi; done 24556 p1 DN+ 0:00,41 grep -l -r ^lib/libssl.so. /var/db/pkg 24557 p1 IN+ 0:00,00 sh -c grep -l -r "^lib/libssl.so." "/var/db/pkg" | while read contents; do sslprefix=`grep "^@cwd " "${contents}" | /usr/bin/head -n 1`; if test "${sslprefix}" = "@cwd /usr/local" ; then echo "${contents}"; break; fi; done So I looked through some Mk stuff, and tryed to optimize it. Since I do not know any magic of bsd.*.mk, I did just a simple test. And with the below shown patch the time is only 1500 s, while with the original version the time is more than 15000 s. I guess the time depens very much on number of installed ports, I have about 1300. The resulting INDEX* files are identical. I should confess I do not understand how my patch works. I expected to see a file /usr/ports/.openssl_installed or /tmp/index_something/.openssl_installed or similar, but could not see it... But I hope this patch will inspire you to do something really working. chu@chu:/usr/ports 22:08:17 $ sudo time nice make index Generating INDEX-7 - please wait..Warning: Duplicate INDEX entry: apr-gdbm-db42-1.3.2 Warning: Duplicate INDEX entry: mod_rpaf-ap2-0.6 Done. 1503.11 real 988.23 user 116.09 sys chu@chu:/usr/ports 22:33:49 $ diff -u Mk/bsd.openssl.mk.orig Mk/bsd.openssl.mk --- Mk/bsd.openssl.mk.orig 2008-05-18 01:34:41.000000000 +0300 +++ Mk/bsd.openssl.mk 2008-07-22 22:07:31.000000000 +0300 @@ -121,11 +121,15 @@ exists(${DESTDIR}/${LOCALBASE}/lib/libcrypto.so) # find installed port and use it for dependency PKG_DBDIR?= ${DESTDIR}/var/db/pkg +.if exists(${WRKDIR}/.openssl_installed) +OPENSSL_INSTALLED!= cat ${WRKDIR}/.openssl_installed +.else OPENSSL_INSTALLED!= grep -l -r "^lib/libssl.so." "${PKG_DBDIR}" | \ while read contents; do \ sslprefix=`grep "^@cwd " "$${contents}" | ${HEAD} -n 1`; \ if test "$${sslprefix}" = "@cwd ${LOCALBASE}" ; then \ - echo "$${contents}"; break; fi; done + echo "$${contents}" | tee ${WRKDIR}/.openssl_installed; break; fi; done +.endif OPENSSL_PORT!= grep "^@comment ORIGIN:" "${OPENSSL_INSTALLED}" | ${CUT} -d : -f 2 OPENSSL_SHLIBFILE!= grep "^lib/libssl.so." "${OPENSSL_INSTALLED}" OPENSSL_SHLIBVER?= ${OPENSSL_SHLIBFILE:E} Without the patch: chu@chu:/usr/ports 22:43:24 $ sudo time nice make index Generating INDEX-7 - please wait..Warning: Duplicate INDEX entry: apr-gdbm-db42-1.3.2 Warning: Duplicate INDEX entry: mod_rpaf-ap2-0.6 Done. 15216.17 real 1045.99 user 261.20 sys -- V. Chukharev