Date: Thu, 17 May 2007 11:44:36 +0200 From: Alexander Leidinger <Alexander@Leidinger.net> To: Robert Noland <rnoland@2hip.net> Cc: "\[LoN\]Kamikaze" <LoN_Kamikaze@gmx.de>, Stephen Montgomery-Smith <stephen@math.missouri.edu>, Pav Lucistnik <pav@freebsd.org>, freebsd-ports@freebsd.org, Kris Kennaway <kris@obsecurity.org> Subject: Re: Time to abandon recursive pulling of dependencies? Message-ID: <20070517114436.3fdd55b1@deskjail> In-Reply-To: <1179353641.79329.11.camel@rnoland-ibm.acs.internap.com> References: <464597C6.3030406@gmx.de> <20070512174011.GA22526@xor.obsecurity.org> <4645FF71.60100@gmx.de> <20070512175824.GA23103@xor.obsecurity.org> <20070512133054.B5588@math.missouri.edu> <4646193E.5040503@gmx.de> <464665D5.1090509@math.missouri.edu> <464676CC.2000203@math.missouri.edu> <46469BF5.9030603@math.missouri.edu> <1179058696.13237.3.camel@ikaros.oook.cz> <20070513101332.L82202@math.missouri.edu> <46479215.1080401@math.missouri.edu> <4647C4FB.9050607@math.missouri.edu> <4647F182.6000402@gmx.de> <46487491.6020806@math.missouri.edu> <46488168.3030300@math.missouri.edu> <464B155A.1080105@math.missouri.edu> <464B7144.2000600@math.missouri.edu> <1179353641.79329.11.camel@rnoland-ibm.acs.internap.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--MP_jdnp5QohagFfqvDK5VQmjDw
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Quoting Robert Noland <rnoland@2hip.net> (Wed, 16 May 2007 18:14:01 -0400):
> On Wed, 2007-05-16 at 16:01 -0500, Stephen Montgomery-Smith wrote:
> > Ok chaps, I think I have it.
> >
> > This involves no recursive calls of make. Furthermore the
> > dependencies
> > it creates are the real dependencies on your system, not what ports
> > thinks it should be, because it gets all the information from
> > /var/db/pkg. On my system it takes a second or two to register a port
> > -
> > it takes about the same amount of time whether it has few
> > dependencies
> > or many, except when there are zero dependencies, when it takes no
> > time
> > at all.
> >
> > If I get some positive feedback on this one, I'll submit a PR. You
> > can
> > try it out by typing "make actual-package-depends" as opposed to
> > "make
> > package-depends."
>
> Ok, I'm really trying to make sense of these numbers...
>
> For gnome2-2.18.1_1 on my -current system...
>
> rnoland-ibm% make package-depends|wc -l
> 362
> rnoland-ibm% make actual-package-depends|wc -l
> 294
>
> Registered the old way:
> rnoland-ibm% grep @pkgdep /var/db/pkg/gnome2-2.18.1_1/+CONTENTS|wc -l
> 176
>
> and the new way:
> rnoland-ibm% grep @pkgdep /var/db/pkg/gnome2-2.18.1_1/+CONTENTS|wc -l
> 294
>
> I put together an awk script yesterday which was very close to the one
> Alexander posted and got 295... and the diff of the output of my awk and
> actual-package-depends is much more than a single line, which I also
> can't yet explain.
I had a look at it. It is missing those dependencies which are not
installed. Depending on when this target is used, this is ok or not.
Some benchmark:
---snip---
[original]
make package-depends 80.18s user 53.92s system 73% cpu 3:01.53 total
make package-depends 80.70s user 52.94s system 87% cpu 2:32.25 total
make package-depends > /tmp/list1 79.58s user 54.28s system 87% cpu 2:32.59 total
[no AWK]
make actual-package-depends 4.76s user 7.46s system 57% cpu 21.082 total
make actual-package-depends 4.87s user 7.34s system 67% cpu 18.019 total
make actual-package-depends > /tmp/list2 4.58s user 7.60s system 87% cpu 13.942 total
[AWK]
make actual-package-depends 3.15s user 7.45s system 85% cpu 12.420 total
make actual-package-depends 3.13s user 7.49s system 84% cpu 12.524 total
make actual-package-depends 3.11s user 7.49s system 88% cpu 11.995 total
make actual-package-depends >| /tmp/list3 3.16s user 7.42s system 89% cpu 11.83 0 total
[AWK + partly missing (only direct dependencies)]
make actual-package-depends > /tmp/list4 4.60s user 7.63s system 90% cpu 13.479 total
make actual-package-depends 4.68s user 7.56s system 87% cpu 13.985 total
make actual-package-depends 4.58s user 7.64s system 89% cpu 13.594 total
...sorting list1 & list2 & list3 & list4...
% wc -l /tmp/list*s
320 /tmp/list1s
308 /tmp/list2s
308 /tmp/list3s
310 /tmp/list4s
1246 total
---snip---
Note, there was trackerd running in the background while testing...
For the difference between the redirected output case: I think the
gnome terminal needs a lot of time to print all the lines. But still,
the awk version takes around 3/4 of the time (interesting is the user
time, not the total time). Stephen's version can be speed up some
fractions by inserting a break into the first while-loop.
Attached is my awk-version and the awk version which also includes the
direct dependencies.
Bye,
Alexander.
--
The only new TV show worth watching will be cancelled.
http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7
http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137
--MP_jdnp5QohagFfqvDK5VQmjDw
Content-Type: text/x-patch; name=bsd.port.mk-awk.diff
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=bsd.port.mk-awk.diff
--- /usr/ports/Mk/bsd.port.mk Sat Apr 7 14:51:47 2007
+++ /space/jails/basejail/usr/ports/Mk/bsd.port.mk Thu May 17 11:19:03 2007
@@ -2321,7 +2321,7 @@
DISABLE_CONFLICTS= YES
.endif
.if !defined(PKG_ARGS)
-PKG_ARGS= -v -c -${COMMENT:Q} -d ${DESCR} -f ${TMPPLIST} -p ${PREFIX} -P "`cd ${.CURDIR} && ${MAKE} package-depends | ${GREP} -v -E ${PKG_IGNORE_DEPENDS} | ${SORT} -u`" ${EXTRA_PKG_ARGS} $${_LATE_PKG_ARGS}
+PKG_ARGS= -v -c -${COMMENT:Q} -d ${DESCR} -f ${TMPPLIST} -p ${PREFIX} -P "`cd ${.CURDIR} && ${MAKE} actual-package-depends | ${GREP} -v -E ${PKG_IGNORE_DEPENDS} | ${SORT} -u`" ${EXTRA_PKG_ARGS} $${_LATE_PKG_ARGS}
.if !defined(NO_MTREE)
PKG_ARGS+= -m ${MTREE_FILE}
.endif
@@ -5145,15 +5145,47 @@
shift 3; \
done; \
checked="$$dir $$childdir $$checked"; \
- else \\
+ else \
${ECHO_MSG} "${PKGNAME}: \"$$dir\" non-existent -- dependency list incomplete" >&2; \
fi; \
done
+ACTUAL-PACKAGE-DEPENDS?= \
+ if [ "${_LIB_RUN_DEPENDS}" != " " ]; then \
+ origins=$$(for pkgname in ${PKG_DBDIR}/*; do \
+ if [ -e $$pkgname/+CONTENTS ]; then \
+ basename $$pkgname; \
+ ${SED} -n -e "s/@comment ORIGIN://p" $$pkgname/+CONTENTS; \
+ fi; \
+ done); \
+ for dir in ${_LIB_RUN_DEPENDS:C,[^:]*:([^:]*):?.*,\1,}; do \
+ dir=`dirname $$dir | xargs basename`/`basename $$dir`; \
+ set -- $$origins; \
+ while [ $$\# != 0 ]; do \
+ if [ $$dir = $$2 ]; then \
+ ${ECHO_CMD} $$1:$$dir; \
+ if [ -e ${PKG_DBDIR}/$$1/+CONTENTS ]; then \
+ ${AWK} -F '( |:)' '/@pkgdep / { pkgname=$$2 } /@comment DEPORIGIN:/ { printf "%s:%s\n", pkgname, $$3 }' ${PKG_DBDIR}/$$1/+CONTENTS; \
+ fi; \
+ break; \
+ fi; \
+ shift 2; \
+ done; \
+ done | sort -u; \
+ fi
+
# Print out package names.
package-depends:
@${PACKAGE-DEPENDS-LIST} | ${AWK} '{print $$1":"$$3}'
+
+actual-package-depends:
+ @${ACTUAL-PACKAGE-DEPENDS}
# Build packages for port and dependencies
--MP_jdnp5QohagFfqvDK5VQmjDw
Content-Type: text/x-patch; name=bsd.port.mk-awk_and_direct_missing.diff
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=bsd.port.mk-awk_and_direct_missing.diff
--- /usr/ports/Mk/bsd.port.mk Sat Apr 7 14:51:47 2007
+++ /space/jails/basejail/usr/ports/Mk/bsd.port.mk Thu May 17 11:19:03 2007
@@ -2321,7 +2321,7 @@
DISABLE_CONFLICTS= YES
.endif
.if !defined(PKG_ARGS)
-PKG_ARGS= -v -c -${COMMENT:Q} -d ${DESCR} -f ${TMPPLIST} -p ${PREFIX} -P "`cd ${.CURDIR} && ${MAKE} package-depends | ${GREP} -v -E ${PKG_IGNORE_DEPENDS} | ${SORT} -u`" ${EXTRA_PKG_ARGS} $${_LATE_PKG_ARGS}
+PKG_ARGS= -v -c -${COMMENT:Q} -d ${DESCR} -f ${TMPPLIST} -p ${PREFIX} -P "`cd ${.CURDIR} && ${MAKE} actual-package-depends | ${GREP} -v -E ${PKG_IGNORE_DEPENDS} | ${SORT} -u`" ${EXTRA_PKG_ARGS} $${_LATE_PKG_ARGS}
.if !defined(NO_MTREE)
PKG_ARGS+= -m ${MTREE_FILE}
.endif
@@ -5145,15 +5145,47 @@
shift 3; \
done; \
checked="$$dir $$childdir $$checked"; \
- else \\
+ else \
${ECHO_MSG} "${PKGNAME}: \"$$dir\" non-existent -- dependency list incomplete" >&2; \
fi; \
done
+ACTUAL-PACKAGE-DEPENDS?= \
+ if [ "${_LIB_RUN_DEPENDS}" != " " ]; then \
+ origins=$$(for pkgname in ${PKG_DBDIR}/*; do \
+ if [ -e $$pkgname/+CONTENTS ]; then \
+ basename $$pkgname; \
+ ${SED} -n -e "s/@comment ORIGIN://p" $$pkgname/+CONTENTS; \
+ fi; \
+ done); \
+ for dir in ${_LIB_RUN_DEPENDS:C,[^:]*:([^:]*):?.*,\1,}; do \
+ dir=`dirname $$dir | xargs basename`/`basename $$dir`; \
+ set -- $$origins; \
+ found=false; \
+ while [ $$\# != 0 ]; do \
+ if [ $$dir = $$2 ]; then \
+ ${ECHO_CMD} $$1:$$dir; \
+ if [ -e ${PKG_DBDIR}/$$1/+CONTENTS ]; then \
+ ${AWK} -F '( |:)' '/@pkgdep / { pkgname=$$2 } /@comment DEPORIGIN:/ { printf "%s:%s\n", pkgname, $$3 }' ${PKG_DBDIR}/$$1/+CONTENTS; \
+ fi; \
+ found=true; \
+ break; \
+ fi; \
+ shift 2; \
+ done; \
+ if [ $$found = false ]; then \
+ ${ECHO_CMD} $$(cd ${PORTSDIR}/$$dir; ${MAKE} -V PKGNAME):$$dir; \
+ fi; \
+ done | sort -u; \
+ fi
+
# Print out package names.
package-depends:
@${PACKAGE-DEPENDS-LIST} | ${AWK} '{print $$1":"$$3}'
+
+actual-package-depends:
+ @${ACTUAL-PACKAGE-DEPENDS}
# Build packages for port and dependencies
--MP_jdnp5QohagFfqvDK5VQmjDw--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070517114436.3fdd55b1>