Date: Thu, 31 Oct 2013 02:16:43 +0100 From: Matthias Andree <mandree@FreeBSD.org> To: ports@FreeBSD.org Cc: portmgr@FreeBSD.org Subject: RFT: bsd.stage.mk overhaul v1 Message-ID: <5271AF7B.40005@FreeBSD.org>
next in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --mca21qshKXmb6CmStkrgfUXJoMF24tHuR Content-Type: multipart/mixed; boundary="------------000705000707070505070604" This is a multi-part message in MIME format. --------------000705000707070505070604 Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: quoted-printable Greetings, I have spent a few hours at getting bsd.stage.mk in a better shape, meaning more easily maintained, faster, more robust, more usable. To that extent, I have split the actual code out from Mk/bsd.stage.mk into Mk/Scripts/check-stagedir.sh and use it for "make check-orphans" and "make makeplist". It prints fewer false positives now, prints more useful statements for direct use in pkg-plist for directories outside prefix. *A detailed description is prepended to the patch.* This needs testing on lots of ports, and after a review and initial tests by others also possibly an -exp run to see where we will end. The patch is attached and also available for download from: http://people.freebsd.org/~mandree/stage-upgrade-v1.patch. To be applied against /usr/ports. Best regards Matthias --------------000705000707070505070604 Content-Type: text/x-patch; name="stage-upgrade-v1.patch" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="stage-upgrade-v1.patch" v1 This patch is to remedy several issues around bsd.stage.mk. Author: Matthias Andree <mandree@> Summary: Fewer false positives, much faster, easier maintenance. Bugfixes: * @cmd in pkg-plist is now properly handled. - It was previously treated the same as though there was a directory following it, missing the prefix. (ordering matters in case...esac) - Due to the cwd=3D${PREFIX} inside the while read line loop, state tracking was broken and every new line assumed that cwd were the prefix. * stage-qa no longer complains about unstripped binaries if debugging is active (WITH_DEBUG set && WITHOUT_DEBUG unset). * The compress-man target uses ECHO_MSG, not ECHO_CMD, to print its build step. Additions: * The plist parser now understands @unexec rmdir ... || ... lines, including those with redirections, so that there are no false positives for directories stripped with @unexec rmdir (usually happens on stuff installed outside $PREFIX, as in /var). * The system's root and var mtrees are now also expanded to avoid false @dirrm positives if a port installs directories under /var and has to create parents in the stagedir that are present in a fully installed system (i. e. in the real $PREFIX). * Given that pkg_create is deemed beyond repair with respect to deleting files outside prefix, generate @unexec rmdir statements for such directories, rather than @dirrmtry, to sidestep the problem. Speedups: * the orphan check now generates sorted lists of staged files, and plisted/mtree files, and compares them with comm(1). This saves us the overhead of running one grep process per file and up to two per directory, and defers the actual list processing to a shell utility. Complexity has not changed, but overhead per item has. * the orphan check now uses one file for directories and one file for files mentioned in pkg-plist, so we need not decorate them with "dir " and parse them out any longer. * qa.sh's shebang scanner only looks at the first line of a file, sed is told to exit from the 2nd line. Other Changes: * Split the makeplist/check-orphans logic out of bsd.stage.mk, it is too unwieldy to maintain in make-escaped shell syntax, and permits shell tracing with "SH=3Dsh -x" (including quotes!) * Unify the functions "makeplist" and "check-orphans" in one script. The only difference is that makeplist assumes an empty pkg-plist, whereas check-orphans parses it. * overhaul the mtree extractor, avoiding awk. Tested on: FreeBSD 10-stable amd64 (with pkgNG, obviously) FreeBSD 9.2-release amd64 (with traditional pkg stuff) FreeBSD 9.1 amd64 Tinderbox (also traditional pkg stuff) FreeBSD 9.2 amd64 poudriere testport for FreeBSD 9.1 amd64 (also no pkgNG) - note that poudriere 3.0.12's own stage orphan checking is b0rked and prints false positives for news/newsstar FreeBSD 9.1 amd64 Tinderbox (also traditional pkg stuff) Index: Mk/Scripts/check-stagedir.sh =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- Mk/Scripts/check-stagedir.sh (revision 0) +++ Mk/Scripts/check-stagedir.sh (working copy) @@ -0,0 +1,113 @@ +#!/bin/sh +# ports/Mk/Scripts/check-stagedir.sh - called from ports/Mk/bsd.stage.mk= + +set -e +export LC_ALL=3DC + +# lists an mtree file's contents, prefixed to dir. +listmtree() { # mtreefile prefix + { + echo '#mtree' + sed 's/nochange$//;' $1 + } | tar -tf- | sed "s,^,$2/,;s,^$2/\.$,$2," +} + +# obtain operating mode from command line +makeplist=3D0 +case "$1" in + orphans) ;; + makeplist) makeplist=3D1 ;; + *) echo >&2 "Usage: $0 {orphans|makelist}" ; exit 1 ;; +esac + +# validate environment +envfault=3D0 +for i in STAGEDIR PREFIX LOCALBASE WRKDIR WRKSRC MTREE_FILE \ + TMPPLIST DATADIR DOCSDIR EXAMPLESDIR +do + if eval test -z "\$$i" ; then + echo >&2 "Environment variable $i undefined. Aborting." + envfault=3D1 + fi +done +if [ $envfault -ne 0 ] ; then + exit 1 +fi + +set -u + +#### EXPAND TMPPLIST TO ABSOLUTE PATHS, SPLITTING FILES AND DIRS TO +# Use file descriptors 1 and 3 so that the while loop can write +# files to the pipe and dirs to a separate file. +if [ $makeplist =3D 0 ] ; then + # check for orphans + cwd=3D${PREFIX} + while read line; do + case $line in + @dirrm*|@unexec*rmdir*) + line=3D"$(printf %s "$line" \ + | sed -Ee 's/\|\|.*//;s|[0-9]*[[:space:]]*>[&]?[[:space:]]*[^[:sp= ace:]]+||g' \ + -e "/^@unexec[[:space:]]+rmdir/s|%D|${PREFIX}|g" \ + -e '/^@unexec[[:space:]]+rmdir/s|"(.*)"[[:space:]]+|\1|g' \ + -e 's/@unexec[[:space:]]+rmdir[[:space:]]+//' \ + -e 's/@dirrm(try)?[[:space:]]+//' \ + -e 's/[[:space:]]+$//')" + case "$line" in + /*) echo >&3 "$line" ;; + *) echo >&3 "$cwd/$line" ;; + esac + ;; + # order matters here - we must check @cwd first because + # otherwise the @cwd* would also match it first, shadowing the + # @cwd) line. + @cwd|@cd) cwd=3D${PREFIX} ;; + @cwd*|@cd*) set -- $line ; cwd=3D$2 ;; + @*) ;; + /*) echo "$line" ;; + *) echo "$cwd/$line" ;; + esac + done < ${TMPPLIST} 3>${WRKDIR}/.plist-dirs-unsorted | sort >${WRKDIR}/.= plist-files +else + # generate plist - pretend the plist had been empty + : >${WRKDIR}/.plist-dirs-unsorted + : >${WRKDIR}/.plist-files +fi + +### PRODUCE MTREE FILE +{ + listmtree /etc/mtree/BSD.root.dist / + #listmtree /etc/mtree/BSD.usr.dist /usr + listmtree /etc/mtree/BSD.var.dist /var + + if [ -n "${MTREE_FILE}" ]; then + listmtree "${MTREE_FILE}" "${PREFIX}" + fi + + a=3D${PREFIX} + while :; do + a=3D${a%/*} + [ -z "${a}" ] && break + echo ${a} + done +} > ${WRKDIR}/.mtree + +### HANDLE FILES +find ${STAGEDIR} -type f -o -type l | sort | sed -e "s,${STAGEDIR},," >$= {WRKDIR}/.staged-files +comm -13 ${WRKDIR}/.plist-files ${WRKDIR}/.staged-files \ + | sed \ + -e "s,${DOCSDIR},%%PORTDOCS%%%%DOCSDIR%%,g" \ + -e "s,${EXAMPLESDIR},%%PORTEXAMPLES%%%%EXAMPLESDIR%%,g" \ + -e "s,${DATADIR},%%DATADIR%%,g" \ + -e "s,${PREFIX}/,,g" | grep -v "^share/licenses" || [ $? =3D 1 ] + +### HANDLE DIRS +cat ${WRKDIR}/.plist-dirs-unsorted ${WRKDIR}/.mtree | sort -u >${WRKDIR}= /.traced-dirs +find ${STAGEDIR} -type d | sed -e "s,^${STAGEDIR},,;/^$/d" | sort >${WRK= DIR}/.staged-dirs +comm -13 ${WRKDIR}/.traced-dirs ${WRKDIR}/.staged-dirs \ + | sort -r | sed \ + -e "s,\(.*\)${DOCSDIR},%%PORTDOCS%%\1%%DOCSDIR%%,g" \ + -e "s,\(.*\)${EXAMPLESDIR},%%PORTEXAMPLES%%\1%%EXAMPLESDIR%%,g" \ + -e "s,${DATADIR},%%DATADIR%%,g" \ + -e "s,${PREFIX}/,,g" \ + -e 's,^,@dirrmtry ,' \ + -e 's,@dirrmtry \(/.*\),@unexec rmdir >/dev/null 2>\&1 \1 || :,' | grep= -v "^@dirrmtry share/licenses" || [ $? =3D 1 ] Property changes on: Mk/Scripts/check-stagedir.sh ___________________________________________________________________ Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=3D%H \ No newline at end of property Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Index: Mk/Scripts/qa.sh =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- Mk/Scripts/qa.sh (revision 332161) +++ Mk/Scripts/qa.sh (working copy) @@ -18,7 +18,7 @@ shebang() { rc=3D0 for f in `find ${STAGEDIR} -type f`; do - interp=3D$(sed -n -e '1s/^#![[:space:]]*\([^[:space:]]*\).*/\1/p' $f) + interp=3D$(sed -n -e '1s/^#![[:space:]]*\([^[:space:]]*\).*/\1/p;2q' $= f) case "$interp" in "") ;; /usr/bin/env) ;; @@ -62,7 +62,8 @@ =20 # For now do not raise an error, just warnings stripped() { - [ -x /usr/bin/file ] || return + [ -x /usr/bin/file ] || return # this is fatal + [ -n "${STRIP}" ] || return 0 for f in `find ${STAGEDIR} -type f`; do output=3D`/usr/bin/file ${f}` case "${output}" in Index: Mk/bsd.stage.mk =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- Mk/bsd.stage.mk (revision 332161) +++ Mk/bsd.stage.mk (working copy) @@ -6,10 +6,22 @@ DESTDIRNAME?=3D DESTDIR =20 MAKE_ARGS+=3D ${DESTDIRNAME}=3D${STAGEDIR} -QA_ENV+=3D STAGEDIR=3D${STAGEDIR} PREFIX=3D${PREFIX} \ +QA_ENV+=3D STAGEDIR=3D${STAGEDIR} \ + PREFIX=3D${PREFIX} \ LOCALBASE=3D${LOCALBASE} \ USESDESKTOPFILEUTILS=3D${USES:Mdesktop-file-utils} \ - USESSHAREDMIMEINFO=3D${USES:Mshared-mime-info} + USESSHAREDMIMEINFO=3D${USES:Mshared-mime-info} \ + "STRIP=3D${STRIP}" +CO_ENV+=3D STAGEDIR=3D${STAGEDIR} \ + PREFIX=3D${PREFIX} \ + LOCALBASE=3D${LOCALBASE} \ + WRKDIR=3D${WRKDIR} \ + WRKSRC=3D${WRKSRC} \ + MTREE_FILE=3D${MTREE_FILE} \ + TMPPLIST=3D${TMPPLIST} \ + DATADIR=3D${DATADIR} \ + DOCSDIR=3D${DOCSDIR} \ + EXAMPLESDIR=3D${EXAMPLESDIR} =20 .if !target(stage-dir) stage-dir: @@ -28,8 +40,8 @@ # Fixes all dead symlinks left by the previous round .if !target(compress-man) compress-man: - @${ECHO_CMD} "=3D=3D=3D=3D> Compressing man pages" ; \ - mdirs=3D ; \ + @${ECHO_MSG} "=3D=3D=3D=3D> Compressing man pages (compress-man)" + @mdirs=3D ; \ for dir in ${MANDIRS:S/^/${STAGEDIR}/} ; do \ [ -d $$dir ] && mdirs=3D"$$mdirs $$dir" ;\ done ; \ @@ -76,90 +88,17 @@ =20 .if !target(makeplist) makeplist: stage - @if [ -n "${MTREE_FILE}" ]; then \ - { ${ECHO_CMD} "#mtree"; ${CAT} ${MTREE_FILE}; } | ${TAR} tf - | \ - awk '{ sub(/^\.$$/, "", $$1); \ - if ($$1 =3D=3D "") print "${PREFIX}"; else print "${PREFIX}/"$$1; }' ;= \ - fi > ${WRKDIR}/.mtree - @a=3D${PREFIX}; \ - while :; do \ - a=3D$${a%/*} ; \ - [ -z "$${a}" ] && break ; \ - ${ECHO_CMD} $${a} >> ${WRKDIR}/.mtree ; \ - done - @${FIND} ${STAGEDIR} -type f -o -type l | ${SORT} | ${SED} -e "s,${STAG= EDIR},,g" \ - -e "s,${DOCSDIR},%%PORTDOCS%%%%DOCSDIR%%,g" \ - -e "s,${EXAMPLESDIR},%%PORTEXAMPLES%%%%EXAMPLESDIR%%,g" \ - -e "s,${DATADIR},%%DATADIR%%,g" \ - -e "s,${PREFIX}/,,g" | ${GREP} -v "^share/licenses" || ${TRUE} - @${FIND} ${STAGEDIR} -type d | ${SED} -e "s,${STAGEDIR},,g" \ - | while read line; do \ - ${GREP} -qw "^$${line}$$" ${WRKDIR}/.mtree || { \ - [ -n "$${line}" ] && ${ECHO_CMD} "@dirrmtry $${line}"; \ - }; \ - done | ${SORT} -r | ${SED} \ - -e "s,\(.*\)${DOCSDIR},%%PORTDOCS%%\1%%DOCSDIR%%,g" \ - -e "s,\(.*\)${EXAMPLESDIR},%%PORTEXAMPLES%%\1%%EXAMPLESDIR%%,g" \ - -e "s,${DATADIR},%%DATADIR%%,g" \ - -e "s,${PREFIX}/,,g" | ${GREP} -v "^@dirrmtry share/licenses" || ${TRU= E} + @${SETENV} ${CO_ENV} ${SH} ${SCRIPTSDIR}/check-stagedir.sh makeplist .endif =20 .if !target(check-orphans) check-orphans: stage - @while read line; do \ - cwd=3D${PREFIX} ; \ - case $$line in \ - @dirrm*) \ - set -- $$line ; \ - case $$2 in \ - /*) ${ECHO_CMD} "dir $$2" ;; \ - *) ${ECHO_CMD} "dir $$cwd/$$2" ;; \ - esac ; \ - ;; \ - @cwd) cwd=3D${PREFIX} ;; \ - @cwd*) set -- $$line ; \ - cwd=3D$$2 ;; \ - @*) ;; \ - /*) ${ECHO_CMD} $$line ;; \ - *) ${ECHO_CMD} $$cwd/$$line ;; \ - esac ; \ - done < ${TMPPLIST} > ${WRKDIR}/.expanded-plist - @if [ -n "${MTREE_FILE}" ]; then \ - { ${ECHO_CMD} "#mtree"; ${CAT} ${MTREE_FILE}; } | ${TAR} tf - | \ - awk '{ sub(/^\.$$/, "", $$1); \ - if ($$1 =3D=3D "") print "${PREFIX}"; else print "${PREFIX}/"$$1; }' ;= \ - fi > ${WRKDIR}/.mtree - @a=3D${PREFIX}; \ - while :; do \ - a=3D$${a%/*} ; \ - [ -z "$${a}" ] && break ; \ - ${ECHO_CMD} $${a} >> ${WRKDIR}/.mtree ; \ - done - @${FIND} ${STAGEDIR} -type f -o -type l | ${SORT} | ${SED} -e "s,${STAG= EDIR},,g" \ - | while read line; do \ - ${GREP} -qw "^$${line}$$" ${WRKDIR}/.expanded-plist || { \ - [ -n "$${line}" ] && ${ECHO_CMD} "$${line}" ; \ - } ; \ - done | ${SED} \ - -e "s,${DOCSDIR},%%PORTDOCS%%%%DOCSDIR%%,g" \ - -e "s,${EXAMPLESDIR},%%PORTEXAMPLES%%%%EXAMPLESDIR%%,g" \ - -e "s,${DATADIR},%%DATADIR%%,g" \ - -e "s,${PREFIX}/,,g" | ${GREP} -v "^share/licenses" || ${TRUE} - @${FIND} ${STAGEDIR} -type d | ${SED} -e "s,${STAGEDIR},,g" \ - | while read line; do \ - ${GREP} -qw "^$${line}$$" ${WRKDIR}/.mtree || \ - ${GREP} -qw "dir\ $${line}$$" ${WRKDIR}/.expanded-plist || { \ - [ -n "$${line}" ] && ${ECHO_CMD} "@dirrmtry $${line}"; \ - } ; \ - done | ${SORT} -r | ${SED} \ - -e "s,\(.*\)${DOCSDIR},%%PORTDOCS%%\1%%DOCSDIR%%,g" \ - -e "s,\(.*\)${EXAMPLESDIR},%%PORTEXAMPLES%%\1%%EXAMPLESDIR%%,g" \ - -e "s,${DATADIR},%%DATADIR%%,g" \ - -e "s,${PREFIX}/,,g" | ${GREP} -v "^@dirrmtry share/licenses" || ${TRU= E} + @${ECHO_MSG} "=3D=3D=3D=3D> Items missing from pkg-plist (check-orphans= )" + @${SETENV} ${CO_ENV} ${SH} ${SCRIPTSDIR}/check-stagedir.sh orphans .endif =20 .if !target(stage-qa) stage-qa: - @${ECHO_CMD} "=3D=3D=3D=3D> Running Q/A tests" ; \ - ${SETENV} ${QA_ENV} ${SH} ${SCRIPTSDIR}/qa.sh + @${ECHO_MSG} "=3D=3D=3D=3D> Running Q/A tests (stage-qa)" + @${SETENV} ${QA_ENV} ${SH} ${SCRIPTSDIR}/qa.sh .endif --------------000705000707070505070604-- --mca21qshKXmb6CmStkrgfUXJoMF24tHuR Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlJxr3sACgkQvmGDOQUufZWX3ACgqejoWJQn858bvfx7lkm0+Mhl 6R8AniaPE0LNsIRI5xxNDZ4OdgVOlnXf =3mvG -----END PGP SIGNATURE----- --mca21qshKXmb6CmStkrgfUXJoMF24tHuR--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5271AF7B.40005>