From owner-freebsd-ports-bugs@FreeBSD.ORG Fri Nov 27 14:10:02 2009 Return-Path: Delivered-To: freebsd-ports-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BD5031065670 for ; Fri, 27 Nov 2009 14:10:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 9BF128FC18 for ; Fri, 27 Nov 2009 14:10:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id nAREA2hb035718 for ; Fri, 27 Nov 2009 14:10:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id nAREA2Hf035717; Fri, 27 Nov 2009 14:10:02 GMT (envelope-from gnats) Resent-Date: Fri, 27 Nov 2009 14:10:02 GMT Resent-Message-Id: <200911271410.nAREA2Hf035717@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-ports-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Eygene Ryabinkin Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CD5AF1065670; Fri, 27 Nov 2009 14:00:48 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from 0.mx.codelabs.ru (0.mx.codelabs.ru [144.206.177.45]) by mx1.freebsd.org (Postfix) with ESMTP id 5C8CE8FC12; Fri, 27 Nov 2009 14:00:48 +0000 (UTC) Received: from void.codelabs.ru (void.codelabs.ru [144.206.177.25]) by 0.mx.codelabs.ru with esmtps (TLSv1:CAMELLIA256-SHA:256) id 1NE1NP-000L3b-F9; Fri, 27 Nov 2009 17:00:47 +0300 Message-Id: <20091127140047.6A690DA81A@void.codelabs.ru> Date: Fri, 27 Nov 2009 17:00:47 +0300 (MSK) From: Eygene Ryabinkin To: FreeBSD-gnats-submit@FreeBSD.org X-Send-Pr-Version: 3.113 Cc: nectar@FreeBSD.org, miwi@FreeBSD.org Subject: ports/140939: [patch] security/vuxml: fix and extend files/newentry.sh X-BeenThere: freebsd-ports-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Eygene Ryabinkin List-Id: Ports bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Nov 2009 14:10:02 -0000 >Number: 140939 >Category: ports >Synopsis: [patch] security/vuxml: fix and extend files/newentry.sh >Confidential: no >Severity: non-critical >Priority: medium >Responsible: freebsd-ports-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Fri Nov 27 14:10:01 UTC 2009 >Closed-Date: >Last-Modified: >Originator: Eygene Ryabinkin >Release: FreeBSD 8.0-RC1 amd64 >Organization: Code Labs >Environment: System: FreeBSD 8.0-RC1 amd64 >Description: There are two problems in the current implementation of newentry.sh: 1. cleanup handler is installed after the temporary file is created, so there is a small window during which the script can receive the signal and temporary file won't be deleted; 2. cleanup handler isn't called when external editor is executed, so temporary file is left in its place. Moreover, since patches to the vuln.xml tend to expire quickly: once the new entry is added to the vuln.xml, the patches that are created for the previous versions won't apply anymore. Since vuln.xml changes rather often, it is desirable to find a way to apply these patches automatically and newentry.sh can be extended to handle it. miwi@ sometimes asks me to redo my diffs for the VuXML entries since they went out-of-date, so this functionality should help in such cases. >How-To-Repeat: Point 1: look at the code: ----- tmp="`mktemp ${TMPDIR:-/tmp}/vuxml.XXXXXXXXXX`" || exit 1 doclean="yes" cleanup() { if [ "${doclean}" = "yes" ]; then rm -f "${tmp}" fi } trap cleanup EXIT 1 2 13 15 ----- Compare with ----- doclean="yes" cleanup() { if [ "${doclean}" = "yes" ]; then [ -n "${tmp}" ] && rm -f "${tmp}" fi } trap cleanup EXIT 1 2 13 15 tmp="`mktemp ${TMPDIR:-/tmp}/vuxml.XXXXXXXXXX`" || exit 1 ----- Point 2: try to invoke 'make newentry' and watch for file(s) vuxml. in the /tmp or $TMPDIR. >Fix: The following patch fixes the point 1: --- 0001-install-cleanup-handler-earlier.diff begins here --- >From 4375203e1602acacfc0ef230fd2cabd42d142463 Mon Sep 17 00:00:00 2001 From: Eygene Ryabinkin Date: Fri, 27 Nov 2009 12:35:17 +0300 There is a gap between 'mktemp' and installation of the trap handler. We can initialize 'doclean' before trapping signals and additionally check if ${tmp} is empty inside the cleanup handler. This will practically eliminate the gap where temporary file won't be removed if the script will receive some signal. Signed-off-by: Eygene Ryabinkin --- security/vuxml/files/newentry.sh | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/security/vuxml/files/newentry.sh b/security/vuxml/files/newentry.sh index b6337da..d08bbc8 100644 --- a/security/vuxml/files/newentry.sh +++ b/security/vuxml/files/newentry.sh @@ -6,14 +6,14 @@ if [ -z "${vuxml_file}" ]; then exit 1 fi -tmp="`mktemp ${TMPDIR:-/tmp}/vuxml.XXXXXXXXXX`" || exit 1 -doclean="yes" cleanup() { if [ "${doclean}" = "yes" ]; then - rm -f "${tmp}" + [ -n "${tmp}" ] && rm -f "${tmp}" fi } +doclean="yes" trap cleanup EXIT 1 2 13 15 +tmp="`mktemp ${TMPDIR:-/tmp}/vuxml.XXXXXXXXXX`" || exit 1 vid="`uuidgen | tr '[:upper:]' '[:lower:]'`" [ -z "$vid" ] && exit 1 -- 1.6.5.3 --- 0001-install-cleanup-handler-earlier.diff ends here --- The following patch fixes the point 2: --- 0002-cleanup-before-spawning-editor.diff begins here --- >From b5ebab7ccd3ac8d484bbe6bc037b94601dc0c7be Mon Sep 17 00:00:00 2001 From: Eygene Ryabinkin Date: Fri, 27 Nov 2009 12:52:07 +0300 Subject: [PATCH 2/4] security/vuxml: newentry.sh: cleanup before executing editor When we do 'exec EDITOR', cleanup handler isn't called, so the temporary file is left in place (/tmp or $TMPDIR). We fix it by explicitely calling cleanup() before exec'ing editor. Signed-off-by: Eygene Ryabinkin --- security/vuxml/files/newentry.sh | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/security/vuxml/files/newentry.sh b/security/vuxml/files/newentry.sh index d08bbc8..f84d0ea 100644 --- a/security/vuxml/files/newentry.sh +++ b/security/vuxml/files/newentry.sh @@ -51,6 +51,7 @@ awk '/^[[:space:]]+> "${tmp}" || exit 1 if cp "${tmp}" "${vuxml_file}"; then + cleanup exec ${EDITOR:-vi} "${vuxml_file}" else doclean="no" -- 1.6.5.3 --- 0002-cleanup-before-spawning-editor.diff ends here --- Two next patches implement the functionality to add existing VuXML entries or single-entry patches for vuln.xml automatically. First patch refactors newentry.sh to place currently spreaded code for creation of a new entry to a single function. Second patch implements new functionality. --- 0003-refactor-newentry.sh.diff begins here --- >From 277a3c98c2449fc9c965ba3d8a46a6e78d4db5f0 Mon Sep 17 00:00:00 2001 From: Eygene Ryabinkin Date: Fri, 27 Nov 2009 13:11:58 +0300 Subject: [PATCH 3/4] security/vuxml: refactor newentry.sh Make new procedure, newentry(), that creates new VuXML entry, thus incapsulating the needed functionality into the single function. Main code just calls it when needed. Also make usage() to be stand-alone function. Signed-off-by: Eygene Ryabinkin --- security/vuxml/files/newentry.sh | 54 ++++++++++++++++++++++++-------------- 1 files changed, 34 insertions(+), 20 deletions(-) diff --git a/security/vuxml/files/newentry.sh b/security/vuxml/files/newentry.sh index f84d0ea..f2b9e61 100644 --- a/security/vuxml/files/newentry.sh +++ b/security/vuxml/files/newentry.sh @@ -1,27 +1,17 @@ #! /bin/sh -vuxml_file="$1" -if [ -z "${vuxml_file}" ]; then - exec >&2 - echo "Usage: newentry.sh /path/to/vuxml/document" - exit 1 -fi -cleanup() { - if [ "${doclean}" = "yes" ]; then - [ -n "${tmp}" ] && rm -f "${tmp}" - fi -} -doclean="yes" -trap cleanup EXIT 1 2 13 15 -tmp="`mktemp ${TMPDIR:-/tmp}/vuxml.XXXXXXXXXX`" || exit 1 +new_entry () { + local vid discovery entry -vid="`uuidgen | tr '[:upper:]' '[:lower:]'`" -[ -z "$vid" ] && exit 1 -discovery="`date -u '+%Y-%m'`-FIXME" || exit 1 -entry="`date -u '+%Y-%m-%d'`" || exit 1 + vid="`uuidgen | tr '[:upper:]' '[:lower:]'`" + discovery="`date -u '+%Y-%m'`-FIXME" || exit 1 + entry="`date -u '+%Y-%m-%d'`" || exit 1 + if [ -z "$vid" -o -z "$discovery" -o -z "$entry" ]; then + false + return + fi -awk '/^<\?/,/^> "${tmp}" || exit 1 -cat << EOF >> "${tmp}" || exit 1 + cat << EOF -- @@ -47,6 +37,30 @@ cat << EOF >> "${tmp}" || exit 1 EOF +} + +usage () { + echo "Usage: newentry.sh /path/to/vuxml/document" >&2 +} + +cleanup() { + if [ "${doclean}" = "yes" ]; then + [ -n "${tmp}" ] && rm -f "${tmp}" + fi +} + +vuxml_file="$1" +if [ -z "${vuxml_file}" ]; then + usage + exit 1 +fi + +doclean="yes" +trap cleanup EXIT 1 2 13 15 +tmp="`mktemp ${TMPDIR:-/tmp}/vuxml.XXXXXXXXXX`" || exit 1 + +awk '/^<\?/,/^> "${tmp}" || exit 1 +new_entry >> "${tmp}" || exit 1 awk '/^[[:space:]]+> "${tmp}" || exit 1 -- 1.6.5.3 --- 0003-refactor-newentry.sh.diff ends here --- --- 0004-teach-to-accept-pre-made-entries.diff begins here --- >From 94dd35b3f635ef665426c4b339fe69982a8fb392 Mon Sep 17 00:00:00 2001 From: Eygene Ryabinkin Date: Fri, 27 Nov 2009 15:06:09 +0300 Subject: [PATCH 4/4] security/vuxml: teach newentry.sh to accept existing entries Now newentry.sh can add not only generic template for the new entry, but existing entries as well. Both plain entries and patches are accepted. This mode was created mainly due to the fact that vuln.xml changes frequently and if one sends patch for the obsolete version of vuln.xml, the patch won't be applied cleanly and manual work is needed. Script automates such insertions. Moreover, plain VuXML entries that are not patches, but are just pieces of vuln.xml can be sent via PRs -- they are inserted automatically too. 'make addentry E=/path/to/entry/file' will do the work -- Makefile was extended to handle the new target. Signed-off-by: Eygene Ryabinkin --- security/vuxml/Makefile | 6 ++ security/vuxml/files/newentry.sh | 115 +++++++++++++++++++++++++++++++++++++- 2 files changed, 119 insertions(+), 2 deletions(-) diff --git a/security/vuxml/Makefile b/security/vuxml/Makefile index d347c39..8aa177e 100644 --- a/security/vuxml/Makefile +++ b/security/vuxml/Makefile @@ -83,4 +83,10 @@ tidy: newentry: @${SH} ${FILESDIR}/newentry.sh "${VUXML_FILE}" +addentry: +.if empty(E) +.error "Set make variable E to the location of the VuXML entry to be added." +.endif + @${SH} ${FILESDIR}/newentry.sh -i "${E}" "${VUXML_FILE}" + .include diff --git a/security/vuxml/files/newentry.sh b/security/vuxml/files/newentry.sh index f2b9e61..d546a79 100644 --- a/security/vuxml/files/newentry.sh +++ b/security/vuxml/files/newentry.sh @@ -1,5 +1,87 @@ #! /bin/sh +# Patch detection strategy: we are looking for the three successive +# lines starting with "--- ", "+++ " and "@@ [+-]". The next line +# should start with space. This catches unified diffs. +# +# The number of the line where the patch contents start (the line with +# first '@@ [+-]') is printed to the standard output if the patch was +# really found. +is_patch () { + awk ' +BEGIN { seen_minus = 0; seen_plus = 0; seen_at = 0; diff = 0 } +/^--- / { seen_minus = 1; seen_plus = 0; seen_at = 0; diff = 0; next; } +seen_minus == 1 && /^\+\+\+ / { seen_plus = 1; next; } +seen_plus == 1 && /^@@ [+-]/ { seen_at = 1; next; } +seen_at == 1 && /^ / { print FNR; diff = 1; exit(0); } +{ seen_minus = 0; seen_plus = 0; seen_at = 0; } +END { exit (diff != 1); } +' "$1" +} + +# Checks if the patch contains one continuous hunk that contains +# only additions. +# Arguments: +# $1 -- patch filename; +# $2 -- line where patch contents start. +is_continuous_patch () { + + if [ -z "$1" -o -z "$2" ]; then + false + return + fi + awk -v start="$2" ' +BEGIN { blks = 0; in_blk = 0; } +FNR < start { next; } +/^-/ { exit(1); } +in_blk == 0 && /^+/ { in_blk = 1; blks++; if (blks > 1) { exit(1); } next; } +in_blk == 1 && !/^+/ { in_blk = 0; next; } +END { exit(blks != 1); }; +' "$1" +} + +# $1 is the file with the existing VuXML entry; it is guaranteed +# to be a real file. +# +# We should check if the supplied file contains a patch that has +# one continuous entry that is meant to be added -- only such files +# are treated as patches that add VuXML entries. If patch has more +# than one chunk or it deletes anything, we won't accept it. +# +# For the non-patch files, we will add them "as is". + +existing_entry () { + local patch_start + + if [ -z "$1" ]; then + false + return + fi + + # Plain file? Just cat it! + patch_start=`is_patch "$1"` + if [ "$?" != 0 ]; then + cat "$1" + echo + return + fi + + if is_continuous_patch "$1" "$patch_start"; then + awk -v start="$patch_start" ' +FNR < start { next; } +{ print; } +' "$1" | grep '^\+' | sed -e's/^\+//' + else + cat >&2 << EOF +Input VuXML entry looks like a patch, but it doesn't contain a single +hunk that I can use as the entry body. Human insight is needed, may +be it is better to merge the entry by hand. +EOF + false + return + fi +} + new_entry () { local vid discovery entry @@ -40,7 +122,7 @@ EOF } usage () { - echo "Usage: newentry.sh /path/to/vuxml/document" >&2 + echo "Usage: newentry.sh [-i /existing/entry] /path/to/vuxml/document" >&2 } cleanup() { @@ -49,6 +131,31 @@ cleanup() { fi } +while getopts "i:h" opt; do + case "$opt" in + h) + usage + exit 0 + ;; + i) + if [ ! -e "$OPTARG" ]; then + echo "File '$OPTARG' does not exist." >&2 + exit 1 + fi + entry="$OPTARG" + ;; + ?) + usage + exit 1 + ;; + -) + break + ;; + esac +done + +shift $(($OPTIND - 1)) + vuxml_file="$1" if [ -z "${vuxml_file}" ]; then usage @@ -60,7 +167,11 @@ trap cleanup EXIT 1 2 13 15 tmp="`mktemp ${TMPDIR:-/tmp}/vuxml.XXXXXXXXXX`" || exit 1 awk '/^<\?/,/^> "${tmp}" || exit 1 -new_entry >> "${tmp}" || exit 1 +if [ -n "$entry" ]; then + existing_entry "$entry" >> "${tmp}" || exit 1 +else + new_entry >> "${tmp}" || exit 1 +fi awk '/^[[:space:]]+> "${tmp}" || exit 1 -- 1.6.5.3 --- 0004-teach-to-accept-pre-made-entries.diff ends here --- To ease the things, patches are mirrored at the following locations: http://codelabs.ru/fbsd/patches/vuxml/newentry.sh/0001-install-cleanup-handler-earlier.diff http://codelabs.ru/fbsd/patches/vuxml/newentry.sh/0002-cleanup-before-spawning-editor.diff http://codelabs.ru/fbsd/patches/vuxml/newentry.sh/0003-refactor-newentry.sh.diff http://codelabs.ru/fbsd/patches/vuxml/newentry.sh/0004-teach-to-accept-pre-made-entries.diff >Release-Note: >Audit-Trail: >Unformatted: