From owner-freebsd-ports@FreeBSD.ORG  Wed Dec 29 00:31:11 2010
Return-Path: <owner-freebsd-ports@FreeBSD.ORG>
Delivered-To: freebsd-ports@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 6A0AE1065670
	for <freebsd-ports@freebsd.org>; Wed, 29 Dec 2010 00:31:11 +0000 (UTC)
	(envelope-from dougb@FreeBSD.org)
Received: from mail2.fluidhosting.com (mx23.fluidhosting.com [204.14.89.6])
	by mx1.freebsd.org (Postfix) with ESMTP id 0FFE18FC12
	for <freebsd-ports@freebsd.org>; Wed, 29 Dec 2010 00:31:10 +0000 (UTC)
Received: (qmail 23551 invoked by uid 399); 29 Dec 2010 00:31:09 -0000
Received: from localhost (HELO doug-optiplex.ka9q.net)
	(dougb@dougbarton.us@127.0.0.1)
	by localhost with ESMTPAM; 29 Dec 2010 00:31:09 -0000
X-Originating-IP: 127.0.0.1
X-Sender: dougb@dougbarton.us
Message-ID: <4D1A814C.7000705@FreeBSD.org>
Date: Tue, 28 Dec 2010 16:31:08 -0800
From: Doug Barton <dougb@FreeBSD.org>
Organization: http://SupersetSolutions.com/
User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US;
	rv:1.9.2.13) Gecko/20101210 Thunderbird/3.1.7
MIME-Version: 1.0
To: Eygene Ryabinkin <rea@freebsd.org>
References: <4D15D275.6000308@gmail.com> <4D194421.9080304@FreeBSD.org>
	<SaPpQ++iANujx2z50Bs5SbHA8lc@QsmfhJNucgI88DfvPJdT1/nyboE>
	<4D1A3288.70604@FreeBSD.org>
	<+pyrRd3Y7AElxrhYeeViR6NWT78@QsmfhJNucgI88DfvPJdT1/nyboE>
In-Reply-To: <+pyrRd3Y7AElxrhYeeViR6NWT78@QsmfhJNucgI88DfvPJdT1/nyboE>
X-Enigmail-Version: 1.1.2
OpenPGP: id=1A1ABC84
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: jhell <jhell@DataIX.net>, Peter Pentchev <roam@ringlet.net>,
	freebsd-ports@freebsd.org
Subject: Re: portmaster: print /usr/ports/UPDATING on update
X-BeenThere: freebsd-ports@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Porting software to FreeBSD <freebsd-ports.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-ports>,
	<mailto:freebsd-ports-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-ports>
List-Post: <mailto:freebsd-ports@freebsd.org>
List-Help: <mailto:freebsd-ports-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-ports>,
	<mailto:freebsd-ports-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 29 Dec 2010 00:31:11 -0000

On 12/28/2010 12:31, Eygene Ryabinkin wrote:
> Doug, good day.
>
> Tue, Dec 28, 2010 at 10:55:04AM -0800, Doug Barton wrote:
>> Do either of you actually have any familiarity at all with how portaudit
>> works, and/or how it is integrated into the ports infrastructure? Based
>> on what you've written today my guess is "no."
>
> I am sorry, but you're wrong here: I am familiar with portaudit and
> its internals:

Excellent!

> But the portaudit has very precise semantics: "if you have the port X
> with versions in<range>, then it is vulnerable, here is the link to
> the advisory".

Please note, I did not say "Must be exactly like portaudit." Obviously 
we need to think about the requirements, and design accordingly.

> Let me explain my worries again.
>
> What I meant is the following: one can use XML/JSON/SomeOtherMarkup
> for the UPDATING.ng.  But why bother if the current UPDATING format
> can be slightly extended and semantics can be attached to it without
> using the new fancy languages that will need some specific ports to be
> installed in order to process them?

First, from the user side we're talking about one port, the one to 
display the information. Users of portaudit do not need to have xml 
installed. Second, the "why?" is so that we can make it easier to deal 
with programmatically, and hopefully as a result to extend its 
functionality.

> I read your argument about
> reusing the VuXML machinery; it is addressed below.
>
> You write that
>>  From the user side, we're not really losing anything by not having
>> "human readable" output readily available, since 99% of users will
>> just want to be able to know what entries are relevant to their
>> installation anyway.
> but do you know how many users rely on the current UPDATING format?

Humorous answer, "not enough." Hopefully useful answer, at this point, 
100% of them, since that's the only alternative. Hopefully _more_ useful 
answer, I think very very few (as in, single digit percentages) would 
even bother to look at a text UPDATING file if they could get the answer 
to "is there anything in there that affects me?" with one quick command 
line.

> Personally, I -- don't, so I can't make such assertions with
> confidence.  Surely, we can have a tool that will output all entries
> in the current UPDATING format -- it will save everyone who relies on
> the current state of this file.

Not to belabor the point, but I specifically said that I believe having 
this ability would be a requirement for the new tool.

> The XML in VuXML (that is used to create the portaudit database and
> entries at http://www.vuxml.org/freebsd/) is mainly needed because
> VuXML entries use HTML markup, so it is just easier to allow the whole
> <body>  tag to contain XHTML, because it can contain links, references,
> lists and other markup.  So, the nested structure of XML fits here
> rather good.  But, if you will suppress the complex<body>  structure,
> you can reformat the VuXML into the pseudo-RFC822 format and write
> a simple validator for it.  XML has XSLT that can be used for transform
> into HTML, so it is another reason why VuXML should be XML, but again,
> mainly the body content plays its role here.  The<affects>  structure
> can be expressed via the RFC822-like headers -- there is no problem
> to parse and validate it.

You bring up a very good point here that someone in #bsdports also 
mentioned, that having UPDATING in xml would allow us to easily produce 
an HTML version of the file for use on the web site.

> Current UPDATING is a lot simpler: it already has the pseudo-RFC822
> form and it is perfectly human-readable.  What we need is the good
> AFFECTS structure with clear semantics and validator for it.  I am not
> saying that we can't XML here, I am just pointing out that it can be
> done without XML and, thus, we don't need the dependencies on the
> XML/DTD/XSLT stuff at all.

And I'm saying that even with the current, limited functionality that we 
expect from UPDATING that a text version parser is a non-starter. So, if 
we're going to need to create both a structure and a parser then we are 
much better off re-using existing tools, knowledge, and infrastructure 
to do that.

> So, in my opinion, if I'll weight the pros (existing tools, standard
> validation) and cons (throws a lot of tags to the reader, unsure how
> to keep the current form of UPDATING's distribution)

Both of those "cons" are totally invalid. The consumers of the portaudit 
data never look at the xml form, nor would the consumers of the data for 
the tool I'm proposing. I've also specified that the tool should be able 
to output a text format of the file.

> Thinking about portaudit, UPDATING in this form will be more-or-less
> equal to the auditfile.  And currently, UPDATING has all important
> properties of an auditfile, but one: AFFECTS have no clear syntax and
> semantics (and UPDATING has slightly other format, but it is
> consistent throughout the whole file, so it really doesn't matter).
>
>
> About reusing the current VuXML machinery for UPDATING:
>
>   - there will be just another schema for UPDATING, because VuXML is
>     created for security vulnerabilities;

No argument there.

>   - auditfile format isn't well-suited for the UPDATING, just
>     because auditfile delegates the entry themselves to the
>     Web server hosting the HTML'ized entries, but UPDATING should
>     have the entry bodies available locally;

My goal would be to have a text output format that includes the text and 
the link, and an HTML output format that creates a proper anchor tag. 
But that's an implementation detail.

>   - semantics of the UPDATING entries is different, so existing
>     portaudit machinery should be rewritten: we can either create
>     the complex tool to handle VuXML and UPDATING or to have two
>     distinct tools that, perhaps, will share some code via the
>     libpkg.

I'm not sure where you're going with this.

> And here comes the next question: what syntax of AFFECTS we will need
> and what semantics we will apply here.
[ ... ]
 > I think that the syntax/semantics problem is more-or-less orthogonal
 > to the matters of the UPDATING source language and tools that are used
 > to create/maintain it,

I'm snipping your particulars here because we're in agreement that the 
semantics will have to be well thought out, but the first question to 
answer is "how?" not "what."

> Since I am trying to push the pkg_audit tool (that will intersect the
> currently-installed ports and auditfile to give the caller that set of
> VuXML entries that are applicable to his ports) to the base, I am
> more-or-less familiar of a business of writing such a tool, so I am
> taking the responsibility to write it once (and if) we will agree on
> how to move UPDATING to the next stage of its life.  And I'll try
> to make the architectural stage of these changes to be alive too.

My concern here is that more than one individual is taking the 
opportunity of this thread to push their own solution, and attempting to 
shape the problem to look like their solution is the right one. For 
better or worse, I have no dog in this hunt, I just want to see 
something better than what we currently have.


Doug

-- 

	Nothin' ever doesn't change, but nothin' changes much.
			-- OK Go

	Breadth of IT experience, and depth of knowledge in the DNS.
	Yours for the right price.  :)  http://SupersetSolutions.com/