From owner-freebsd-arch@FreeBSD.ORG Sun Jul 27 02:29:12 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1A33021D for ; Sun, 27 Jul 2014 02:29:12 +0000 (UTC) Received: from mail-pd0-f182.google.com (mail-pd0-f182.google.com [209.85.192.182]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E5C08293A for ; Sun, 27 Jul 2014 02:29:11 +0000 (UTC) Received: by mail-pd0-f182.google.com with SMTP id fp1so7839218pdb.27 for ; Sat, 26 Jul 2014 19:29:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=VfvFDdNPx0/zajT4dhWhJjxUIAKNiDLl1XpFeL7YX/A=; b=K16Y+HAeZXP0DesDXv7xKlgMe9isRrDUnoVVuhC49bIdi5BfHpko4aS81YtTNycqd+ mn7G0t6LRLp8tylU5Rj7eB1g6Ry57ULuCmlmyKp1KDchf7hFWkEQgUqMMnLL3jo/Md4F VlK3LHyCJb/x80KjCd6GLaywSHikL+NVfcLtOfd1VbSSmJEDMNxSNclhtqarctwmQ5B1 /98ePBduV13i2v+aoYwPrvE/mhmXMmdR1uI3LycgQmBumljl24j//uB3H3aiqULTwpP5 DYMnt6CJgdZbj/0CyV2izHUJ2jWNcCKD5DelELlVeH0fluqh7uBOEfZ6cMMlxcJzY2Gy 1FAw== X-Gm-Message-State: ALoCoQnRUgU9p/b7uzfTwzzVUzjvdzbARDsgEZdoUSEJNoMuO9IvHi7wUlVeJza8qrcO2L6M+kyH MIME-Version: 1.0 X-Received: by 10.66.161.69 with SMTP id xq5mr28814247pab.62.1406428145181; Sat, 26 Jul 2014 19:29:05 -0700 (PDT) Received: by 10.70.101.41 with HTTP; Sat, 26 Jul 2014 19:29:04 -0700 (PDT) Received: by 10.70.101.41 with HTTP; Sat, 26 Jul 2014 19:29:04 -0700 (PDT) In-Reply-To: <20140725044921.9F0D3580A2@chaos.jnpr.net> References: <20140725044921.9F0D3580A2@chaos.jnpr.net> Date: Sat, 26 Jul 2014 19:29:04 -0700 Message-ID: Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Jos Backus To: arch@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Jul 2014 02:29:12 -0000 It's a little sad to see that the more human-friendly and expressive YAML format appears to not be supported. Instead, here too we are stuck with JSON, the VHS of serialization formats. Better than nothing, I suppose. Jos From owner-freebsd-arch@FreeBSD.ORG Sun Jul 27 23:01:16 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5DB85BD4; Sun, 27 Jul 2014 23:01:16 +0000 (UTC) Received: from mail-wi0-x231.google.com (mail-wi0-x231.google.com [IPv6:2a00:1450:400c:c05::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C832321D0; Sun, 27 Jul 2014 23:01:15 +0000 (UTC) Received: by mail-wi0-f177.google.com with SMTP id ho1so3460097wib.4 for ; Sun, 27 Jul 2014 16:01:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:subject:message-id:mime-version:content-type :content-disposition:user-agent; bh=uZ7PsdhBTP40iUYalIV3uytvS1S9dSjU1eBF1Y0IK80=; b=RM7TR8Lxldggbp3zX2BONQKLKZZ6f+mJitvzVq1AIjMkDhPZr+G2eisB73VB/T/xGe j/tI5Bq8ytyoSWrWIDjOE/mDikgTAwsf/Tvpi8tQYKYwmRKS1K9acR8/eZkS3Ricza85 TnAokkSrWvDAuWI0D38m056VR1rx6Bhby1UtVEbJP6zYDCxlzT48dPqCE7htWLSgfuSw zf7faYhaNTYKmyGU26tdlNprU1Q5yxuGKPxXLlHEhoXaHtfr3uPYF2G02o5/PdcAMoE2 gA3Yphfb/l11bSzSm/ZN0VPz366DKvu1b+MKTj+i68fw8wJbTa6ktMYj+seMbitBF1JN RE6w== X-Received: by 10.180.39.34 with SMTP id m2mr24056526wik.80.1406502073980; Sun, 27 Jul 2014 16:01:13 -0700 (PDT) Received: from ivaldir.etoilebsd.net ([2001:41d0:8:db4c::1]) by mx.google.com with ESMTPSA id ex2sm24211194wic.24.2014.07.27.16.01.12 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 27 Jul 2014 16:01:13 -0700 (PDT) Sender: Baptiste Daroussin Date: Mon, 28 Jul 2014 01:01:10 +0200 From: Baptiste Daroussin To: arch@FreeBSD.org, hackers@FreeBSD.org Subject: Add ohash (OpenBSD hash) into libutil Message-ID: <20140727230110.GI50802@ivaldir.etoilebsd.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ZY5CS28jBCfb727c" Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Jul 2014 23:01:16 -0000 --ZY5CS28jBCfb727c Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi, If noone objects I would like to bring ohash from OpenBSD into our libutil. Ohash is already in base (usr.bin/m4/ohash) having it into libutil will increase compatibility with OpenBSD as well as allowing the rest of the base system to be able to benefit from ohash implementation. regards, Bapt --ZY5CS28jBCfb727c Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlPVhLYACgkQ8kTtMUmk6ExTPwCdE4dr6/QSrXummgrQPNDYwYWd 3jEAoL19AKYlgDDe1bKupWA3T3DpmuSO =fK29 -----END PGP SIGNATURE----- --ZY5CS28jBCfb727c-- From owner-freebsd-arch@FreeBSD.ORG Mon Jul 28 05:42:23 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 513A854D for ; Mon, 28 Jul 2014 05:42:23 +0000 (UTC) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2lp0235.outbound.protection.outlook.com [207.46.163.235]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0838C1EF1 for ; Mon, 28 Jul 2014 05:42:22 +0000 (UTC) Received: from CO2PR05CA029.namprd05.prod.outlook.com (10.141.241.157) by BLUPR05MB722.namprd05.prod.outlook.com (10.141.207.150) with Microsoft SMTP Server (TLS) id 15.0.995.14; Mon, 28 Jul 2014 05:42:21 +0000 Received: from BN1BFFO11FD044.protection.gbl (2a01:111:f400:7c10::1:104) by CO2PR05CA029.outlook.office365.com (2a01:111:e400:1429::29) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Mon, 28 Jul 2014 05:42:20 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BN1BFFO11FD044.mail.protection.outlook.com (10.58.144.107) with Microsoft SMTP Server (TLS) id 15.0.990.10 via Frontend Transport; Mon, 28 Jul 2014 05:42:20 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Sun, 27 Jul 2014 22:42:18 -0700 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6S5gHn70511; Sun, 27 Jul 2014 22:42:17 -0700 (PDT) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id AC1A0580A2; Sun, 27 Jul 2014 22:42:17 -0700 (PDT) To: Jos Backus Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: References: <20140725044921.9F0D3580A2@chaos.jnpr.net> Comments: In-reply-to: Jos Backus message dated "Sat, 26 Jul 2014 19:29:04 -0700." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Sun, 27 Jul 2014 22:42:17 -0700 Message-ID: <20140728054217.AC1A0580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(199002)(189002)(57986006)(104166001)(81342001)(77982001)(102836001)(76506005)(33656002)(93546004)(101356003)(21056001)(50466002)(97736001)(81542001)(64706001)(86362001)(107046002)(20776003)(47776003)(110136001)(76176999)(48376002)(74502001)(80022001)(50986999)(74662001)(70486001)(99396002)(31966008)(46102001)(76482001)(62966002)(77156001)(83322001)(105596002)(87286001)(50226001)(81156004)(4396001)(106466001)(69596002)(79102001)(84676001)(102176002)(85306003)(6806004)(87936001)(44976005)(92726001)(95666004)(89996001)(92566001)(90896003)(83072002)(85852003)(42262001); DIR:OUT; SFP:; SCL:1; SRVR:BLUPR05MB722; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0286D7B531 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=sjg@juniper.net; X-OriginatorOrg: juniper.net Cc: arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jul 2014 05:42:23 -0000 On Sat, 26 Jul 2014 19:29:04 -0700, Jos Backus writes: >It's a little sad to see that the more human-friendly and expressive YAML >format appears to not be supported. Instead, here too we are stuck with Is there a use case for something like vmstat outputting YAML? It is a simple format, I guess it could be added, especially if it has no format quirks worse that HTML and JSON. Note: I'm just speculating. From owner-freebsd-arch@FreeBSD.ORG Mon Jul 28 05:53:43 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1F47B72D for ; Mon, 28 Jul 2014 05:53:43 +0000 (UTC) Received: from mail-wi0-x234.google.com (mail-wi0-x234.google.com [IPv6:2a00:1450:400c:c05::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A8127201F for ; Mon, 28 Jul 2014 05:53:42 +0000 (UTC) Received: by mail-wi0-f180.google.com with SMTP id n3so3759709wiv.1 for ; Sun, 27 Jul 2014 22:53:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=6IC+o15CyE4Mv5sFfnSSRm4HbaF9K9aA9jPkdj0v+PE=; b=f2a761Arm/ya7qNPYbYwQIesm+fIR2+TZnO9QTdly7QJDj8kNM5x9qdmI7wP2KFy1B +gvrZHg2G2MVz6hZi6CTQ2dlsEitQNN9BYMxr7STTkm4ZbH0SnZ1c+TdG0cSrziYFzZZ YFadK2WFvg0Kvxz5mWko59XOK9rSgR9TlaERPC4GJSkvgxWUWkVJ6IlivXzEjcNrBCNX /olQcrGkeetZE+Vlqet2pMgOmQLsoXhYRmpK+RdTgf676e9bjHLrmTTXd2lASo3BAGEK qjt01kK/4RjlR5AwhQrYQxrrwQeYe6sPMCP2niZmCXfZzaElgkzMnZcCwQMFTUKd0YqD qbCA== X-Received: by 10.194.63.228 with SMTP id j4mr46249053wjs.7.1406526820159; Sun, 27 Jul 2014 22:53:40 -0700 (PDT) Received: from ivaldir.etoilebsd.net ([2001:41d0:8:db4c::1]) by mx.google.com with ESMTPSA id p3sm46874589wjw.13.2014.07.27.22.53.38 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 27 Jul 2014 22:53:39 -0700 (PDT) Sender: Baptiste Daroussin Date: Mon, 28 Jul 2014 07:53:37 +0200 From: Baptiste Daroussin To: "Simon J. Gerraty" Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Message-ID: <20140728055336.GJ50802@ivaldir.etoilebsd.net> References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="zOcTNEe3AzgCmdo9" Content-Disposition: inline In-Reply-To: <20140728054217.AC1A0580A2@chaos.jnpr.net> User-Agent: Mutt/1.5.23 (2014-03-12) Cc: arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jul 2014 05:53:43 -0000 --zOcTNEe3AzgCmdo9 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Jul 27, 2014 at 10:42:17PM -0700, Simon J. Gerraty wrote: >=20 > On Sat, 26 Jul 2014 19:29:04 -0700, Jos Backus writes: > >It's a little sad to see that the more human-friendly and expressive YAML > >format appears to not be supported. Instead, here too we are stuck with >=20 > Is there a use case for something like vmstat outputting YAML? > It is a simple format, I guess it could be added, especially if it has > no format quirks worse that HTML and JSON. > Note: I'm just speculating. YAML is all but a simple format, creating a subset of YAML that is good eno= ugh is a simple format :), exporting a complete YAML (strongly typed etc) is something else. As machine readable format YAML is a nightmare to parse, JSON is very simpl= er and easier (while json being a valid subject of YAML). regards, Bapt --zOcTNEe3AzgCmdo9 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlPV5WAACgkQ8kTtMUmk6EyfugCfWrlwpOwBZjZ/jCgEz5k2qm1V RjAAnA9RdYqV7tewWGMoiKbmyc1I/r9c =MAOM -----END PGP SIGNATURE----- --zOcTNEe3AzgCmdo9-- From owner-freebsd-arch@FreeBSD.ORG Mon Jul 28 06:07:05 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AC36EB8B for ; Mon, 28 Jul 2014 06:07:05 +0000 (UTC) Received: from mail-pa0-f43.google.com (mail-pa0-f43.google.com [209.85.220.43]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 81E0E214D for ; Mon, 28 Jul 2014 06:07:05 +0000 (UTC) Received: by mail-pa0-f43.google.com with SMTP id lf10so9867171pab.2 for ; Sun, 27 Jul 2014 23:06:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=YdOCZ0nl3umXAUuF9MKMd40LedwHNE7RGeBESD1ZUjU=; b=iOoC2JjY+K7O2HNejTg58/RQ6X9mblG4taJnP8ellRdiNZ29y6wKEnYDv6SnsVHH6B lVNJh0YNeOOc0RWdekYKdd4LdZFKmADl6yItGoy1J9fOI1hUAUR2OS6HOr06n/Jsrot8 EhbbXdX6Auj9XQHTzwQ8cyf9wWObXoyc1vXvFKrzZKj/54FKnCwXU90cnjJb72s8a4xb MZd44q8SIGy4dtLnCoAnb1MKqHQ+TJ3xcnvIG7aNaYczbow6fo82LDQZYY3Fz9aFIWbO /ZjKkOpK7kgt/xTWZPl8M46AODYB5hVKacJqx5rmLl7P29ifPN9upIMdXcL7nKHVRnCN 2g/g== X-Gm-Message-State: ALoCoQm0KuU0zcrPjP3By4YiurmHd9ig5VD8GzQL2I1E8NNZtPZGYRIGK51eCJ4MZnxY5SKGd05W MIME-Version: 1.0 X-Received: by 10.66.180.98 with SMTP id dn2mr29782798pac.83.1406527616756; Sun, 27 Jul 2014 23:06:56 -0700 (PDT) Received: by 10.70.101.41 with HTTP; Sun, 27 Jul 2014 23:06:56 -0700 (PDT) Received: by 10.70.101.41 with HTTP; Sun, 27 Jul 2014 23:06:56 -0700 (PDT) In-Reply-To: <20140728055336.GJ50802@ivaldir.etoilebsd.net> References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> Date: Sun, 27 Jul 2014 23:06:56 -0700 Message-ID: Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Jos Backus To: arch@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jul 2014 06:07:05 -0000 On Jul 27, 2014 10:53 PM, "Baptiste Daroussin" wrote: > > On Sun, Jul 27, 2014 at 10:42:17PM -0700, Simon J. Gerraty wrote: > > > > On Sat, 26 Jul 2014 19:29:04 -0700, Jos Backus writes: > > >It's a little sad to see that the more human-friendly and expressive YAML > > >format appears to not be supported. Instead, here too we are stuck with > > > > Is there a use case for something like vmstat outputting YAML? > > It is a simple format, I guess it could be added, especially if it has > > no format quirks worse that HTML and JSON. > > Note: I'm just speculating. > > YAML is all but a simple format, creating a subset of YAML that is good enough > is a simple format :), exporting a complete YAML (strongly typed etc) is > something else. The full power/expressiveness of YAML may not be needed, we may just want the right subset. And there may be cases where the extra expressiveness is useful as more utilities are converted. > > As machine readable format YAML is a nightmare to parse, JSON is very simpler > and easier (while json being a valid subject of YAML). Well, the work has been done already (libyaml), so barring any bugs and maintenance it's not an issue, no? It would be great if libyaml and libucl would converge, but instead it's likely that the number of solutions trying to solve the same problem will continue to proliferate, and we are stuck with more and more configuration file formats :-( Jos > > regards, > Bapt From owner-freebsd-arch@FreeBSD.ORG Tue Jul 29 04:49:17 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6B8A13D7 for ; Tue, 29 Jul 2014 04:49:17 +0000 (UTC) Received: from mail.iXsystems.com (newknight.ixsystems.com [206.40.55.70]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 44EAF22C2 for ; Tue, 29 Jul 2014 04:49:16 +0000 (UTC) Received: from localhost (mail.ixsystems.com [10.2.55.1]) by mail.iXsystems.com (Postfix) with ESMTP id 6FFF77C7D9; Mon, 28 Jul 2014 21:49:15 -0700 (PDT) Received: from mail.iXsystems.com ([10.2.55.1]) by localhost (mail.ixsystems.com [10.2.55.1]) (maiad, port 10024) with ESMTP id 25929-10; Mon, 28 Jul 2014 21:49:15 -0700 (PDT) Received: from [10.8.0.10] (unknown [10.8.0.10]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.iXsystems.com (Postfix) with ESMTPSA id 28D937C7D6; Mon, 28 Jul 2014 21:49:13 -0700 (PDT) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Jordan Hubbard In-Reply-To: Date: Mon, 28 Jul 2014 21:49:08 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> To: Jos Backus X-Mailer: Apple Mail (2.1878.6) Cc: arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jul 2014 04:49:17 -0000 On Jul 27, 2014, at 11:06 PM, Jos Backus wrote: > It would be great if libyaml and libucl would converge, but instead = it's > likely that the number of solutions trying to solve the same problem = will > continue to proliferate, and we are stuck with more and more = configuration > file formats :-( I=92m a huge fan of unified data formats; Apple picked XML and the plist = DTD a long time ago, a decision which has worked rather nicely in = practice, but I=92m more in love with the unification that produced than = I am in love with XML itself. That said, it seems like this late push = for YAML is a similar case for divergence just because=85erm=85 you = don=92t like JSON? It seems like libucl has basically backed JSON with = the addition of a little syntactic sugar, so what=92s wrong with that? Is there some reason JSON is not sufficient? I think that=92s a better = question to ask, since the conversation otherwise quickly tends to sound = a little like =93I=92ll accept any single unified format as long as it=92s= the specific one I like!=94 :) I think the greater good argument = would suggest just picking one that=92s expressive enough (roll a pair = of dice), put on your bikeshed-proof sunglasses, and proceed. - Jordan From owner-freebsd-arch@FreeBSD.ORG Tue Jul 29 05:18:44 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CB8E48B5 for ; Tue, 29 Jul 2014 05:18:44 +0000 (UTC) Received: from mail-pa0-f52.google.com (mail-pa0-f52.google.com [209.85.220.52]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9DFD425D5 for ; Tue, 29 Jul 2014 05:18:44 +0000 (UTC) Received: by mail-pa0-f52.google.com with SMTP id bj1so11797920pad.11 for ; Mon, 28 Jul 2014 22:18:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=vjSaFQ3Q2nCeV5sxlXHqytT86dIjMGVrpUNGrX5JWns=; b=Czz0So76UKbmEakDk2pxzxItXKnelkGei+r6ipNrnGg5JgE5SbOZkLW5JNZKAI2f8T 8Y0T4bTVZXos6Yrt++6FMvIxBjNaueJfFhr+jNcLPcAZXDGOz61x5Lvuk+d4gezPhHlt GwDQwHuTH5macaMP/rl0dgB7euUw77PPckfMf/CmrX7CBPDIGpDKC12QD79eF7FEFpN6 E9zuVAOGRaAekGNMzHQ/RHX309UhfCKdut3LI0n+QwHh1rHtfbCxBADBMXEBEuw/nN/d qpM1xnh0KyEHE7EK+sP8pHXm5ihXwa+BplWXmazJeCoZycNRXn398KMfarwjFcIA5cIB 4yAg== X-Gm-Message-State: ALoCoQmFZWkVyweTAqPl6TWUBECjUS2oN9DgF2d5fnu8hrdEMB8NBQkKdehiywGS2K15nxDJdZgX MIME-Version: 1.0 X-Received: by 10.68.65.36 with SMTP id u4mr4437711pbs.127.1406611118107; Mon, 28 Jul 2014 22:18:38 -0700 (PDT) Received: by 10.70.101.41 with HTTP; Mon, 28 Jul 2014 22:18:38 -0700 (PDT) Received: by 10.70.101.41 with HTTP; Mon, 28 Jul 2014 22:18:38 -0700 (PDT) In-Reply-To: References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> Date: Mon, 28 Jul 2014 22:18:38 -0700 Message-ID: Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Jos Backus To: Jordan Hubbard Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jul 2014 05:18:45 -0000 On Jul 28, 2014 9:49 PM, "Jordan Hubbard" wrote: > > > On Jul 27, 2014, at 11:06 PM, Jos Backus wrote: > > > It would be great if libyaml and libucl would converge, but instead it'= s > > likely that the number of solutions trying to solve the same problem will > > continue to proliferate, and we are stuck with more and more configuration > > file formats :-( > > I=E2=80=99m a huge fan of unified data formats; Apple picked XML and the = plist DTD a long time ago, a decision which has worked rather nicely in practice, but I=E2=80=99m more in love with the unification that produced than I am i= n love with XML itself. That said, it seems like this late push for YAML is a similar case for divergence just because=E2=80=A6erm=E2=80=A6 you don=E2=80= =99t like JSON? It seems like libucl has basically backed JSON with the addition of a little syntactic sugar, so what=E2=80=99s wrong with that? In general, as a tool, JSON is more limited/less expressive than YAML. Now YAGNI may apply here but I personally am not sure so I'm tempted to opt for the more flexible tool because of that. I could be wrong and maybe JSON is all that's ever needed. > Is there some reason JSON is not sufficient? I think that=E2=80=99s a be= tter question to ask, since the conversation otherwise quickly tends to sound a little like =E2=80=9CI=E2=80=99ll accept any single unified format as long = as it=E2=80=99s the specific one I like!=E2=80=9D :) I think the greater good argument would = suggest just picking one that=E2=80=99s expressive enough (roll a pair of dice), pu= t on your bikeshed-proof sunglasses, and proceed. That's a good point, and one I don't really disagree with. The main goal here is to get us machine parsable output. But part of me is sad because it's a lost opportunity to promote the more flexible format. One of the reasons JSON is so popular is the network effect, I think (it's popular because it's popular). Oh well. :) Jos > > - Jordan > From owner-freebsd-arch@FreeBSD.ORG Tue Jul 29 05:57:24 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B1AD9F8 for ; Tue, 29 Jul 2014 05:57:24 +0000 (UTC) Received: from mail-qa0-x235.google.com (mail-qa0-x235.google.com [IPv6:2607:f8b0:400d:c00::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6FFE128DD for ; Tue, 29 Jul 2014 05:57:24 +0000 (UTC) Received: by mail-qa0-f53.google.com with SMTP id v10so9019228qac.12 for ; Mon, 28 Jul 2014 22:57:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=tyocR1CnG4YUY1Z/RvX0pDDni4bI93vBb5El1yGxrOk=; b=x7YwZhDTfP36eb3EZ1hBxBL8EoPL61RBHBZPX3hDX05Fc/0/XnLjR2KfxVWzsCgSBe 3q1trQo/SA6U2ePHYF9AMhnVxfFw1dtNNdc7C0vMhkKTV9iVyy/3XF5dtIhLyPy87bM0 at2Gowj24tILKxOxnp3Hd6HGiLvrCjM7nUdYYh/bafJhefZFo7WP7fdC8bVPZyOf5e2R XSRXZUNya967yI7u2kTT6D0Na++ZNnWqycWeCrDOngO32EYyrZGzoI7imdnxHLjfIPNl vACR8o/ikZm1eFb6tlHbuHcDt7DL7Kf5iCkL+lZq5q1oDY35YTY6JY//mpcvZXZvjzZQ zVAQ== MIME-Version: 1.0 X-Received: by 10.224.55.131 with SMTP id u3mr66772744qag.98.1406613443615; Mon, 28 Jul 2014 22:57:23 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.1.6 with HTTP; Mon, 28 Jul 2014 22:57:23 -0700 (PDT) In-Reply-To: References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> Date: Mon, 28 Jul 2014 22:57:23 -0700 X-Google-Sender-Auth: Lc59y-T8c95JZr1mNNAhzlGtVx8 Message-ID: Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Adrian Chadd To: Jos Backus Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-arch@freebsd.org" , Jordan Hubbard X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jul 2014 05:57:24 -0000 Holy ... ! What really matter is whether the library API that you're going to shoehorn into plenty of utilities is expressive enough to express a whole bunch of different output types. So it doesn't matter if you want JSON, or YAML, or XML, or the native tool output. The trick is whether the library API is good enough. So if you want to win people over, just make sure it gets adopted throughout other tools. :) -a From owner-freebsd-arch@FreeBSD.ORG Tue Jul 29 06:04:03 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 337995CF for ; Tue, 29 Jul 2014 06:04:03 +0000 (UTC) Received: from mail-pd0-f169.google.com (mail-pd0-f169.google.com [209.85.192.169]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8BA0F299C for ; Tue, 29 Jul 2014 06:04:02 +0000 (UTC) Received: by mail-pd0-f169.google.com with SMTP id y10so11187037pdj.28 for ; Mon, 28 Jul 2014 23:03:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=LjTEwycfJJsc6fKOCgXaEHgGX7auEN4+hJFJ8NSkjTE=; b=irJUG6ec3w0zKIo2/XGMN2amesfLmIWGUfSg3w8UuDH+wnrck30pphcH2gX56zVpOp 8xh4oqjGGvmt5V7iPRELxgdj33tks9NdxwLgTeyK1SVjm874DAFCeEwVQ3HZ5vxaRuB0 yOEkT8tKdeD5G7Loev8rCpIEE7ic9ABpBtkNkdwWjPBfKilcRt+GRHWHT6U+8IGVYYh7 R7QimcMq9vtb9zG5i9sd1UHg2ZzhPB9IHCT1fg38nvqxHp1epzDk1GsW+wB/XhNLU09A RB7BTBueeeq4iXHMdfVzTgL76QH5o3Ts8hyizIqijC737+G9FQe4vXJi5F3ZkDn9F+73 accw== X-Gm-Message-State: ALoCoQklQGtRYO5ur25C/gq7RDDje0iCJTIab9L4jlF7B7NDg/E061ypta1gzeJKPen+J1qH+AOe MIME-Version: 1.0 X-Received: by 10.68.225.105 with SMTP id rj9mr43732209pbc.108.1406613835798; Mon, 28 Jul 2014 23:03:55 -0700 (PDT) Received: by 10.70.101.41 with HTTP; Mon, 28 Jul 2014 23:03:55 -0700 (PDT) Received: by 10.70.101.41 with HTTP; Mon, 28 Jul 2014 23:03:55 -0700 (PDT) In-Reply-To: References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> Date: Mon, 28 Jul 2014 23:03:55 -0700 Message-ID: Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Jos Backus To: Adrian Chadd Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: arch@freebsd.org, Jordan Hubbard X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jul 2014 06:04:03 -0000 On Jul 28, 2014 10:57 PM, "Adrian Chadd" wrote: > > Holy ... ! > > What really matter is whether the library API that you're going to > shoehorn into plenty of utilities is expressive enough to express a > whole bunch of different output types. > > So it doesn't matter if you want JSON, or YAML, or XML, or the native > tool output. The trick is whether the library API is good enough. > > So if you want to win people over, just make sure it gets adopted > throughout other tools. :) Wouldn't the API be a way to build up an in-memory combination of lists and maps in most cases, which then gets serialized out at emission time? Kind of like calling object.to_yaml, where object is a Hash, but in C? Jos > > -a From owner-freebsd-arch@FreeBSD.ORG Tue Jul 29 06:29:01 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B41FBAB7 for ; Tue, 29 Jul 2014 06:29:01 +0000 (UTC) Received: from mail-qg0-x22f.google.com (mail-qg0-x22f.google.com [IPv6:2607:f8b0:400d:c04::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 706382B6D for ; Tue, 29 Jul 2014 06:29:01 +0000 (UTC) Received: by mail-qg0-f47.google.com with SMTP id i50so9869015qgf.34 for ; Mon, 28 Jul 2014 23:29:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=jPt1BXAiPKBo+P11kB/mU52bN/mBHbQ6KS59H9VtmBo=; b=UiZ6BnGHuh/U3Fr/ed1TJKcX5FPzs0w3GfDLDtC/d77PbShecqB4i4WpQZOghv7XCM bcTrpmqL9ibg4f8D20wDIv1LScP6jAhP1LD3mx9woanYdpz756n2Ul/tNlUdH8uYt+J9 rrRr31l+O4kX+3cOSp8Y1g48+umz/RDrOOt7D0lsuSbnu8rI/XnZQXOTATWrQT15VFBB oX2oU56qN6dsD+FlSBI7If7BGJFUVucHcYieAwB4OxbJ598Ajmb/LjSmQ+uxk9khMjNv v7WZfTgJENVh4SVtzSneVqN3GCCVSVjM6/oDpT9eeVR9Af7GK3kwjZA6gssfNTKoyUzV DzFw== MIME-Version: 1.0 X-Received: by 10.140.38.169 with SMTP id t38mr67876533qgt.3.1406615340686; Mon, 28 Jul 2014 23:29:00 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.1.6 with HTTP; Mon, 28 Jul 2014 23:29:00 -0700 (PDT) In-Reply-To: References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> Date: Mon, 28 Jul 2014 23:29:00 -0700 X-Google-Sender-Auth: keeH2KDUG3BGmxUbZ_CDh0c53WA Message-ID: Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Adrian Chadd To: Jos Backus Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-arch@freebsd.org" , Jordan Hubbard X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jul 2014 06:29:01 -0000 On 28 July 2014 23:03, Jos Backus wrote: > > On Jul 28, 2014 10:57 PM, "Adrian Chadd" wrote: >> >> Holy ... ! >> >> What really matter is whether the library API that you're going to >> shoehorn into plenty of utilities is expressive enough to express a >> whole bunch of different output types. >> >> So it doesn't matter if you want JSON, or YAML, or XML, or the native >> tool output. The trick is whether the library API is good enough. >> >> So if you want to win people over, just make sure it gets adopted >> throughout other tools. :) > > Wouldn't the API be a way to build up an in-memory combination of lists and > maps in most cases, which then gets serialized out at emission time? Kind of > like calling object.to_yaml, where object is a Hash, but in C? > Not everything can be buffered like that over time. Time series data may have stuff buffered up during an output / sample period (eg the one a second output from vmstat 1) -a From owner-freebsd-arch@FreeBSD.ORG Tue Jul 29 06:31:54 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 66685B73 for ; Tue, 29 Jul 2014 06:31:54 +0000 (UTC) Received: from mail-pd0-f174.google.com (mail-pd0-f174.google.com [209.85.192.174]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 360D52C85 for ; Tue, 29 Jul 2014 06:31:52 +0000 (UTC) Received: by mail-pd0-f174.google.com with SMTP id fp1so11169921pdb.33 for ; Mon, 28 Jul 2014 23:31:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=AwPBV1foKskzzau4Cat5mN5L1CoD2TxwL9x2Dgo6YCw=; b=mDaQ0eLRz6HLsU0ANVCUdfTGQ+htsrvByJpNPxzga13WqHvI/zd9MEd5zkoW7sqYnP v0XInbqfRYRCzW3MVjo4buALm083rkdGcopLEi7Vh/YU6KPf0RYRKIElua8qokgEF7gA 9Ux30uTpmKcZ23bZfyQyIsjuzflRUFZ+QjTcccRXvX0//rGF19CinDNEXFmHCq+DWHkn n4juX+7DXIES/a4jf82YLB6P6dmpLbe1NePrGf+gViRCnMX6dmG8vlofIiZtpQIBHn+1 q8MJbgCqyEJxJ4j3XUC0QTkBDig3AcgsZAfv5o1T4iPRHFAEXK7n+keOgWIaJilEBn2N dIiw== X-Gm-Message-State: ALoCoQleCCeiLJAf3yilDCYuAp4SbdtZjPEGFPScfZFUe+WQVNcKfe1kPBc5MU3UJYy8m5y3W08I MIME-Version: 1.0 X-Received: by 10.68.186.97 with SMTP id fj1mr42897819pbc.37.1406615511738; Mon, 28 Jul 2014 23:31:51 -0700 (PDT) Received: by 10.70.101.41 with HTTP; Mon, 28 Jul 2014 23:31:51 -0700 (PDT) Received: by 10.70.101.41 with HTTP; Mon, 28 Jul 2014 23:31:51 -0700 (PDT) In-Reply-To: References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> Date: Mon, 28 Jul 2014 23:31:51 -0700 Message-ID: Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Jos Backus To: Adrian Chadd Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: arch@freebsd.org, Jordan Hubbard X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jul 2014 06:31:54 -0000 On Jul 28, 2014 11:29 PM, "Adrian Chadd" wrote: > > On 28 July 2014 23:03, Jos Backus wrote: > > > > On Jul 28, 2014 10:57 PM, "Adrian Chadd" wrote: > >> > >> Holy ... ! > >> > >> What really matter is whether the library API that you're going to > >> shoehorn into plenty of utilities is expressive enough to express a > >> whole bunch of different output types. > >> > >> So it doesn't matter if you want JSON, or YAML, or XML, or the native > >> tool output. The trick is whether the library API is good enough. > >> > >> So if you want to win people over, just make sure it gets adopted > >> throughout other tools. :) > > > > Wouldn't the API be a way to build up an in-memory combination of lists and > > maps in most cases, which then gets serialized out at emission time? Kind of > > like calling object.to_yaml, where object is a Hash, but in C? > > > > Not everything can be buffered like that over time. Time series data > may have stuff buffered up during an output / sample period (eg the > one a second output from vmstat 1) YAML has support for document streams, which might come in handy here :) Jos > > > -a From owner-freebsd-arch@FreeBSD.ORG Tue Jul 29 07:41:04 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0C3FEFAF for ; Tue, 29 Jul 2014 07:41:04 +0000 (UTC) Received: from eastrmfepi101.cox.net (eastrmfepi101.cox.net [68.230.241.197]) by mx1.freebsd.org (Postfix) with ESMTP id A733B2475 for ; Tue, 29 Jul 2014 07:41:03 +0000 (UTC) Received: from eastrmimpo109 ([68.230.241.222]) by eastrmfepo102.cox.net (InterMail vM.8.01.05.15 201-2260-151-145-20131218) with ESMTP id <20140729055640.JXAA18526.eastrmfepo102.cox.net@eastrmimpo109> for ; Tue, 29 Jul 2014 01:56:40 -0400 Received: from [192.168.0.33] ([72.194.65.37]) by eastrmimpo109 with cox id Y5we1o0090oDpc8015wfJt; Tue, 29 Jul 2014 01:56:39 -0400 X-CT-Class: Clean X-CT-Score: 0.00 X-CT-RefID: str=0001.0A020208.53D73797.0101,ss=1,re=0.000,fgs=0 X-CT-Spam: 0 X-Authority-Analysis: v=2.0 cv=F4SJgNdN c=1 sm=1 a=Z5AWCFQ5VKA4phzWx86Kmg==:17 a=TM20W-5uK7UA:10 a=2b1Y-s4cCm8A:10 a=dBRESv0yCI8A:10 a=N659UExz7-8A:10 a=6I5d2MoRAAAA:8 a=89n-p_j9wxG2N1lDamgA:9 a=pILNOxqGKmIA:10 a=Z5AWCFQ5VKA4phzWx86Kmg==:117 X-CM-Score: 0.00 Authentication-Results: cox.net; auth=pass (PLAIN) smtp.auth=myhuge@cox.net Message-ID: <53D73796.9060102@FreeBSD.org> Date: Mon, 28 Jul 2014 22:56:38 -0700 From: Stephen Hurd User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:29.0) Gecko/20100101 Firefox/29.0 SeaMonkey/2.26.1 MIME-Version: 1.0 To: Jordan Hubbard , Jos Backus Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit Cc: arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jul 2014 07:41:04 -0000 Jordan Hubbard wrote: > Is there some reason JSON is not sufficient? I think thats a better question to ask, since the conversation otherwise quickly tends to sound a little like Ill accept any single unified format as long as its the specific one I like! :) I think the greater good argument would suggest just picking one thats expressive enough (roll a pair of dice), put on your bikeshed-proof sunglasses, and proceed. The biggest problem I tend to have with JSON is that there is no comment format. I rarely update a configuration without adding a comment regarding it, and since JSON doesn't have comments, it's simply a non-starter for most of my usage. The whole "just a key that you know isn't real" hack is terrible. From owner-freebsd-arch@FreeBSD.ORG Tue Jul 29 07:56:44 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from ptrcrt.ch (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D8B2C574; Tue, 29 Jul 2014 07:56:42 +0000 (UTC) Date: Tue, 29 Jul 2014 09:56:40 +0200 From: Pietro Cerutti To: Stephen Hurd Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Message-ID: <20140729075640.GB76693@ptrcrt.ch> Reply-To: gahr@FreeBSD.org References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> <53D73796.9060102@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="RASg3xLB4tUQ4RcS" Content-Disposition: inline In-Reply-To: <53D73796.9060102@FreeBSD.org> X-PGP-Key: fp="DA6D E106 A5B8 54B8 5DD8 6D49 ADD0 D38E A192 089E"; id="0xA192089E"; get=; get=; User-Agent: Mutt/1.5.23 (2014-03-12) Cc: Jordan Hubbard , arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jul 2014 07:56:44 -0000 --RASg3xLB4tUQ4RcS Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2014-Jul-28, 22:56, Stephen Hurd wrote: > Jordan Hubbard wrote: > > Is there some reason JSON is not sufficient? I think that=E2=80=99s a = better question to ask, since the conversation otherwise quickly tends to s= ound a little like =E2=80=9CI=E2=80=99ll accept any single unified format a= s long as it=E2=80=99s the specific one I like!=E2=80=9D :) I think the g= reater good argument would suggest just picking one that=E2=80=99s expressi= ve enough (roll a pair of dice), put on your bikeshed-proof sunglasses, and= proceed. >=20 > The biggest problem I tend to have with JSON is that there is no comment > format. I rarely update a configuration without adding a comment > regarding it, and since JSON doesn't have comments, it's simply a > non-starter for most of my usage. The whole "just a key that you know > isn't real" hack is terrible. There's an interesting post about whether this really is a problem: https://groups.yahoo.com/neo/groups/json/conversations/topics/156 I'm not saying it isn't, just that there are reasons why comments are not part of JSON. --=20 Pietro Cerutti The FreeBSD Project gahr@FreeBSD.org PGP Public Key: http://gahr.ch/pgp --RASg3xLB4tUQ4RcS Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJT11O3XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXREQTZERTEwNkE1Qjg1NEI4NUREODZENDlB REQwRDM4RUExOTIwODlFAAoJEK3Q046hkgiectgP/jThW8PocDricR4t8lEDonyD PN8bNukxBSOLIaZMKRDlsomDWb2RCkgyKIotnpgBLzvAZeJolh8qz+scw3ZbSPuG vp95ly52HusOLOomFZELkfgQkS0w+wCoU4ztUwoxj92vagumjkR/PK+BEgUo44nF lZ27uonJGcv9SHQcpDjV2silZdd02PZFX49HqagSO1lJfeUCUrien2fu2AiO+XLP Xc2ovRSIzJRhJmWdX7KRB5CrqZBEdSoj6oRGlXrmBJpXGqCzYnoX6HQfBMmnEe+7 6YsAh4SI8RiWYGhnzi5q8tL/r5AzAuUZ3kr6df7N49EtVthJZtoDeG72VK0JHoN3 OW+D8rgNCG+Kc6XuJs1S0xIoRUoasDEdx1FvPP9UebO7lej02v+1AtaKTZLMr4Qv ybVkQvo8YuuLHAnoKxJ4qFoCz2tf2okIOCgSUsVNEIl9+oe1wGn+gThj4T02uYiL p+Z5ojCDzEssH+6wWONKSZYLyWlPxfrqCXtTSBrswRk32Z5kYv8lR/mJ0RMXpTRN Le+TdARqv7TYaONFEZkG745NG/P/BMZ1hAjqmZAMaMcRNXTw0aQ9nBXd/TXbcV1t Jfqd5WUl4hCiGlbEZBpLPIt91s+zFy3LZMhscwrdm4nvylCkQDd7iycswjO5ZAWN bg6cYbqLYBm7iT3njnC7 =PFlL -----END PGP SIGNATURE----- --RASg3xLB4tUQ4RcS-- From owner-freebsd-arch@FreeBSD.ORG Tue Jul 29 08:45:37 2014 Return-Path: Delivered-To: arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 57A9CD04; Tue, 29 Jul 2014 08:45:37 +0000 (UTC) Received: from mail143c7.megamailservers.com (mail745.megamailservers.com [69.49.98.55]) (using TLSv1.1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D8D062ACA; Tue, 29 Jul 2014 08:45:36 +0000 (UTC) X-Authenticated-User: hurds.sasktel.net Received: from [192.168.0.33] (ip72-194-65-37.oc.oc.cox.net [72.194.65.37]) (authenticated bits=0) by mail143c7.megamailservers.com (8.13.6/8.13.1) with ESMTP id s6T8jOmp003940; Tue, 29 Jul 2014 04:45:26 -0400 Message-ID: <53D75F24.6090507@sasktel.net> Date: Tue, 29 Jul 2014 01:45:24 -0700 From: Stephen Hurd User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:29.0) Gecko/20100101 Firefox/29.0 SeaMonkey/2.26.1 MIME-Version: 1.0 To: gahr@FreeBSD.org, Stephen Hurd Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> <53D73796.9060102@FreeBSD.org> <20140729075640.GB76693@ptrcrt.ch> In-Reply-To: <20140729075640.GB76693@ptrcrt.ch> X-Enigmail-Version: 1.6.1_pre20140112 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-CTCH-RefID: str=0001.0A020207.53D75F27.010B, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0 X-CTCH-VOD: Unknown X-CTCH-Spam: Unknown X-CTCH-Score: 0.000 X-CTCH-Rules: X-CTCH-Flags: 0 X-CTCH-ScoreCust: 0.000 X-CSC: 0 X-CHA: v=2.1 cv=cZDr8BzM c=1 sm=1 tr=0 a=Z5AWCFQ5VKA4phzWx86Kmg==:117 a=Z5AWCFQ5VKA4phzWx86Kmg==:17 a=kviXuzpPAAAA:8 a=BDKbP5mgAAAA:8 a=J0_EhWaTXrIA:10 a=2b1Y-s4cCm8A:10 a=YxfxW3ofkq8A:10 a=IkcTkHD0fZMA:10 a=uhPMnebkAAAA:8 a=CjxXgO3LAAAA:8 a=QfKxxUxMAAAA:8 a=LDA7PybYtlmFmPbqO_IA:9 a=QEXdDO2ut3YA:10 a=uljPtXyl9CQA:10 Cc: Jordan Hubbard , arch@FreeBSD.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jul 2014 08:45:37 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Pietro Cerutti wrote: > On 2014-Jul-28, 22:56, Stephen Hurd wrote: >> Jordan Hubbard wrote: >>> Is there some reason JSON is not sufficient? I think that’s a better question to ask, since the conversation otherwise quickly tends to sound a little like “I’ll accept any single unified format as long as it’s the specific one I like!” :) I think the greater good argument would suggest just picking one that’s expressive enough (roll a pair of dice), put on your bikeshed-proof sunglasses, and proceed. >> >> The biggest problem I tend to have with JSON is that there is no comment >> format. I rarely update a configuration without adding a comment >> regarding it, and since JSON doesn't have comments, it's simply a >> non-starter for most of my usage. The whole "just a key that you know >> isn't real" hack is terrible. > > There's an interesting post about whether this really is a problem: > > https://groups.yahoo.com/neo/groups/json/conversations/topics/156 > > I'm not saying it isn't, just that there are reasons why comments are > not part of JSON. > Yep, program to program formats obviously don't need comments, comments are for something that a human is modifying on an infrequent basis. But invariably, a serialization format designed to be human readable ends up getting used as a configuration format... and my config files need comments to ensure that future me hasn't forgotten the lessons that present me has learnt. As for comments being "the most difficult feature to support", I find that hard to believe... I don't see how it could possibly be more difficult than string support with escaping - especially for the "to end of line" style comment. As for "removing comments aligns JSON more closely with YAML" could be trivially solved by supporting YAML style comments. The first reason is fine, and while I don't agree with the fourth reason, I can accept it. But really, the ship has sailed, JSON doesn't have comments, and that's the biggest problem I have with JSON. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 Comment: Using GnuPG with SeaMonkey - http://www.enigmail.net/ iQF8BAEBCgBmBQJT118kXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRDQTUyMTJFRTc2MEFCQ0FBMEQ1RTI5NzRC M0I1QTFCNDk4Q0U1Q0QwAAoJELO1obSYzlzQgc8H/R79tqlzQpjY0jLpwNwUaSgH M5lroa4f4KEpvRqLKRj0Owt/N+rHzXjPL/mmD6XGckGb3JG0HbEmofq2rz/XoUSI mksGLsdGkpN4eQppKaAU+mNAPObLoX6KZA51dbNlE6NzFcJirigWfHqUghOZt0X3 xKAWk86YcUvQZ4uYSU7jrkWVpoQeOg2UcSQ3AVvn+cVrYahN3w+r8ypgudd3IcAY cI0zoTp2w/oiq3mmECdboNFsaZmKlhvXaj0sKrtSlLVHIkPYqzP+2o3QgT96ovHV 0R64R+mg4czpimNcQUrUai3pX5uazwnUSdXy4TnWzYE9IYS33vZjUOqA7wavuDE= =EaXN -----END PGP SIGNATURE----- From owner-freebsd-arch@FreeBSD.ORG Tue Jul 29 17:52:02 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5C4AF819; Tue, 29 Jul 2014 17:52:02 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 33A222F0B; Tue, 29 Jul 2014 17:52:02 +0000 (UTC) Received: from [192.168.1.75] (75-48-77-17.lightspeed.cncrca.sbcglobal.net [75.48.77.17]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 8A5B9B91C; Tue, 29 Jul 2014 13:51:59 -0400 (EDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: [rfc] where to put cpuid_t ? From: John Baldwin In-Reply-To: Date: Tue, 29 Jul 2014 10:51:57 -0700 Content-Transfer-Encoding: 7bit Message-Id: <41D207A0-D82F-456E-B0A4-9CA9520D4F77@FreeBSD.org> References: To: Adrian Chadd X-Mailer: Apple Mail (2.1878.6) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 29 Jul 2014 13:52:00 -0400 (EDT) Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jul 2014 17:52:02 -0000 On Jul 24, 2014, at 12:03 AM, Adrian Chadd wrote: > Hi! > > I've been chipping away at a cpuid_t in a local branch for a while and > I think I have the very basics working well enough. > > However, the most annoying thing that's crept up is the most bikeshed > topic of them all - where should it live. > > I'd like to avoid having to include sys/pcpu.h or sys/_cpuset.h just > for the id type - it seems a bit overkill. > > So - suggestions? Otherwise I'm going to leave it in sys/pcpu.h and > just polluate appropriately. > > I'd like to try and get cpuid_t and a handful of KBI-changing things > into -HEAD before 11 is branched so we at least have a hope of trying > to support > 128 CPUs out of the box in the immediate future and > 253 > CPUs in the later future. __cpuid_t should be in and cpuid_t should be in . This is how the old cpumask_t worked (though it allowed __cpumask_t to be MD whereas I think you can probably make __cpuid_t MI). -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Tue Jul 29 23:04:01 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C805A631 for ; Tue, 29 Jul 2014 23:04:01 +0000 (UTC) Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2lp0204.outbound.protection.outlook.com [207.46.163.204]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6B1F322A3 for ; Tue, 29 Jul 2014 23:03:59 +0000 (UTC) Received: from BY2PR05CA001.namprd05.prod.outlook.com (10.242.32.31) by DM2PR05MB733.namprd05.prod.outlook.com (10.141.178.18) with Microsoft SMTP Server (TLS) id 15.0.995.14; Tue, 29 Jul 2014 23:03:51 +0000 Received: from BN1BFFO11FD013.protection.gbl (2a01:111:f400:7c10::1:179) by BY2PR05CA001.outlook.office365.com (2a01:111:e400:2c2a::31) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Tue, 29 Jul 2014 23:03:50 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BN1BFFO11FD013.mail.protection.outlook.com (10.58.144.76) with Microsoft SMTP Server (TLS) id 15.0.990.10 via Frontend Transport; Tue, 29 Jul 2014 23:03:50 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Tue, 29 Jul 2014 16:03:47 -0700 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6TN3jn05862; Tue, 29 Jul 2014 16:03:45 -0700 (PDT) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id 31E9B580A2; Tue, 29 Jul 2014 16:03:45 -0700 (PDT) To: Jordan Hubbard Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> Comments: In-reply-to: Jordan Hubbard message dated "Mon, 28 Jul 2014 21:49:08 -0700." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Tue, 29 Jul 2014 16:03:45 -0700 Message-ID: <20140729230345.31E9B580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(189002)(199002)(69596002)(76176999)(50226001)(21056001)(77156001)(93886003)(104166001)(33656002)(107046002)(97736001)(93546004)(102836001)(64706001)(81542001)(48376002)(50986999)(79102001)(68736004)(93916002)(110136001)(62966002)(99396002)(77982001)(4396001)(80022001)(76506005)(57986006)(101356003)(102176002)(81342001)(92566001)(85852003)(83072002)(76482001)(90896003)(92726001)(83322001)(46102001)(44976005)(20776003)(47776003)(88136002)(89996001)(87286001)(87936001)(81156004)(106466001)(31966008)(6806004)(86362001)(70486001)(95666004)(74502001)(85306003)(42262001); DIR:OUT; SFP:; SCL:1; SRVR:DM2PR05MB733; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0287BBA78D Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=sjg@juniper.net; X-OriginatorOrg: juniper.net Cc: arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jul 2014 23:04:02 -0000 On Mon, 28 Jul 2014 21:49:08 -0700, Jordan Hubbard writes: >I=92m a huge fan of unified data formats; Apple picked XML and the plist DT= We did too, and it has indeed been useful. Being able to render rich html has also proven very cool given the improvment in browsers in the last several years. The point I was making earlier (perhaps not very well) was that the api Phil has proposed provides enough clue to allow outputting plain text as well as that rich html. IIRC the main wrinkle json imposes is a need for extra structure calls - due to the way lists (I think) are handled, anyway if the api can handle json and html it should be possible to add rendering for others should that prove necessary one day - hopefully without having to revisit any of the apps. From owner-freebsd-arch@FreeBSD.ORG Tue Jul 29 23:19:34 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5A087CC1; Tue, 29 Jul 2014 23:19:34 +0000 (UTC) Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2lp0206.outbound.protection.outlook.com [207.46.163.206]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C4FD324C2; Tue, 29 Jul 2014 23:19:32 +0000 (UTC) Received: from BY2PR05CA045.namprd05.prod.outlook.com (10.141.250.35) by DM2PR05MB736.namprd05.prod.outlook.com (10.141.178.25) with Microsoft SMTP Server (TLS) id 15.0.995.14; Tue, 29 Jul 2014 23:19:22 +0000 Received: from BN1AFFO11FD024.protection.gbl (2a01:111:f400:7c10::137) by BY2PR05CA045.outlook.office365.com (2a01:111:e400:2c5f::35) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Tue, 29 Jul 2014 23:19:22 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BN1AFFO11FD024.mail.protection.outlook.com (10.58.52.84) with Microsoft SMTP Server (TLS) id 15.0.990.10 via Frontend Transport; Tue, 29 Jul 2014 23:19:21 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Tue, 29 Jul 2014 16:19:20 -0700 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6TNJJn17687; Tue, 29 Jul 2014 16:19:19 -0700 (PDT) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id 75DF2580A2; Tue, 29 Jul 2014 16:19:19 -0700 (PDT) To: Adrian Chadd Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> Comments: In-reply-to: Adrian Chadd message dated "Mon, 28 Jul 2014 22:57:23 -0700." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Tue, 29 Jul 2014 16:19:19 -0700 Message-ID: <20140729231919.75DF2580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(189002)(199002)(70486001)(99396002)(87286001)(87936001)(74502001)(31966008)(57986006)(92566001)(92726001)(76506005)(81342001)(97736001)(77982001)(90896003)(79102001)(88136002)(76482001)(89996001)(85852003)(46102001)(83072002)(48376002)(85306003)(68736004)(69596002)(110136001)(21056001)(106466001)(81156004)(93916002)(86362001)(95666004)(4396001)(50226001)(20776003)(47776003)(93886003)(83322001)(62966002)(101356003)(50986999)(76176999)(64706001)(77156001)(80022001)(33656002)(93546004)(81542001)(102836001)(107046002)(104166001)(6806004)(44976005)(102176002)(42262001); DIR:OUT; SFP:; SCL:1; SRVR:DM2PR05MB736; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0287BBA78D Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=sjg@juniper.net; X-OriginatorOrg: juniper.net Cc: Jordan Hubbard , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jul 2014 23:19:34 -0000 On Mon, 28 Jul 2014 22:57:23 -0700, Adrian Chadd writes: >What really matter is whether the library API that you're going to >shoehorn into plenty of utilities is expressive enough to express a >whole bunch of different output types. Exactly. >So it doesn't matter if you want JSON, or YAML, or XML, or the native >tool output. The trick is whether the library API is good enough. Yes. If the API is sufficiently expressive you can add others renderings if needed. I think the API proposed meets that criteria - with the open issue of wide char support (or perhaps UTF8) to be resolved. From owner-freebsd-arch@FreeBSD.ORG Tue Jul 29 23:23:44 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2DDBAE34; Tue, 29 Jul 2014 23:23:44 +0000 (UTC) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2lp0239.outbound.protection.outlook.com [207.46.163.239]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A7B012573; Tue, 29 Jul 2014 23:23:42 +0000 (UTC) Received: from BL2PR05CA0048.namprd05.prod.outlook.com (10.255.226.48) by DM2PR05MB734.namprd05.prod.outlook.com (10.141.178.22) with Microsoft SMTP Server (TLS) id 15.0.995.14; Tue, 29 Jul 2014 23:23:40 +0000 Received: from BN1AFFO11FD043.protection.gbl (2a01:111:f400:7c10::105) by BL2PR05CA0048.outlook.office365.com (2a01:111:e400:c04::48) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Tue, 29 Jul 2014 23:23:40 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BN1AFFO11FD043.mail.protection.outlook.com (10.58.52.190) with Microsoft SMTP Server (TLS) id 15.0.990.10 via Frontend Transport; Tue, 29 Jul 2014 23:23:40 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Tue, 29 Jul 2014 16:23:39 -0700 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6TNNcn19734; Tue, 29 Jul 2014 16:23:38 -0700 (PDT) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id 40AA6580A2; Tue, 29 Jul 2014 16:23:38 -0700 (PDT) To: Jos Backus Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> Comments: In-reply-to: Jos Backus message dated "Mon, 28 Jul 2014 23:03:55 -0700." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Tue, 29 Jul 2014 16:23:38 -0700 Message-ID: <20140729232338.40AA6580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(189002)(199002)(85852003)(74502001)(6806004)(92726001)(81342001)(92566001)(31966008)(47776003)(87286001)(83072002)(97736001)(90896003)(70486001)(95666004)(44976005)(101356003)(93886003)(93916002)(86362001)(20776003)(93546004)(80022001)(85306003)(33656002)(104166001)(102176002)(76176999)(102836001)(81542001)(4396001)(50226001)(68736004)(83322001)(69596002)(57986006)(64706001)(50986999)(76482001)(62966002)(77156001)(89996001)(21056001)(48376002)(81156004)(106466001)(99396002)(87936001)(46102001)(77982001)(107046002)(79102001)(110136001)(88136002)(76506005)(42262001); DIR:OUT; SFP:; SCL:1; SRVR:DM2PR05MB734; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0287BBA78D Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=sjg@juniper.net; X-OriginatorOrg: juniper.net Cc: arch@freebsd.org, Adrian Chadd , Jordan Hubbard X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jul 2014 23:23:44 -0000 On Mon, 28 Jul 2014 23:03:55 -0700, Jos Backus writes: >Wouldn't the API be a way to build up an in-memory combination of lists and >maps in most cases, which then gets serialized out at emission time? Kind You cannot rely on being able to do that. Routing tables can be "big" especially when encoded in xml ;-) You certainly cannot wait for all of it to arrive before you start rendering. From owner-freebsd-arch@FreeBSD.ORG Tue Jul 29 23:24:12 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 55C91EE7; Tue, 29 Jul 2014 23:24:12 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 1475C2576; Tue, 29 Jul 2014 23:24:11 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s6TNO5Dc098239 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 29 Jul 2014 16:24:05 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s6TNO4CL098238; Tue, 29 Jul 2014 16:24:05 -0700 (PDT) (envelope-from jmg) Date: Tue, 29 Jul 2014 16:24:04 -0700 From: John-Mark Gurney To: Gleb Smirnoff Subject: Re: [CFT/review] new sendfile(2) Message-ID: <20140729232404.GF43962@funkthat.com> Mail-Followup-To: Gleb Smirnoff , arch@freebsd.org References: <20140529102054.GX50679@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140529102054.GX50679@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Tue, 29 Jul 2014 16:24:05 -0700 (PDT) Cc: arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jul 2014 23:24:12 -0000 Gleb Smirnoff wrote this message on Thu, May 29, 2014 at 14:20 +0400: > One of the approaches we are experimenting with is new sendfile(2) > implementation, that doesn't block on the I/O done from the file > descriptor. I know this is a reply to an old message, but... How is this different from: SF_NODISKIO. This flag causes any sendfile() call which would block on disk I/O to instead return EBUSY. Busy servers may bene- fit by transferring requests that would block to a separate I/O worker thread. > The patch for review is available at: > > https://phabric.freebsd.org/D102 Not public, so couldn't see the state... > And for those who prefer email attachments, it is also attached. > The patch has 3 logically separate changes in itself: > > 1) Split of socket buffer sb_cc field into sb_acc and sb_ccc. Where > sb_acc stands for "available character count" and sb_ccc is "claimed > character count". This allows us to write a data to a socket, that is > not ready yet. The data sits in the socket, consumes its space, and > keeps itself in the right order with earlier or later writes to socket. > But it can be send only after it is marked as ready. This change is > split across many files. This change really should be split out and possibly committed seperately after a review by the proper people... > 2) A new vnode operation: VOP_GETPAGES_ASYNC(). This one lives in sys/vm. > > 3) Actual implementation of new sendfile(2). This one lives in > kern/uipc_syscalls.c -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Tue Jul 29 23:30:54 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BF36B144 for ; Tue, 29 Jul 2014 23:30:54 +0000 (UTC) Received: from mail-pd0-f179.google.com (mail-pd0-f179.google.com [209.85.192.179]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8F57E263A for ; Tue, 29 Jul 2014 23:30:54 +0000 (UTC) Received: by mail-pd0-f179.google.com with SMTP id ft15so413485pdb.10 for ; Tue, 29 Jul 2014 16:30:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=67A+A/sYZ0HoSDIF8IeOjxzbc8xjpogPHE0GXaVtY2A=; b=ZojjdJ/9S92UvQcKeFHQ3b5EXdvfaK6o5j9osu3JrgxQl+73puZavP6VDaf53k7I+4 f9zsPwTKQMOjbOKbA64cEJ1jpuj73NafrqD5AHmAM1MjUP8Bf9RNhwPHMGeqRtTFrwgP kUG24ILko0iD6K7aNIY383EyS16xFRH9VOFGAZ6ho2/KyXHo742qTKUwqTSfEECCWKYN ux2le4i4Cpn4sz33BcuyvnQvY5fymrY6c74s73ptECXM6dYvr8mENtcD+hyYwC0tOiJj dRwCwK2RcZmvYt6KuKXVg1tac4L0fFwUyA8FC1ggldBQouwIrAIehmyUWzgKwAyXpgH4 kkrA== X-Gm-Message-State: ALoCoQk9uPCtI5O2f7/lll/d+JKRbEAijZQH0oh44h29UGcqmLp9UPR8zUurJ04L/fK8ieuMpO4u MIME-Version: 1.0 X-Received: by 10.66.157.40 with SMTP id wj8mr404774pab.42.1406676653582; Tue, 29 Jul 2014 16:30:53 -0700 (PDT) Received: by 10.70.101.41 with HTTP; Tue, 29 Jul 2014 16:30:53 -0700 (PDT) Received: by 10.70.101.41 with HTTP; Tue, 29 Jul 2014 16:30:53 -0700 (PDT) In-Reply-To: <20140729232338.40AA6580A2@chaos.jnpr.net> References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> <20140729232338.40AA6580A2@chaos.jnpr.net> Date: Tue, 29 Jul 2014 16:30:53 -0700 Message-ID: Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Jos Backus To: "Simon J. Gerraty" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: arch@freebsd.org, Adrian Chadd , Jordan Hubbard X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jul 2014 23:30:54 -0000 On Jul 29, 2014 4:23 PM, "Simon J. Gerraty" wrote: > > > On Mon, 28 Jul 2014 23:03:55 -0700, Jos Backus writes: > >Wouldn't the API be a way to build up an in-memory combination of lists and > >maps in most cases, which then gets serialized out at emission time? Kind > > You cannot rely on being able to do that. > Routing tables can be "big" especially when encoded in xml ;-) > You certainly cannot wait for all of it to arrive before you start > rendering. Understood. This is why a serialization output format that supports streaming data is useful. Jos From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 01:51:55 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4CD23727 for ; Wed, 30 Jul 2014 01:51:55 +0000 (UTC) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 34FEA2439 for ; Wed, 30 Jul 2014 01:51:54 +0000 (UTC) Received: from u10-2-16-021.office.norse-data.com (unknown [50.204.88.51]) by elvis.mu.org (Postfix) with ESMTPSA id 368DB346DE1C for ; Tue, 29 Jul 2014 18:51:54 -0700 (PDT) Message-ID: <53D85001.4030707@mu.org> Date: Tue, 29 Jul 2014 18:53:05 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.0 MIME-Version: 1.0 To: freebsd-arch@freebsd.org Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 01:51:55 -0000 On 7/28/14 11:29 PM, Adrian Chadd wrote: > On 28 July 2014 23:03, Jos Backus wrote: >> On Jul 28, 2014 10:57 PM, "Adrian Chadd" wrote: >>> Holy ... ! >>> >>> What really matter is whether the library API that you're going to >>> shoehorn into plenty of utilities is expressive enough to express a >>> whole bunch of different output types. >>> >>> So it doesn't matter if you want JSON, or YAML, or XML, or the native >>> tool output. The trick is whether the library API is good enough. >>> >>> So if you want to win people over, just make sure it gets adopted >>> throughout other tools. :) >> Wouldn't the API be a way to build up an in-memory combination of lists and >> maps in most cases, which then gets serialized out at emission time? Kind of >> like calling object.to_yaml, where object is a Hash, but in C? >> > Not everything can be buffered like that over time. Time series data > may have stuff buffered up during an output / sample period (eg the > one a second output from vmstat 1) Yup, exactly! This is very much needed and will be part of the gsoc project being worked on. -Alfred > > -a > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 02:11:26 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6F486BEB for ; Wed, 30 Jul 2014 02:11:26 +0000 (UTC) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 5A6A82600 for ; Wed, 30 Jul 2014 02:11:26 +0000 (UTC) Received: from u10-2-16-021.office.norse-data.com (unknown [50.204.88.51]) by elvis.mu.org (Postfix) with ESMTPSA id 0833B346DE0F for ; Tue, 29 Jul 2014 19:11:26 -0700 (PDT) Message-ID: <53D85495.4050408@mu.org> Date: Tue, 29 Jul 2014 19:12:37 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.0 MIME-Version: 1.0 To: freebsd-arch@freebsd.org Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> <20140729230345.31E9B580A2@chaos.jnpr.net> In-Reply-To: <20140729230345.31E9B580A2@chaos.jnpr.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 02:11:26 -0000 On 7/29/14 4:03 PM, Simon J. Gerraty wrote: > On Mon, 28 Jul 2014 21:49:08 -0700, Jordan Hubbard writes: >> I=92m a huge fan of unified data formats; Apple picked XML and the plist DT= > We did too, and it has indeed been useful. > Being able to render rich html has also proven very cool given the > improvment in browsers in the last several years. > > The point I was making earlier (perhaps not very well) was that the api > Phil has proposed provides enough clue to allow outputting plain text as > well as that rich html. IIRC the main wrinkle json imposes is a need for > extra structure calls - due to the way lists (I think) are handled, > anyway if the api can handle json and html it should be possible to add > rendering for others should that prove necessary one day - hopefully > without having to revisit any of the apps. > Is JSON not handled? How many utils are converted over at this point? Have you seen the GSOC project which aims to do this as well? "machine readable output from userland utilities" -> https://www.google-melange.com/gsoc/project/details/google/gsoc2014/zarko_korchev/5676830073815040 -Alfred From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 03:26:13 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8078B967 for ; Wed, 30 Jul 2014 03:26:13 +0000 (UTC) Received: from smtp10.server.rpi.edu (gateway.canit.rpi.edu [128.113.2.230]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3A8C12CCE for ; Wed, 30 Jul 2014 03:26:12 +0000 (UTC) Received: from smtp-auth2.server.rpi.edu (route.canit.rpi.edu [128.113.2.232]) by smtp10.server.rpi.edu (8.14.3/8.14.3/Debian-9.4) with ESMTP id s6U3L4Rh006289; Tue, 29 Jul 2014 23:21:04 -0400 Received: from smtp-auth2.server.rpi.edu (localhost [127.0.0.1]) by smtp-auth2.server.rpi.edu (Postfix) with ESMTP id 3D3A118018; Tue, 29 Jul 2014 23:21:04 -0400 (EDT) Received: from [128.113.24.47] (gilead-im124.netel.rpi.edu [128.113.124.180]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: drosih) by smtp-auth2.server.rpi.edu (Postfix) with ESMTPSA id 27307181BC; Tue, 29 Jul 2014 23:21:03 -0400 (EDT) From: "Garance A Drosehn" To: "Alfred Perlstein" Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Date: Tue, 29 Jul 2014 23:19:39 -0400 Message-ID: <448BB170-6DDC-442B-A5C5-973EC35FF7B9@rpi.edu> In-Reply-To: <53D85495.4050408@mu.org> References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> <20140729230345.31E9B580A2@chaos.jnpr.net> <53D85495.4050408@mu.org> MIME-Version: 1.0 Content-Type: text/plain; format=flowed Content-Transfer-Encoding: quoted-printable X-Mailer: MailMate (1.7.2r3905) X-Virus-Scanned: ClamAV using ClamSMTP X-Bayes-Prob: 0.0001 (Score 0, tokens from: outgoing, @@RPTN) X-Spam-Score: 0.00 () [Hold at 15.10] X-CanIt-Incident-Id: 03Mwrl4KQ X-CanIt-Geo: ip=128.113.124.180; country=US; region=New York; city=Troy; latitude=42.7495; longitude=-73.5951; http://maps.google.com/maps?q=42.7495,-73.5951&z=6 X-CanItPRO-Stream: outgoing X-Canit-Stats-ID: Bayes signature not available X-Scanned-By: CanIt (www . roaringpenguin . com) on 128.113.2.230 Cc: freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 03:26:13 -0000 On 29 Jul 2014, at 22:12, Alfred Perlstein wrote: > > How many utils are converted over at this point? > > Have you seen the GSOC project which aims to do this as well? > "machine readable output from userland utilities" -> = > https://www.google-melange.com/gsoc/project/details/google/gsoc2014/zar= ko_korchev/5676830073815040 > > -Alfred FWIW, I've been experimenting a bit with something like this for 'lpq' output, although I'm doing it in a script outside of 'lpq'. Which is to say, I haven't tried modifying the code of lpq itself because that's a real mess to modify due to the way lpq builds each line that it = prints. The above URL is just a pointer to the project listing at gsoc. The = wiki page might also be more informative: https://wiki.freebsd.org/SummerOfCode2014/MachineReadableFromUserlandUtil= s although I must admit I don't know how to check what source changes have been made so far by starting that that wiki entry. One question that comes up is how to organize the data in the output the command creates. For instance, my experiment is just trying to mimic a rather inefficient process was implemented by some student consultants, and I wouldn't have organized the JSON object the way they did. I = wouldn't have picked the same names for keys, for instance. But there are some = more complicated issues which come up, due to the fact that one 'lpq' command can be showing data from multiple processes which are running on = multiple hosts, and 'lpq' on the current host can't depend on getting JSON output from the other hosts or processes that it gets information from. Given the way lpq's source is organized, I'm also thinking that maybe = it'd be better to do this as a separate command, maybe something like = 'lpserialize'. I personally don't expect unix commands to output HTML, and I find XML a = bit unwieldy to work with. But in my own bikeshed I'm interested in formats = of json, yaml, and edn (from the world of Clojure). In any case, I am interested to see how the GSOC project works out, or whatever happens with this libxo project. -- = Garance Alistair Drosehn =3D drosih@rpi.edu Senior Systems Programmer or gad@FreeBSD.org Rensselaer Polytechnic Institute; Troy, NY; USA From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 03:41:39 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4AE2DBAA for ; Wed, 30 Jul 2014 03:41:39 +0000 (UTC) Received: from smtp10.server.rpi.edu (gateway.canit.rpi.edu [128.113.2.230]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 049A62F26 for ; Wed, 30 Jul 2014 03:41:38 +0000 (UTC) Received: from smtp-auth1.server.rpi.edu (smtp-auth1.server.rpi.edu [128.113.2.231]) by smtp10.server.rpi.edu (8.14.3/8.14.3/Debian-9.4) with ESMTP id s6U3fbue008633 for ; Tue, 29 Jul 2014 23:41:37 -0400 Received: from smtp-auth1.server.rpi.edu (localhost [127.0.0.1]) by smtp-auth1.server.rpi.edu (Postfix) with ESMTP id 653C65806D for ; Tue, 29 Jul 2014 23:41:37 -0400 (EDT) Received: from [128.113.24.47] (gilead-im124.netel.rpi.edu [128.113.124.180]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: drosih) by smtp-auth1.server.rpi.edu (Postfix) with ESMTPSA id 57C1D5800D for ; Tue, 29 Jul 2014 23:41:37 -0400 (EDT) From: "Garance A Drosehn" To: freebsd-arch@freebsd.org Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Date: Tue, 29 Jul 2014 23:40:14 -0400 Message-ID: <1092D020-F782-4587-A199-C7313BC2CD20@rpi.edu> In-Reply-To: <448BB170-6DDC-442B-A5C5-973EC35FF7B9@rpi.edu> References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> <20140729230345.31E9B580A2@chaos.jnpr.net> <53D85495.4050408@mu.org> <448BB170-6DDC-442B-A5C5-973EC35FF7B9@rpi.edu> MIME-Version: 1.0 X-Mailer: MailMate (1.7.2r3905) X-Virus-Scanned: ClamAV using ClamSMTP X-Bayes-Prob: 0.0001 (Score 0, tokens from: outgoing, @@RPTN) X-Spam-Score: 0.00 () [Hold at 15.10] X-CanIt-Incident-Id: 03MwrFBTJ X-CanIt-Geo: ip=128.113.124.180; country=US; region=New York; city=Troy; latitude=42.7495; longitude=-73.5951; http://maps.google.com/maps?q=42.7495,-73.5951&z=6 X-CanItPRO-Stream: outgoing X-Canit-Stats-ID: Bayes signature not available X-Scanned-By: CanIt (www . roaringpenguin . com) on 128.113.2.230 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 03:41:39 -0000 On 29 Jul 2014, at 23:19, Garance A Drosehn wrote: > > In any case, I am interested to see how the GSOC project works out, or > whatever happens with this libxo project. I also wanted to highlight an issue that Simon brought up in the initial message for this thread: "The main open issue (assuming this functionality is still desired) is support of wide characters. We figure the worst case solution is a sed(1) script to generate the wide version of the API from the normal one, but perhaps simply always using UTF8 would be a better solution?" In my own experiments I've pretended that all the output 'lpq' generates is simple ascii, although that's not necessarily true. The filename field is set by whatever generated the output, which might be a PC or Mac which may or may not be unicode-aware. The field might be gibberish (aka "invalid unicode"). Obviously "pretending" is the wrong way to handle this issue. -- Garance Alistair Drosehn = drosih@rpi.edu Senior Systems Programmer or gad@FreeBSD.org Rensselaer Polytechnic Institute; Troy, NY; USA From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 03:46:53 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7C064CA3; Wed, 30 Jul 2014 03:46:53 +0000 (UTC) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2lp0236.outbound.protection.outlook.com [207.46.163.236]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9650E2F4C; Wed, 30 Jul 2014 03:46:51 +0000 (UTC) Received: from CO2PR05CA052.namprd05.prod.outlook.com (10.141.241.180) by BLUPR05MB723.namprd05.prod.outlook.com (10.141.207.153) with Microsoft SMTP Server (TLS) id 15.0.995.14; Wed, 30 Jul 2014 03:46:44 +0000 Received: from BN1BFFO11FD032.protection.gbl (2a01:111:f400:7c10::1:140) by CO2PR05CA052.outlook.office365.com (2a01:111:e400:1429::52) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Wed, 30 Jul 2014 03:46:44 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BN1BFFO11FD032.mail.protection.outlook.com (10.58.144.95) with Microsoft SMTP Server (TLS) id 15.0.990.10 via Frontend Transport; Wed, 30 Jul 2014 03:46:43 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Tue, 29 Jul 2014 20:46:42 -0700 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6U3kfn93941; Tue, 29 Jul 2014 20:46:41 -0700 (PDT) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id 46ABE580A2; Tue, 29 Jul 2014 20:46:41 -0700 (PDT) To: Jos Backus Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> <20140729232338.40AA6580A2@chaos.jnpr.net> Comments: In-reply-to: Jos Backus message dated "Tue, 29 Jul 2014 16:30:53 -0700." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Tue, 29 Jul 2014 20:46:41 -0700 Message-ID: <20140730034641.46ABE580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(199002)(189002)(51704005)(31966008)(21056001)(76482001)(74662001)(69596002)(105596002)(106466001)(74502001)(46102001)(88136002)(77982001)(81156004)(85306003)(93546004)(110136001)(95666004)(79102001)(84676001)(57986006)(93916002)(33656002)(68736004)(102836001)(47776003)(20776003)(6806004)(85852003)(558084003)(81342001)(97736001)(48376002)(86362001)(64706001)(83322001)(81542001)(90896003)(83072002)(102176002)(50986999)(92566001)(76506005)(89996001)(4396001)(87286001)(87936001)(62966002)(104166001)(92726001)(99396002)(101356003)(50466002)(50226001)(70486001)(80022001)(77156001)(107046002)(93886003)(76176999)(42262001); DIR:OUT; SFP:; SCL:1; SRVR:BLUPR05MB723; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0288CD37D9 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=sjg@juniper.net; X-OriginatorOrg: juniper.net Cc: arch@freebsd.org, Adrian Chadd , Jordan Hubbard X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 03:46:53 -0000 On Tue, 29 Jul 2014 16:30:53 -0700, Jos Backus writes: >> You certainly cannot wait for all of it to arrive before you start >> rendering. > >Understood. This is why a serialization output format that supports >streaming data is useful. Indeed; XML works fine for that. From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 04:44:25 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E6818F5C for ; Wed, 30 Jul 2014 04:44:24 +0000 (UTC) Received: from mail-pd0-f182.google.com (mail-pd0-f182.google.com [209.85.192.182]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B54CA2545 for ; Wed, 30 Jul 2014 04:44:24 +0000 (UTC) Received: by mail-pd0-f182.google.com with SMTP id fp1so781456pdb.41 for ; Tue, 29 Jul 2014 21:44:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=lFik24OAeNk3PBNZVzSAvxCvH8v13Ju9n6/ZSaOLSWQ=; b=Myd1kCyCr3INdG2nm1ePjQPqfdoqPNIRVJXzWudNSo4jZcVELNS9vgEQ6zbzwInDPz mqHzknyBV3l6kJSz+TpAVo/hqGa1wErmY+RvgV4H3A9tpAZ/b3F1eFanZBfcXhOZ5e91 /akWmdlR0RMqTLa79PwamNG0m1gWy2FM61ul4czFcyHrjgUxEGZbtKO3jlC680MfsV+z 21MUs7foKrUTnuYo4li4G5I2cLTKKScVKeZGJ/vL1r8owFX2t0obQbziP0PPH7tHmcxn G+c903n23m+DfnGyVMmH8IhFU3pL9iJYV8YPAytHFBkkhlKyeSkPXPp8Enbodi7yORtM iIeA== X-Gm-Message-State: ALoCoQn5tYcs1J+Y5f5rj7HfzG5ZXWGWA7CetLdA0ZAGG8uAuoThLTz7k7MnG8hnXGMP3mAqt4ns MIME-Version: 1.0 X-Received: by 10.66.161.194 with SMTP id xu2mr1664158pab.128.1406695458122; Tue, 29 Jul 2014 21:44:18 -0700 (PDT) Received: by 10.70.101.41 with HTTP; Tue, 29 Jul 2014 21:44:17 -0700 (PDT) Received: by 10.70.101.41 with HTTP; Tue, 29 Jul 2014 21:44:17 -0700 (PDT) In-Reply-To: <20140730034641.46ABE580A2@chaos.jnpr.net> References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> <20140729232338.40AA6580A2@chaos.jnpr.net> <20140730034641.46ABE580A2@chaos.jnpr.net> Date: Tue, 29 Jul 2014 21:44:17 -0700 Message-ID: Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Jos Backus To: "Simon J. Gerraty" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: arch@freebsd.org, Adrian Chadd , Jordan Hubbard X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 04:44:25 -0000 On Jul 29, 2014 8:46 PM, "Simon J. Gerraty" wrote: > > > On Tue, 29 Jul 2014 16:30:53 -0700, Jos Backus writes: > >> You certainly cannot wait for all of it to arrive before you start > >> rendering. > > > >Understood. This is why a serialization output format that supports > >streaming data is useful. > > Indeed; XML works fine for that. Not to beat a dead horse, but so does YAML, and it's more lightweight/less verbose so I personally find it more elegant. But sure, XML would work as well. Jos From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 05:35:22 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A0BD050D for ; Wed, 30 Jul 2014 05:35:22 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1lp0139.outbound.protection.outlook.com [207.46.163.139]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4441B2AEB for ; Wed, 30 Jul 2014 05:35:21 +0000 (UTC) Received: from BY2PR05CA002.namprd05.prod.outlook.com (10.242.32.32) by DM2PR05MB735.namprd05.prod.outlook.com (10.141.178.24) with Microsoft SMTP Server (TLS) id 15.0.995.14; Wed, 30 Jul 2014 05:35:07 +0000 Received: from BN1AFFO11FD046.protection.gbl (2a01:111:f400:7c10::154) by BY2PR05CA002.outlook.office365.com (2a01:111:e400:2c2a::32) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Wed, 30 Jul 2014 05:35:07 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BN1AFFO11FD046.mail.protection.outlook.com (10.58.53.61) with Microsoft SMTP Server (TLS) id 15.0.990.10 via Frontend Transport; Wed, 30 Jul 2014 05:35:07 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Tue, 29 Jul 2014 22:35:04 -0700 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6U5Yln69494; Tue, 29 Jul 2014 22:34:47 -0700 (PDT) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id DCE8D580A2; Tue, 29 Jul 2014 22:34:46 -0700 (PDT) To: Alfred Perlstein Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <53D85495.4050408@mu.org> References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> <20140729230345.31E9B580A2@chaos.jnpr.net> <53D85495.4050408@mu.org> Comments: In-reply-to: Alfred Perlstein message dated "Tue, 29 Jul 2014 19:12:37 -0700." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Tue, 29 Jul 2014 22:34:46 -0700 Message-ID: <20140730053446.DCE8D580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(51704005)(189002)(199002)(102836001)(20776003)(64706001)(50986999)(69596002)(76176999)(99396002)(74502001)(106466001)(68736004)(81156004)(86362001)(107046002)(81542001)(79102001)(85306003)(74662001)(81342001)(70486001)(105596002)(101356003)(93916002)(93886003)(46102001)(50466002)(80022001)(97736001)(31966008)(110136001)(77982001)(76482001)(33656002)(76506005)(19580395003)(84676001)(4396001)(77156001)(50226001)(85852003)(90896003)(83072002)(83322001)(92726001)(89996001)(88136002)(92566001)(21056001)(47776003)(87286001)(6806004)(57986006)(104166001)(102176002)(15975445006)(95666004)(93546004)(48376002)(62966002)(87936001)(42262001); DIR:OUT; SFP:; SCL:1; SRVR:DM2PR05MB735; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0288CD37D9 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=sjg@juniper.net; X-OriginatorOrg: juniper.net Cc: freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 05:35:22 -0000 On Tue, 29 Jul 2014 19:12:37 -0700, Alfred Perlstein writes: >Is JSON not handled? See the start of the thread. TXT,XML,JSON and HTML are all handled. I assumed (and Phil confirmed) that YAML *could* be added, but no one has shown a use-case for it yet. The point is the API is rich enough to cover it. >How many utils are converted over at this point? Only 'w' - as a demo - again see the start of the thread. We wanted to get buyin before introducing churn. >Have you seen the GSOC project which aims to do this as well? "machine I though the GSOC projects were not "approved"? >readable output from userland utilities" -> >https://www.google-melange.com/gsoc/project/details/google/gsoc2014/zarko_korc >hev/5676830073815040 I took another look - where are the details? From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 05:41:50 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 388FB5DD for ; Wed, 30 Jul 2014 05:41:50 +0000 (UTC) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2lp0235.outbound.protection.outlook.com [207.46.163.235]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F2B2C2B86 for ; Wed, 30 Jul 2014 05:41:49 +0000 (UTC) Received: from CO2PR05CA017.namprd05.prod.outlook.com (10.141.241.145) by CO2PR05MB731.namprd05.prod.outlook.com (10.141.228.21) with Microsoft SMTP Server (TLS) id 15.0.995.14; Wed, 30 Jul 2014 05:41:41 +0000 Received: from BL2FFO11FD036.protection.gbl (2a01:111:f400:7c09::163) by CO2PR05CA017.outlook.office365.com (2a01:111:e400:1429::17) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Wed, 30 Jul 2014 05:41:41 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BL2FFO11FD036.mail.protection.outlook.com (10.173.161.132) with Microsoft SMTP Server (TLS) id 15.0.990.10 via Frontend Transport; Wed, 30 Jul 2014 05:41:40 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Tue, 29 Jul 2014 22:41:40 -0700 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6U5fdn74109; Tue, 29 Jul 2014 22:41:39 -0700 (PDT) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id CE881580A2; Tue, 29 Jul 2014 22:41:38 -0700 (PDT) To: Garance A Drosehn Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <448BB170-6DDC-442B-A5C5-973EC35FF7B9@rpi.edu> References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> <20140729230345.31E9B580A2@chaos.jnpr.net> <53D85495.4050408@mu.org> <448BB170-6DDC-442B-A5C5-973EC35FF7B9@rpi.edu> Comments: In-reply-to: Garance A Drosehn message dated "Tue, 29 Jul 2014 23:19:39 -0400." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Tue, 29 Jul 2014 22:41:38 -0700 Message-ID: <20140730054138.CE881580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(189002)(199002)(76176999)(86362001)(81156004)(50986999)(57986006)(79102001)(50466002)(101356003)(87286001)(106466001)(77982001)(107046002)(93916002)(80022001)(104166001)(102836001)(93546004)(48376002)(105596002)(81542001)(93886003)(87936001)(110136001)(76482001)(33656002)(84676001)(77156001)(68736004)(99396002)(6806004)(81342001)(74662001)(31966008)(88136002)(74502001)(50226001)(102176002)(92566001)(85852003)(69596002)(62966002)(4396001)(46102001)(2171001)(95666004)(70486001)(97736001)(85306003)(76506005)(89996001)(64706001)(83072002)(90896003)(21056001)(92726001)(47776003)(20776003)(83322001)(42262001); DIR:OUT; SFP:; SCL:1; SRVR:CO2PR05MB731; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0288CD37D9 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=sjg@juniper.net; X-OriginatorOrg: juniper.net Cc: Alfred Perlstein , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 05:41:50 -0000 On Tue, 29 Jul 2014 23:19:39 -0400, Garance A Drosehn writes: >One question that comes up is how to organize the data in the output the >command creates. For instance, my experiment is just trying to mimic a Good point and frankly a good reason to have the work done by someone who's been doing exactly that sort of thing (successfully) for well over a decade. >I personally don't expect unix commands to output HTML, and I find XML a >bit >unwieldy to work with. But in my own bikeshed I'm interested in formats Both are horrendous formats ;-) but there are abundent tools to work with them and json too I believe. A modern browser with off the shell javacript/json "thingies" (you can tell I'm not a web developer ;-) can allow some very slick stuff with the sort of info we can output from our UI - and this lib let's any app do the same thing. --sjg From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 05:54:46 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BE5087A3 for ; Wed, 30 Jul 2014 05:54:46 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1blp0188.outbound.protection.outlook.com [207.46.163.188]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6040C2CCC for ; Wed, 30 Jul 2014 05:54:45 +0000 (UTC) Received: from BY2PR05CA051.namprd05.prod.outlook.com (10.141.250.41) by DM2PR05MB736.namprd05.prod.outlook.com (10.141.178.25) with Microsoft SMTP Server (TLS) id 15.0.995.14; Wed, 30 Jul 2014 05:54:36 +0000 Received: from BL2FFO11FD025.protection.gbl (2a01:111:f400:7c09::139) by BY2PR05CA051.outlook.office365.com (2a01:111:e400:2c5f::41) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Wed, 30 Jul 2014 05:54:36 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BL2FFO11FD025.mail.protection.outlook.com (10.173.161.104) with Microsoft SMTP Server (TLS) id 15.0.990.10 via Frontend Transport; Wed, 30 Jul 2014 05:54:35 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Tue, 29 Jul 2014 22:54:33 -0700 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6U5sUn83834; Tue, 29 Jul 2014 22:54:31 -0700 (PDT) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id A78B1580A2; Tue, 29 Jul 2014 22:54:30 -0700 (PDT) To: Garance A Drosehn Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <1092D020-F782-4587-A199-C7313BC2CD20@rpi.edu> References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> <20140729230345.31E9B580A2@chaos.jnpr.net> <53D85495.4050408@mu.org> <448BB170-6DDC-442B-A5C5-973EC35FF7B9@rpi.edu> <1092D020-F782-4587-A199-C7313BC2CD20@rpi.edu> Comments: In-reply-to: Garance A Drosehn message dated "Tue, 29 Jul 2014 23:40:14 -0400." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Tue, 29 Jul 2014 22:54:30 -0700 Message-ID: <20140730055430.A78B1580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(164054003)(189002)(199002)(51704005)(2171001)(70486001)(88136002)(99396002)(87286001)(87936001)(57986006)(74502001)(31966008)(92726001)(81342001)(77982001)(97736001)(76506005)(90896003)(74662001)(110136001)(79102001)(76482001)(85852003)(89996001)(46102001)(83072002)(48376002)(50466002)(85306003)(106466001)(68736004)(69596002)(81156004)(21056001)(93916002)(86362001)(95666004)(4396001)(50226001)(105596002)(20776003)(47776003)(93886003)(83322001)(62966002)(101356003)(81542001)(64706001)(50986999)(76176999)(80022001)(33656002)(92566001)(93546004)(84676001)(77156001)(102836001)(107046002)(104166001)(102176002)(6806004)(42262001); DIR:OUT; SFP:; SCL:1; SRVR:DM2PR05MB736; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0288CD37D9 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=sjg@juniper.net; X-OriginatorOrg: juniper.net Cc: phil@juniper.net, freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 05:54:46 -0000 [+phil who somehow got dropped] On Tue, 29 Jul 2014 23:40:14 -0400, Garance A Drosehn writes: >I also wanted to highlight an issue that Simon brought up in the initial >message for this thread: > > "The main open issue (assuming this functionality is still desired) > is support of wide characters. > > We figure the worst case solution is a sed(1) script to generate > the wide version of the API from the normal one, but perhaps simply > always using UTF8 would be a better solution?" Thanks, good not to lose sight of that in all the color discussions ;-) >In my own experiments I've pretended that all the output 'lpq' generates >is simple ascii, although that's not necessarily true. The filename >field is set by whatever generated the output, which might be a PC or Mac >which may or may not be unicode-aware. The field might be gibberish (aka >"invalid unicode"). Obviously "pretending" is the wrong way to handle >this issue. Indeed. UTF-8 has its attractions since i18n cannot simply be ignored in this case. From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 06:59:57 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8CE703AA; Wed, 30 Jul 2014 06:59:57 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 47A892286; Wed, 30 Jul 2014 06:59:56 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s6U6xrbv004508 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 29 Jul 2014 23:59:54 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s6U6xrWE004507; Tue, 29 Jul 2014 23:59:53 -0700 (PDT) (envelope-from jmg) Date: Tue, 29 Jul 2014 23:59:53 -0700 From: John-Mark Gurney To: Adrian Chadd Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Message-ID: <20140730065953.GJ43962@funkthat.com> Mail-Followup-To: Adrian Chadd , Jos Backus , "freebsd-arch@freebsd.org" , Jordan Hubbard References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Tue, 29 Jul 2014 23:59:54 -0700 (PDT) Cc: Jordan Hubbard , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 06:59:57 -0000 Adrian Chadd wrote this message on Mon, Jul 28, 2014 at 22:57 -0700: > Holy ... ! > > What really matter is whether the library API that you're going to > shoehorn into plenty of utilities is expressive enough to express a > whole bunch of different output types. > > So it doesn't matter if you want JSON, or YAML, or XML, or the native > tool output. The trick is whether the library API is good enough. Actually, it does.. there are somethings that you can express in XML or others that you can't express in JSON... Like there is no binary data type in JSON... You only have unicode strings... I hope the library that outputs JSON ensures that all strings are valid UTF-8.. If not, I can imagine that someone will have a security issue due to this... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 07:15:06 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 21994AA1; Wed, 30 Jul 2014 07:15:06 +0000 (UTC) Received: from mail-wi0-x22e.google.com (mail-wi0-x22e.google.com [IPv6:2a00:1450:400c:c05::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7E13F2510; Wed, 30 Jul 2014 07:15:05 +0000 (UTC) Received: by mail-wi0-f174.google.com with SMTP id d1so7011478wiv.1 for ; Wed, 30 Jul 2014 00:15:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=wC3GzXDBN0AxWU8bVhvZzazY4hy1PEm+juhc+cJcHyM=; b=AlrlL+5bdInJnK9dAJsR33VfUlaOyJjCN+ckT3W/cd+lzvzCFZ/z7ljBr0dSwTpdgd 4zXA1OjlzRLK6b3hsYR7PIg5wTF0GuscYw4wOu5vlbgjUHsP7zMzqHPmO1f5OKlXE1tn +LNMjAa1qSDfMviC2Z6ZbmMmP2NCc33/l/wC75HKIReytRxkXA3TWgr8MCg+RN9Qu4Nb x9SV6lrSs+cEhXLnxGmvyESy+rlYkYpXfTi03Q/iGURE2NZZlFNxJAkTD4Ij21W/1xx5 8xSEOE7I9BxASEHgFRhToMDS4oxQae/kUWfPkC/B+3E/NhdagRt88/FZRfLhLaDrxvbQ 2tEA== X-Received: by 10.180.79.202 with SMTP id l10mr4240249wix.18.1406704503522; Wed, 30 Jul 2014 00:15:03 -0700 (PDT) Received: from ivaldir.etoilebsd.net ([2001:41d0:8:db4c::1]) by mx.google.com with ESMTPSA id ex4sm52234556wic.2.2014.07.30.00.15.02 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 30 Jul 2014 00:15:02 -0700 (PDT) Sender: Baptiste Daroussin Date: Wed, 30 Jul 2014 09:15:00 +0200 From: Baptiste Daroussin To: Jos Backus Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Message-ID: <20140730071500.GF37672@ivaldir.etoilebsd.net> References: <20140728055336.GJ50802@ivaldir.etoilebsd.net> <20140729232338.40AA6580A2@chaos.jnpr.net> <20140730034641.46ABE580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="bGR76rFJjkSxVeRa" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Cc: arch@freebsd.org, Adrian Chadd , Jordan Hubbard , "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 07:15:06 -0000 --bGR76rFJjkSxVeRa Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jul 29, 2014 at 09:44:17PM -0700, Jos Backus wrote: > On Jul 29, 2014 8:46 PM, "Simon J. Gerraty" wrote: > > > > > > On Tue, 29 Jul 2014 16:30:53 -0700, Jos Backus writes: > > >> You certainly cannot wait for all of it to arrive before you start > > >> rendering. > > > > > >Understood. This is why a serialization output format that supports > > >streaming data is useful. > > > > Indeed; XML works fine for that. >=20 > Not to beat a dead horse, but so does YAML, and it's more lightweight/less > verbose so I personally find it more elegant. But sure, XML would work as > well. >=20 YAML is not more lightweight at all, it is really heavy to parse compared to XML or JSON. regards, Bapt --bGR76rFJjkSxVeRa Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlPYm3QACgkQ8kTtMUmk6EzsyACfVx9xexgW11eZU0i1T8ZYWL2S ut4An10j2MGKsx3HRd7/smhevaGfggad =lgVD -----END PGP SIGNATURE----- --bGR76rFJjkSxVeRa-- From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 07:19:35 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 91DDDE89; Wed, 30 Jul 2014 07:19:35 +0000 (UTC) Received: from mail-wi0-x232.google.com (mail-wi0-x232.google.com [IPv6:2a00:1450:400c:c05::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 05F112547; Wed, 30 Jul 2014 07:19:34 +0000 (UTC) Received: by mail-wi0-f178.google.com with SMTP id hi2so1771884wib.11 for ; Wed, 30 Jul 2014 00:19:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:subject:message-id:mail-followup-to :mime-version:content-type:content-disposition:user-agent; bh=r3KPDzewZ3B9o79stgpbkiOM4GBJ4bQre83Vki7MRJQ=; b=oXe9br7hfMOWwarQMkCypipSdHxOkJiUd3yMAYKUTSgJHgnvi0N5sXyh3kKC7N8udn j07jbijZcJ0au4+d7MfnYXLZP0828uy8jPmQaL1AZvwH3GGf3IaWm+AVEnEe44oQmLIr DD+5LdShFfRRkJ/6M0v99mk+yTS3zQV9QMmXS1Oly69jMWgadsX7hE/FaX8TVPlD8CE0 Pl6cf31PUrwjUJULVM1zK/P7v/MdgkUNxBChi++HD8AvZKSJCPqr79po61ZNoFxW9ITn kIHRq3HLnmP49SCAHZHLS6RJ4w5biL0IFI90bIhCX7RC/hEuPCaeIvENUYn/heNrnBNK mbZQ== X-Received: by 10.194.89.168 with SMTP id bp8mr3157579wjb.73.1406704773315; Wed, 30 Jul 2014 00:19:33 -0700 (PDT) Received: from pc5.home (adhd145.neoplus.adsl.tpnet.pl. [79.184.159.145]) by mx.google.com with ESMTPSA id o2sm52171524wij.24.2014.07.30.00.19.32 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 30 Jul 2014 00:19:32 -0700 (PDT) Sender: =?UTF-8?Q?Edward_Tomasz_Napiera=C5=82a?= Date: Wed, 30 Jul 2014 09:19:33 +0200 From: Edward Tomasz =?utf-8?Q?Napiera=C5=82a?= To: freebsd-current@FreeBSD.org, freebsd-arch@FreeBSD.org Subject: [CFT] Autofs. Message-ID: <20140730071933.GA20122@pc5.home> Mail-Followup-To: freebsd-current@FreeBSD.org, freebsd-arch@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 07:19:35 -0000 At the link below you will find a patch that adds the new automounter. The patch is against yesterdays 11.0-CURRENT. http://people.freebsd.org/~trasz/autofs-head-20140729.diff Slides that explain the project scope and deliverables are here: http://people.freebsd.org/~trasz/autofs.pdf Testing is welcome. Please start with manual pages, eg. automount(8). Note that you need not only to rebuild both kernel and world, but also to run mergemaster, to install required /etc files. To run at startup, add 'autofs_enable="YES"' to /etc/rc.conf. This project is being sponsored by FreeBSD Foundation. From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 07:22:56 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F2874350 for ; Wed, 30 Jul 2014 07:22:56 +0000 (UTC) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id B0DCD25F5 for ; Wed, 30 Jul 2014 07:22:56 +0000 (UTC) Received: from critter.freebsd.dk (unknown [192.168.48.2]) by phk.freebsd.dk (Postfix) with ESMTP id 889591578 for ; Wed, 30 Jul 2014 07:22:54 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.14.9/8.14.9) with ESMTP id s6U7MsVw004607 for ; Wed, 30 Jul 2014 07:22:54 GMT (envelope-from phk@phk.freebsd.dk) To: "freebsd-arch@freebsd.org" Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-reply-to: <20140730065953.GJ43962@funkthat.com> From: "Poul-Henning Kamp" References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> <20140730065953.GJ43962@funkthat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-ID: <4605.1406704974.1@critter.freebsd.dk> Content-Transfer-Encoding: 8bit Date: Wed, 30 Jul 2014 07:22:54 +0000 Message-ID: <4606.1406704974@critter.freebsd.dk> X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 07:22:57 -0000 -------- Some of you may recall that I did a keynote at EuroBSDcon 2010 called "Software Tools -- Mission Accomplished or Mission Failure ?" I tried to do a status review of 40 years of UNIX and Software Tools, in part inspired by what I saw as a "user" of the platform while working on Varnish. Historical analysis is useless if it doesn't point us into the future, and that I did, concluding that we needed to move beyond 80 char wide ASCII text-files finishing my talk with this "ridiculous" idea: Solution: Change kernel & userland to understand XML instead of flat ASCII. grep --tag H3 ”crazy idea” index.html My keynote doesn't seem to exist on the web (I'm pretty sure it was video-taped ?) but I've dug out my slides: http://phk.freebsd.dk/pubs/EuroBSDcon2010_SoftwareTools.pdf In case anybody is interesting in the deeper and wider perspective on why libxo is long overdue. Thumbs up for the people finally realizing it. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 16:18:48 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6424BBB9 for ; Wed, 30 Jul 2014 16:18:48 +0000 (UTC) Received: from mail-pa0-f41.google.com (mail-pa0-f41.google.com [209.85.220.41]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 32709249C for ; Wed, 30 Jul 2014 16:18:47 +0000 (UTC) Received: by mail-pa0-f41.google.com with SMTP id rd3so1790546pab.14 for ; Wed, 30 Jul 2014 09:18:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=UZW1OmtGRD2ujzPQVEjBztgLML8ObdvzmK/N+ZPxdIE=; b=E4rEl3Jg9Ek53AnQp4K9EsKnyvnfjwr59Mt/cLgV+5Fm3ARkhJU+BsOt1h4VgrAQTE 7/K+Jk7gaToxSU85N0FgWVkup/wfjA25FJlcZHNryiEeHtQh0bUXInDEGpAfhmMvoI9P LdcO7CIHOeM6rDk9Yy0lsMkrDRjmjsYPwRbnq6AzTN8H40bIYX3EfvXqBNbaW91M6dIB Gw+9+iUE0G9XfAKgN3qsAb6OdcwC7B5QBCKbI8Tu8rEtFjw4EqG+OJUUqiVOHIqEvRCv 3+PmgkH3EWxzWeqbLFQvb4uCbgIroVODIrphWI4mrCZl1edmwkyeE01b/hiyE2nF39UE iiQA== X-Gm-Message-State: ALoCoQm6El46QW9Qt62i2LCe5fh6UjkIzbh6LetwrEr2FQoCt8dZ9UH5Jdlk+N4TBxmV5XymKlxF MIME-Version: 1.0 X-Received: by 10.70.38.4 with SMTP id c4mr6207318pdk.108.1406737120892; Wed, 30 Jul 2014 09:18:40 -0700 (PDT) Received: by 10.70.101.41 with HTTP; Wed, 30 Jul 2014 09:18:40 -0700 (PDT) Received: by 10.70.101.41 with HTTP; Wed, 30 Jul 2014 09:18:40 -0700 (PDT) In-Reply-To: <20140730071500.GF37672@ivaldir.etoilebsd.net> References: <20140728055336.GJ50802@ivaldir.etoilebsd.net> <20140729232338.40AA6580A2@chaos.jnpr.net> <20140730034641.46ABE580A2@chaos.jnpr.net> <20140730071500.GF37672@ivaldir.etoilebsd.net> Date: Wed, 30 Jul 2014 09:18:40 -0700 Message-ID: Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Jos Backus To: Baptiste Daroussin Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: arch@freebsd.org, Adrian Chadd , Jordan Hubbard , "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 16:18:48 -0000 On Jul 30, 2014 12:15 AM, "Baptiste Daroussin" wrote: > > On Tue, Jul 29, 2014 at 09:44:17PM -0700, Jos Backus wrote: > > On Jul 29, 2014 8:46 PM, "Simon J. Gerraty" wrote: > > > > > > > > > On Tue, 29 Jul 2014 16:30:53 -0700, Jos Backus writes: > > > >> You certainly cannot wait for all of it to arrive before you start > > > >> rendering. > > > > > > > >Understood. This is why a serialization output format that supports > > > >streaming data is useful. > > > > > > Indeed; XML works fine for that. > > > > Not to beat a dead horse, but so does YAML, and it's more lightweight/less > > verbose so I personally find it more elegant. But sure, XML would work as > > well. > > > YAML is not more lightweight at all, it is really heavy to parse compared to > XML or JSON. By lightweight I meant syntax verbosity, not computational load (although it seems easy to emit). It's a more flexible format, and that comes with a certain price. The question is whether that flexibility is needed or useful. If JSON can't be used because of its limitations, I would personally prefer the less verbose YAML over XML. Jos. > > regards, > Bapt From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 17:04:02 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4B78260E; Wed, 30 Jul 2014 17:04:02 +0000 (UTC) Received: from mail-we0-x230.google.com (mail-we0-x230.google.com [IPv6:2a00:1450:400c:c03::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A73D9293F; Wed, 30 Jul 2014 17:04:01 +0000 (UTC) Received: by mail-we0-f176.google.com with SMTP id q58so1537878wes.21 for ; Wed, 30 Jul 2014 10:03:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=UB6FDSCMACqevbFpJDC+etOjkro6BgPX8EZZsE0CpVQ=; b=ytGHZoxN/g6GQDlykXl6boeLE+K62idjUwl3fmMR0TX1tpXKSHh+aVU8kDgQBvHXHm LjZJxCgdV+5XCgR1kpio76RXkfxMI8dLdmeuKF55D2SVac4x+vhD0LW4pF2pyx5W36z3 vESjXJOwIYdUY8fAsO7jt0wOs6+Fl669s/OMtJjySQWgiQ3NpQYsY6tes3plOBvfDmwv hfa4PzhF5XqwaswCZGMe3yd1W3ygAXm6BQgFldCUPg7D1pPSrX8HHiVNtrMVuN4TRAMw gSpGrX8GXRn5sGlueoqLDOKrVO+Ab5UeSDTnjJUPguiRyAP81E6XWfEpGJW3HVNkbamJ PWGQ== X-Received: by 10.194.77.177 with SMTP id t17mr7841449wjw.55.1406739839920; Wed, 30 Jul 2014 10:03:59 -0700 (PDT) Received: from ivaldir.etoilebsd.net ([2001:41d0:8:db4c::1]) by mx.google.com with ESMTPSA id wi9sm6831485wjc.23.2014.07.30.10.03.58 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 30 Jul 2014 10:03:58 -0700 (PDT) Sender: Baptiste Daroussin Date: Wed, 30 Jul 2014 19:03:56 +0200 From: Baptiste Daroussin To: Jos Backus Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Message-ID: <20140730170355.GM37672@ivaldir.etoilebsd.net> References: <20140729232338.40AA6580A2@chaos.jnpr.net> <20140730034641.46ABE580A2@chaos.jnpr.net> <20140730071500.GF37672@ivaldir.etoilebsd.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Nj4mAaUCx+wbOcQD" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Cc: arch@freebsd.org, Adrian Chadd , Jordan Hubbard , "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 17:04:02 -0000 --Nj4mAaUCx+wbOcQD Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Jul 30, 2014 at 09:18:40AM -0700, Jos Backus wrote: > On Jul 30, 2014 12:15 AM, "Baptiste Daroussin" wrote: > > > > On Tue, Jul 29, 2014 at 09:44:17PM -0700, Jos Backus wrote: > > > On Jul 29, 2014 8:46 PM, "Simon J. Gerraty" wrote: > > > > > > > > > > > > On Tue, 29 Jul 2014 16:30:53 -0700, Jos Backus writes: > > > > >> You certainly cannot wait for all of it to arrive before you sta= rt > > > > >> rendering. > > > > > > > > > >Understood. This is why a serialization output format that supports > > > > >streaming data is useful. > > > > > > > > Indeed; XML works fine for that. > > > > > > Not to beat a dead horse, but so does YAML, and it's more > lightweight/less > > > verbose so I personally find it more elegant. But sure, XML would work > as > > > well. > > > > > YAML is not more lightweight at all, it is really heavy to parse compar= ed > to > > XML or JSON. >=20 > By lightweight I meant syntax verbosity, not computational load (although > it seems easy to emit). It's a more flexible format, and that comes with a > certain price. The question is whether that flexibility is needed or > useful. If JSON can't be used because of its limitations, I would > personally prefer the less verbose YAML over XML. >=20 About json what limitation are you talking about? In yaml you have 2 syntax, on which is inconsistent but user friendly and t= he other which as ugly as XML imho this_is_string: treu this__bool: true so_if_i_want_a_string_true_i_need_quote: "true" If I want to be consistent I need to use the canonical form of yaml: --- !!map { ? !!str "so_if_i_want_a_string_true_i_need_quote" : !!str "true", ? !!str "this__bool" : !!bool "true", ? !!str "this_is_string" : !!str "treu", } and now this is very very ugly :( Plus yaml is context dependant and space dependant resulting in people gett= ing lost about: "Why this yaml is not valid": hu: ha hi: - test - test2 Or why this one is not valid either? hu: ha hi: test I have been there with pkg(8) after being a huge suppoter of YAML I'm now m= ore moderated :) YAML was not machine friendly at all in the end and very very error prone f= or humans :( regards, Bapt --Nj4mAaUCx+wbOcQD Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlPZJXsACgkQ8kTtMUmk6EwPrgCeIxpG5zQBeuKRa/3McD3UnNjy trAAn3/K/mbSzbb2FWzc4mfkTqiXt/wd =e/5o -----END PGP SIGNATURE----- --Nj4mAaUCx+wbOcQD-- From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 18:49:00 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 89400D62 for ; Wed, 30 Jul 2014 18:49:00 +0000 (UTC) Received: from mail-pa0-f41.google.com (mail-pa0-f41.google.com [209.85.220.41]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 56CC2261D for ; Wed, 30 Jul 2014 18:49:00 +0000 (UTC) Received: by mail-pa0-f41.google.com with SMTP id rd3so2044407pab.0 for ; Wed, 30 Jul 2014 11:48:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=T9whZRmsPpmIbV/HNP4btf9NoiQAVWP6aKlCTVrGtvc=; b=KNfJznBpICXWo4ki8UEOCd8ZHp2ns90xUTMPUlOblOxYG30i+xwZUgjYFuUzcDACVg BWXyrbtcF3bMQ9FexVy/H9y5ycP3HdhqSBybdkILPOB7bRkscHE0k8V1mddLQ8klf8D7 dtLKMJhVk0jvppqH4kBr2pY03HsJxZNZCQ79/nkyURLBPThBzYC7bjGGMNRT8FMkPbKB u2m2bImaVZ25UJPej0XXeSBDfMFzVdBWEhdPCM0zFMglj0cee5lLrQVGdavQTjZSn0KX BV3cY8pF84agBS0cKdhADlCZiSIbbvyYioizrxhtoXE4qeiJDxNu/4+2zDycmnKh5cnJ dWkw== X-Gm-Message-State: ALoCoQmnG6mGgv4QMjjVU4NNPefMxU3mFos/oR1R0c5H2rd3RN31uef+K6CkwZN6J0L/MWkMEXLY MIME-Version: 1.0 X-Received: by 10.68.113.133 with SMTP id iy5mr6922696pbb.135.1406746139556; Wed, 30 Jul 2014 11:48:59 -0700 (PDT) Received: by 10.70.101.41 with HTTP; Wed, 30 Jul 2014 11:48:59 -0700 (PDT) Received: by 10.70.101.41 with HTTP; Wed, 30 Jul 2014 11:48:59 -0700 (PDT) In-Reply-To: <20140730170355.GM37672@ivaldir.etoilebsd.net> References: <20140729232338.40AA6580A2@chaos.jnpr.net> <20140730034641.46ABE580A2@chaos.jnpr.net> <20140730071500.GF37672@ivaldir.etoilebsd.net> <20140730170355.GM37672@ivaldir.etoilebsd.net> Date: Wed, 30 Jul 2014 11:48:59 -0700 Message-ID: Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Jos Backus To: Baptiste Daroussin Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: arch@freebsd.org, Adrian Chadd , Jordan Hubbard , "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 18:49:00 -0000 On Jul 30, 2014 10:04 AM, "Baptiste Daroussin" wrote: > > On Wed, Jul 30, 2014 at 09:18:40AM -0700, Jos Backus wrote: > > On Jul 30, 2014 12:15 AM, "Baptiste Daroussin" wrote: > > > > > > On Tue, Jul 29, 2014 at 09:44:17PM -0700, Jos Backus wrote: > > > > On Jul 29, 2014 8:46 PM, "Simon J. Gerraty" wrote: > > > > > > > > > > > > > > > On Tue, 29 Jul 2014 16:30:53 -0700, Jos Backus writes: > > > > > >> You certainly cannot wait for all of it to arrive before you start > > > > > >> rendering. > > > > > > > > > > > >Understood. This is why a serialization output format that supports > > > > > >streaming data is useful. > > > > > > > > > > Indeed; XML works fine for that. > > > > > > > > Not to beat a dead horse, but so does YAML, and it's more > > lightweight/less > > > > verbose so I personally find it more elegant. But sure, XML would work > > as > > > > well. > > > > > > > YAML is not more lightweight at all, it is really heavy to parse compared > > to > > > XML or JSON. > > > > By lightweight I meant syntax verbosity, not computational load (although > > it seems easy to emit). It's a more flexible format, and that comes with a > > certain price. The question is whether that flexibility is needed or > > useful. If JSON can't be used because of its limitations, I would > > personally prefer the less verbose YAML over XML. > > > About json what limitation are you talking about? Several limitations have been mentioned: no support for comments, binary data, streaming. YAML is a superset of JSON so there are things one can do with YAML that one cannot do (easily) with JSON. The question is whether those things matter enough. > In yaml you have 2 syntax, on which is inconsistent but user friendly and the > other which as ugly as XML imho > > this_is_string: treu > this__bool: true > so_if_i_want_a_string_true_i_need_quote: "true" > > If I want to be consistent I need to use the canonical form of yaml: > > --- > !!map { > ? !!str "so_if_i_want_a_string_true_i_need_quote" > : !!str "true", > ? !!str "this__bool" > : !!bool "true", > ? !!str "this_is_string" > : !!str "treu", > } > > and now this is very very ugly :( Cute, but rare. How often can one not use the easy format, and how often does one want the string" true" rather than the Boolean value? > Plus yaml is context dependant and space dependant resulting in people getting lost about: > "Why this yaml is not valid": > > hu: ha > hi: > - test > - test2 > > Or why this one is not valid either? > > hu: ha > hi: test > > > I have been there with pkg(8) after being a huge suppoter of YAML I'm now more moderated :) Sure, one has to apply some care with whitespace. This hasn't prevented Python from becoming popular so it must not be as big a deal, and I know the Ruby community uses YAML effectively quite a bit. > > YAML was not machine friendly at all in the end and very very error prone for humans :( Granted, JSON is a simpler format. Jos P. S. I've said everything I planned to say so I am going to move on from this bikeshed now. > > regards, > Bapt From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 18:53:24 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C7F44EA7; Wed, 30 Jul 2014 18:53:24 +0000 (UTC) Received: from mail-ie0-x231.google.com (mail-ie0-x231.google.com [IPv6:2607:f8b0:4001:c03::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8D75F26D2; Wed, 30 Jul 2014 18:53:24 +0000 (UTC) Received: by mail-ie0-f177.google.com with SMTP id at20so2121582iec.22 for ; Wed, 30 Jul 2014 11:53:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=690lI0wQIpTrXsPqOJ66ajjpWf+6VdgVsfBE/pP0EwE=; b=W9up9v6YfisuI3R/6J66aaqrDd4iMZ7Fslb1AsBXLHr2o0lgezxjMREQ5J3VvXh+bA F77IJs7tjwo2EnTb/Q58kFPRkICXt8qudbhypWaGrgrr7OvB4FpfmBcGBBv/Udbo37eI Ww++UiDdqWr6cm+LnbyZRRs65O9v0EGlzXuIgzsekgjTrX48Wvtnyhaj6CUHFqSudLEM zSPFFXqK4NV2EtLimoOkhkhKjEFi2GbHNI/+A8regfZvaszl0P/3m2SCIEVPum+nSjpT oLatA4h47I38Xuhh8mBPpO7FG54od/l7pmtmTYHwHLiSIr6L5A3OlrHr0yJgJAgSeecu 8Rsw== MIME-Version: 1.0 X-Received: by 10.50.43.193 with SMTP id y1mr10578848igl.32.1406746404017; Wed, 30 Jul 2014 11:53:24 -0700 (PDT) Sender: kob6558@gmail.com Received: by 10.107.163.148 with HTTP; Wed, 30 Jul 2014 11:53:23 -0700 (PDT) In-Reply-To: <20140730071933.GA20122@pc5.home> References: <20140730071933.GA20122@pc5.home> Date: Wed, 30 Jul 2014 11:53:23 -0700 X-Google-Sender-Auth: taYXqfOSHndf4oImoAvuh3IfQS0 Message-ID: Subject: Re: [CFT] Autofs. From: Kevin Oberman To: FreeBSD Current , "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 18:53:25 -0000 On Wed, Jul 30, 2014 at 12:19 AM, Edward Tomasz Napiera=C5=82a wrote: > At the link below you will find a patch that adds the new automounter. > The patch is against yesterdays 11.0-CURRENT. > > http://people.freebsd.org/~trasz/autofs-head-20140729.diff > > Slides that explain the project scope and deliverables are here: > > http://people.freebsd.org/~trasz/autofs.pdf > > Testing is welcome. Please start with manual pages, eg. automount(8). > Note that you need not only to rebuild both kernel and world, but also > to run mergemaster, to install required /etc files. To run at startup, > add 'autofs_enable=3D"YES"' to /etc/rc.conf. > > This project is being sponsored by FreeBSD Foundation. > > This may be a silly question, but can autofs properly handle fusefs file systems like ntfs-3g and exfat? -- R. Kevin Oberman, Network Engineer, Retired E-mail: rkoberman@gmail.com From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 19:17:07 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 77F2B9C6 for ; Wed, 30 Jul 2014 19:17:07 +0000 (UTC) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 614A52900 for ; Wed, 30 Jul 2014 19:17:07 +0000 (UTC) Received: from u10-2-16-021.office.norse-data.com (unknown [50.204.88.51]) by elvis.mu.org (Postfix) with ESMTPSA id C64EB346DE41; Wed, 30 Jul 2014 12:17:00 -0700 (PDT) Message-ID: <53D944F5.7000207@freebsd.org> Date: Wed, 30 Jul 2014 12:18:13 -0700 From: Alfred Perlstein Organization: FreeBSD User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.0 MIME-Version: 1.0 To: "Simon J. Gerraty" Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> <20140729230345.31E9B580A2@chaos.jnpr.net> <53D85495.4050408@mu.org> <20140730053446.DCE8D580A2@chaos.jnpr.net> In-Reply-To: <20140730053446.DCE8D580A2@chaos.jnpr.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 19:17:07 -0000 The goal of a GSOC project is to get the code into FreeBSD. The code can be seen here: https://socsvn.freebsd.org/socsvn/soc2014/zkorchev/ Since Juniper has many years of experience AND the GSOC project has many, many utils converted I'm suggesting that Juniper engage in the review process and help us get the best of both worlds in. Simon, can you help us with the review and make suggestions on what needs to be changed/augmented to get the best of both efforts in? The details for the code are here: https://socsvn.freebsd.org/socsvn/soc2014/zkorchev/ You should be able to do an svn checkout and then get diffs to see what is going on. If you require any assistance please let me know. -Alfred On 7/29/14 10:34 PM, Simon J. Gerraty wrote: > On Tue, 29 Jul 2014 19:12:37 -0700, Alfred Perlstein writes: >> Is JSON not handled? > See the start of the thread. > TXT,XML,JSON and HTML are all handled. > I assumed (and Phil confirmed) that YAML *could* be added, > but no one has shown a use-case for it yet. > The point is the API is rich enough to cover it. > >> How many utils are converted over at this point? > Only 'w' - as a demo - again see the start of the thread. > We wanted to get buyin before introducing churn. > >> Have you seen the GSOC project which aims to do this as well? "machine > I though the GSOC projects were not "approved"? > >> readable output from userland utilities" -> >> https://www.google-melange.com/gsoc/project/details/google/gsoc2014/zarko_korc >> hev/5676830073815040 > I took another look - where are the details? > > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 19:38:21 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3C333131; Wed, 30 Jul 2014 19:38:21 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 14A352AEF; Wed, 30 Jul 2014 19:38:20 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s6UJcJAu014435 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 30 Jul 2014 12:38:19 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s6UJcJUa014434; Wed, 30 Jul 2014 12:38:19 -0700 (PDT) (envelope-from jmg) Date: Wed, 30 Jul 2014 12:38:19 -0700 From: John-Mark Gurney To: Simon Gerraty Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Message-ID: <20140730193819.GM43962@funkthat.com> Mail-Followup-To: Simon Gerraty , arch@freebsd.org, sjg@freebsd.org, marcel@freebsd.org, phil@juniper.net References: <20140725044921.9F0D3580A2@chaos.jnpr.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140725044921.9F0D3580A2@chaos.jnpr.net> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Wed, 30 Jul 2014 12:38:19 -0700 (PDT) Cc: sjg@freebsd.org, arch@freebsd.org, marcel@freebsd.org, phil@juniper.net X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 19:38:21 -0000 Simon Gerraty wrote this message on Thu, Jul 24, 2014 at 21:49 -0700: > The main open issue (assuming this functionality is still desired) is > support of wide charachters. > > We figure the worst case solution is a sed(1) script to generate the wide > version of the API from the normal one, but perhaps simply always using > UTF8 would be a better solution? My vote would be to use and *enforce* UTF-8 by the API. That means if someone passes a string in, it must be properly formed UTF-8... This doesn't prevent a wide char API from also being present if someone really wants to add it.. I would also like to see support for binary data, but as I mentioned, that would prevent usage from outputing JSON... Yes, you can hex encode (why doesn't libc include a hex code/decode function) it, but then it becomes harder to autodetect typing.. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 19:45:56 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 060BC474; Wed, 30 Jul 2014 19:45:56 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1lp0142.outbound.protection.outlook.com [207.46.163.142]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 88AE12BAC; Wed, 30 Jul 2014 19:45:55 +0000 (UTC) Received: from CO2PR05CA026.namprd05.prod.outlook.com (10.141.241.154) by BY2PR05MB726.namprd05.prod.outlook.com (10.141.223.19) with Microsoft SMTP Server (TLS) id 15.0.995.14; Wed, 30 Jul 2014 19:45:41 +0000 Received: from BN1AFFO11FD010.protection.gbl (2a01:111:f400:7c10::158) by CO2PR05CA026.outlook.office365.com (2a01:111:e400:1429::26) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Wed, 30 Jul 2014 19:45:41 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BN1AFFO11FD010.mail.protection.outlook.com (10.58.52.70) with Microsoft SMTP Server (TLS) id 15.0.990.10 via Frontend Transport; Wed, 30 Jul 2014 19:45:39 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Wed, 30 Jul 2014 12:45:37 -0700 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6UJjWn16601; Wed, 30 Jul 2014 12:45:35 -0700 (PDT) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id B58FE580A2; Wed, 30 Jul 2014 12:45:22 -0700 (PDT) To: Alfred Perlstein Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <53D944F5.7000207@freebsd.org> References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> <20140729230345.31E9B580A2@chaos.jnpr.net> <53D85495.4050408@mu.org> <20140730053446.DCE8D580A2@chaos.jnpr.net> <53D944F5.7000207@freebsd.org> Comments: In-reply-to: Alfred Perlstein message dated "Wed, 30 Jul 2014 12:18:13 -0700." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Wed, 30 Jul 2014 12:45:22 -0700 Message-ID: <20140730194522.B58FE580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(199002)(189002)(51704005)(89996001)(93916002)(93546004)(92566001)(106466001)(81342001)(90896003)(104166001)(110136001)(50466002)(93886003)(4396001)(74502001)(87286001)(74662001)(102836001)(33656002)(15975445006)(107046002)(50986999)(92726001)(86362001)(83072002)(84676001)(46102001)(57986006)(85852003)(99396002)(68736004)(76176999)(50226001)(48376002)(102176002)(97736001)(76482001)(69596002)(81542001)(31966008)(62966002)(19580395003)(77156001)(21056001)(95666004)(64706001)(44976005)(6806004)(80022001)(88136002)(81156004)(70486001)(85306003)(87936001)(76506005)(101356003)(105596002)(79102001)(77982001)(83322001)(47776003)(20776003)(42262001); DIR:OUT; SFP:; SCL:1; SRVR:BY2PR05MB726; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0288CD37D9 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=sjg@juniper.net; X-OriginatorOrg: juniper.net Cc: freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 19:45:56 -0000 On Wed, 30 Jul 2014 12:18:13 -0700, Alfred Perlstein writes: >The goal of a GSOC project is to get the code into FreeBSD. > >The code can be seen here: > https://socsvn.freebsd.org/socsvn/soc2014/zkorchev/ > >Since Juniper has many years of experience AND the GSOC project has >many, many utils converted I'm suggesting that Juniper engage in the >review process and help us get the best of both worlds in. That would of course depend on whether we like what has been done ;-) The library and API are more important than how many apps have been converted. Will take a look. From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 21:28:30 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BC3A47C6 for ; Wed, 30 Jul 2014 21:28:30 +0000 (UTC) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 9C3C32740 for ; Wed, 30 Jul 2014 21:28:30 +0000 (UTC) Received: from u10-2-16-021.office.norse-data.com (unknown [50.204.88.51]) by elvis.mu.org (Postfix) with ESMTPSA id E8E07346DE13; Wed, 30 Jul 2014 14:28:29 -0700 (PDT) Message-ID: <53D963C7.8030602@freebsd.org> Date: Wed, 30 Jul 2014 14:29:43 -0700 From: Alfred Perlstein Organization: FreeBSD User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.0 MIME-Version: 1.0 To: "Simon J. Gerraty" Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> <20140729230345.31E9B580A2@chaos.jnpr.net> <53D85495.4050408@mu.org> <20140730053446.DCE8D580A2@chaos.jnpr.net> <53D944F5.7000207@freebsd.org> <20140730194522.B58FE580A2@chaos.jnpr.net> In-Reply-To: <20140730194522.B58FE580A2@chaos.jnpr.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 21:28:30 -0000 On 7/30/14 12:45 PM, Simon J. Gerraty wrote: > On Wed, 30 Jul 2014 12:18:13 -0700, Alfred Perlstein writes: >> The goal of a GSOC project is to get the code into FreeBSD. >> >> The code can be seen here: >> https://socsvn.freebsd.org/socsvn/soc2014/zkorchev/ >> >> Since Juniper has many years of experience AND the GSOC project has >> many, many utils converted I'm suggesting that Juniper engage in the >> review process and help us get the best of both worlds in. > That would of course depend on whether we like what has been done ;-) > The library and API are more important than how many apps have been > converted. > > Will take a look. > Cool, here is a list of what has been done already, it's quite a bit. ~ % svn diff -r268704 https://socsvn.freebsd.org/socsvn/soc2014/zkorchev/ | grep ^Index Index: freebsd_head/usr.bin/du/du.c Index: freebsd_head/usr.bin/du/Makefile Index: freebsd_head/usr.bin/netstat/flowtable.c Index: freebsd_head/usr.bin/netstat/unix.c Index: freebsd_head/usr.bin/netstat/main.c Index: freebsd_head/usr.bin/netstat/inet6.c Index: freebsd_head/usr.bin/netstat/netstat.h Index: freebsd_head/usr.bin/netstat/mbuf.c Index: freebsd_head/usr.bin/netstat/Makefile Index: freebsd_head/usr.bin/netstat/route.c Index: freebsd_head/usr.bin/netstat/if.c Index: freebsd_head/usr.bin/netstat/inet.c Index: freebsd_head/usr.bin/netstat/mroute6.c Index: freebsd_head/usr.bin/netstat/netisr.c Index: freebsd_head/usr.bin/netstat/bpf.c Index: freebsd_head/usr.bin/netstat/mroute.c Index: freebsd_head/usr.bin/wc/wc.c Index: freebsd_head/usr.bin/wc/Makefile Index: freebsd_head/usr.bin/last/last.c Index: freebsd_head/usr.bin/last/Makefile Index: freebsd_head/usr.bin/sockstat/sockstat.c Index: freebsd_head/usr.bin/sockstat/Makefile Index: freebsd_head/usr.bin/w/w.c Index: freebsd_head/usr.bin/w/extern.h Index: freebsd_head/usr.bin/w/Makefile Index: freebsd_head/usr.bin/w/pr_time.c Index: freebsd_head/usr.bin/finger/finger.c Index: freebsd_head/usr.bin/finger/Makefile Index: freebsd_head/usr.bin/finger/finger.h Index: freebsd_head/usr.bin/finger/sprint.c Index: freebsd_head/usr.bin/fstat/Makefile Index: freebsd_head/usr.bin/fstat/fstat.c Index: freebsd_head/usr.bin/procstat/procstat_kstack.c Index: freebsd_head/usr.bin/procstat/procstat.c Index: freebsd_head/usr.bin/procstat/procstat_basic.c Index: freebsd_head/usr.bin/procstat/procstat_vm.c Index: freebsd_head/usr.bin/procstat/procstat_auxv.c Index: freebsd_head/usr.bin/procstat/procstat.h Index: freebsd_head/usr.bin/procstat/procstat_rusage.c Index: freebsd_head/usr.bin/procstat/procstat_threads.c Index: freebsd_head/usr.bin/procstat/procstat_args.c Index: freebsd_head/usr.bin/procstat/procstat_rlimit.c Index: freebsd_head/usr.bin/procstat/procstat_files.c Index: freebsd_head/usr.bin/procstat/procstat_sigs.c Index: freebsd_head/usr.bin/procstat/procstat_bin.c Index: freebsd_head/usr.bin/procstat/procstat_cred.c Index: freebsd_head/usr.bin/procstat/Makefile Index: freebsd_head/usr.bin/vmstat/Makefile Index: freebsd_head/usr.bin/vmstat/vmstat.c Index: freebsd_head/sbin/sysctl/sysctl.c Index: freebsd_head/sbin/sysctl/Makefile Index: freebsd_head/sbin/ifconfig/ifconfig.h Index: freebsd_head/sbin/ifconfig/af_inet6.c Index: freebsd_head/sbin/ifconfig/Makefile Index: freebsd_head/sbin/ifconfig/af_nd6.c Index: freebsd_head/sbin/ifconfig/ifmedia.c Index: freebsd_head/sbin/ifconfig/af_link.c Index: freebsd_head/sbin/ifconfig/af_inet.c Index: freebsd_head/sbin/ifconfig/ifmac.c Index: freebsd_head/sbin/ifconfig/ifconfig.c Index: freebsd_head/sbin/ifconfig/carp.c Index: freebsd_head/usr.sbin/iostat/Makefile Index: freebsd_head/usr.sbin/iostat/iostat.c Index: freebsd_head/lib/libsol/sol.c Index: freebsd_head/lib/libsol/sol.h Index: freebsd_head/lib/libsol/Makefile Index: freebsd_head/bin/ls/ls.h Index: freebsd_head/bin/ls/extern.h Index: freebsd_head/bin/ls/Makefile Index: freebsd_head/bin/ls/print.c Index: freebsd_head/bin/ls/util.c Index: freebsd_head/bin/ls/ls.c ~ % From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 21:53:55 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D72E6FFD; Wed, 30 Jul 2014 21:53:55 +0000 (UTC) Received: from mail-we0-x229.google.com (mail-we0-x229.google.com [IPv6:2a00:1450:400c:c03::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4616129D2; Wed, 30 Jul 2014 21:53:55 +0000 (UTC) Received: by mail-we0-f169.google.com with SMTP id u56so1859219wes.0 for ; Wed, 30 Jul 2014 14:53:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:mail-followup-to :references:mime-version:content-type:content-disposition :content-transfer-encoding:in-reply-to:user-agent; bh=ciE7qob5TNnlbQibL++oS3fXsvjqBA9Rblz9skFYbdk=; b=EeY6OsSp+kGWTmI9Ei6C3LZVqFQjrPbRa1Dkjwq3yemkUYXFfq5VzI7scbZGswyhCF UeQt4UtzIuFKHKsh8WFIn44FCNzryPz3+vXsf1//jmTjpcMK89YfxEKjTcbD45/FJjpG hvE8JStvH6omqQgrhEj6f7QRPgndOCkJoSQ9bUPBEMo+jDzm20/PyFX0uTNMISMr5aCh iWiIT9bkRfPrMiRqvUlPMNrBY8L9S1BdCm+//KQi2ncTcqke6/ReKiVwr0fQWNUwqHYJ SZEA+aZVjZwDfOxoEwhvngGAs9L5M+DGk0WgIAaTlCdNDqJ2p2uFqoot70HwVkiAFiVG rx/Q== X-Received: by 10.194.22.166 with SMTP id e6mr10518018wjf.88.1406757233526; Wed, 30 Jul 2014 14:53:53 -0700 (PDT) Received: from brick.home (acd38.neoplus.adsl.tpnet.pl. [83.25.55.38]) by mx.google.com with ESMTPSA id ev18sm602005wid.1.2014.07.30.14.53.52 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 30 Jul 2014 14:53:53 -0700 (PDT) Sender: =?UTF-8?Q?Edward_Tomasz_Napiera=C5=82a?= Date: Wed, 30 Jul 2014 23:53:51 +0200 From: Edward Tomasz =?utf-8?Q?Napiera=C5=82a?= To: Kevin Oberman Subject: Re: [CFT] Autofs. Message-ID: <20140730215351.GC11391@brick.home> Mail-Followup-To: Kevin Oberman , FreeBSD Current , "freebsd-arch@freebsd.org" References: <20140730071933.GA20122@pc5.home> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Cc: FreeBSD Current , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 21:53:55 -0000 On 0730T1153, Kevin Oberman wrote: > On Wed, Jul 30, 2014 at 12:19 AM, Edward Tomasz Napierała > wrote: > > > At the link below you will find a patch that adds the new automounter. > > The patch is against yesterdays 11.0-CURRENT. > > > > http://people.freebsd.org/~trasz/autofs-head-20140729.diff > > > > Slides that explain the project scope and deliverables are here: > > > > http://people.freebsd.org/~trasz/autofs.pdf > > > > Testing is welcome. Please start with manual pages, eg. automount(8). > > Note that you need not only to rebuild both kernel and world, but also > > to run mergemaster, to install required /etc files. To run at startup, > > add 'autofs_enable="YES"' to /etc/rc.conf. > > > > This project is being sponsored by FreeBSD Foundation. > > > > This may be a silly question, but can autofs properly handle fusefs file > systems like ntfs-3g and exfat? Yes, it's pretty much filesystem-agnostic, except for the -hosts map, which is NFS-specific by definition. From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 23:24:59 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 04CF4981; Wed, 30 Jul 2014 23:24:59 +0000 (UTC) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2lp0235.outbound.protection.outlook.com [207.46.163.235]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 723DB24B8; Wed, 30 Jul 2014 23:24:56 +0000 (UTC) Received: from BY2PR05CA062.namprd05.prod.outlook.com (10.141.250.52) by CO2PR05MB732.namprd05.prod.outlook.com (10.141.228.22) with Microsoft SMTP Server (TLS) id 15.0.995.14; Wed, 30 Jul 2014 23:24:55 +0000 Received: from BY2FFO11FD024.protection.gbl (2a01:111:f400:7c0c::162) by BY2PR05CA062.outlook.office365.com (2a01:111:e400:2c5f::52) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Wed, 30 Jul 2014 23:24:54 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BY2FFO11FD024.mail.protection.outlook.com (10.1.15.213) with Microsoft SMTP Server (TLS) id 15.0.990.10 via Frontend Transport; Wed, 30 Jul 2014 23:24:54 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Wed, 30 Jul 2014 16:24:53 -0700 Received: from idle.juniper.net (idleski.juniper.net [172.25.4.26]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6UNOpn65386; Wed, 30 Jul 2014 16:24:52 -0700 (PDT) (envelope-from phil@juniper.net) Received: from idle.juniper.net (localhost [127.0.0.1]) by idle.juniper.net (8.14.4/8.14.3) with ESMTP id s6UNOB2H087915; Wed, 30 Jul 2014 19:24:30 -0400 (EDT) (envelope-from phil@idle.juniper.net) Message-ID: <201407302324.s6UNOB2H087915@idle.juniper.net> To: John-Mark Gurney Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <20140730193819.GM43962@funkthat.com> Date: Wed, 30 Jul 2014 19:24:11 -0400 From: Phil Shafer MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(189002)(164054003)(199002)(106466001)(76506005)(81156004)(76482001)(110136001)(105596002)(77982001)(46102001)(95666004)(6806004)(85306003)(50466002)(50986999)(79102001)(4396001)(81542001)(83322001)(68736004)(81342001)(53416004)(69596002)(31966008)(86362001)(47776003)(48376002)(92726001)(44976005)(99396002)(84676001)(54356999)(97736001)(20776003)(87936001)(103666002)(102836001)(107046002)(83072002)(80022001)(74662001)(21056001)(85852003)(64706001)(92566001)(74502001); DIR:OUT; SFP:; SCL:1; SRVR:CO2PR05MB732; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0288CD37D9 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=phil@juniper.net; X-OriginatorOrg: juniper.net Cc: sjg@freebsd.org, arch@freebsd.org, marcel@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 23:24:59 -0000 John-Mark Gurney writes: >My vote would be to use and *enforce* UTF-8 by the API. That means if >someone passes a string in, it must be properly formed UTF-8... I can certainly see making this an option, detecting the high-bit and inspecting the following 1-5 bytes to ensure the corresponding high two bits are set appropriately. But what action would you expect the library to take when invalid strings are passed in? libxo supports a warning flag, that will trigger warnings on stderr for things like invalid or malformed format strings, but I'm not sure I'd be happy if the library skipped invalid strings. BTW, this issue is driven by "w"s use of wide characters (for days of the week). Thanks, Phil From owner-freebsd-arch@FreeBSD.ORG Wed Jul 30 23:51:49 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4A4B0D1E; Wed, 30 Jul 2014 23:51:49 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1lp0145.outbound.protection.outlook.com [207.46.163.145]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E04F026CF; Wed, 30 Jul 2014 23:51:48 +0000 (UTC) Received: from BY2PR05CA054.namprd05.prod.outlook.com (10.141.250.44) by DM2PR05MB733.namprd05.prod.outlook.com (10.141.178.18) with Microsoft SMTP Server (TLS) id 15.0.995.14; Wed, 30 Jul 2014 23:51:46 +0000 Received: from BY2FFO11FD003.protection.gbl (2a01:111:f400:7c0c::190) by BY2PR05CA054.outlook.office365.com (2a01:111:e400:2c5f::44) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Wed, 30 Jul 2014 23:51:46 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BY2FFO11FD003.mail.protection.outlook.com (10.1.14.125) with Microsoft SMTP Server (TLS) id 15.0.990.10 via Frontend Transport; Wed, 30 Jul 2014 23:51:46 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Wed, 30 Jul 2014 16:51:24 -0700 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6UNpNn82364; Wed, 30 Jul 2014 16:51:24 -0700 (PDT) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id AC944580A2; Wed, 30 Jul 2014 16:51:23 -0700 (PDT) To: Alfred Perlstein , Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <20140730194522.B58FE580A2@chaos.jnpr.net> References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> <20140729230345.31E9B580A2@chaos.jnpr.net> <53D85495.4050408@mu.org> <20140730053446.DCE8D580A2@chaos.jnpr.net> <53D944F5.7000207@freebsd.org> <20140730194522.B58FE580A2@chaos.jnpr.net> Comments: In-reply-to: "Simon J. Gerraty" message dated "Wed, 30 Jul 2014 12:45:22 -0700." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Wed, 30 Jul 2014 16:51:23 -0700 Message-ID: <20140730235123.AC944580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(199002)(189002)(51704005)(87286001)(87936001)(88136002)(15975445006)(47776003)(89996001)(81156004)(106466001)(31966008)(19580395003)(76482001)(83322001)(46102001)(92726001)(84676001)(20776003)(6806004)(44976005)(74662001)(74502001)(86362001)(92566001)(85306003)(95666004)(70486001)(93886003)(105596002)(97736001)(104166001)(102836001)(93546004)(107046002)(80022001)(77156001)(69596002)(76176999)(21056001)(50226001)(4396001)(77982001)(101356003)(102176002)(57986006)(99396002)(90896003)(33656002)(85852003)(83072002)(48376002)(81342001)(50986999)(50466002)(64706001)(81542001)(68736004)(79102001)(62966002)(76506005)(93916002)(42262001); DIR:OUT; SFP:; SCL:1; SRVR:DM2PR05MB733; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0288CD37D9 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=sjg@juniper.net; X-OriginatorOrg: juniper.net Cc: phil@juniper.net X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2014 23:51:49 -0000 On Wed, 30 Jul 2014 12:45:22 -0700, "Simon J. Gerraty" writes: >>The code can be seen here: >> https://socsvn.freebsd.org/socsvn/soc2014/zkorchev/ >> >>Since Juniper has many years of experience AND the GSOC project has >>many, many utils converted I'm suggesting that Juniper engage in the >>review process and help us get the best of both worlds in. > >That would of course depend on whether we like what has been done ;-) >The library and API are more important than how many apps have been >converted. Ok I took a look... There's no nice way to say it I'm afraid; This looks like the same sort of approach that Juniper used a over a decade ago for the BSD apps to get XML out, and which we felt was just too ugly to upstream, I don't see how we could support this. Unfortunately it took me a couple of years to get a few hours of Phil's time, to come up with a neat API to avoid that #ifdef and if/else mess. On the plus side that means it covers a lot more than just XML. We now have an API and library that can avoid the need to double the cost of adding new output to apps, ie you only write to libxo's api and it does plain TXT for you if needed. Our only concern was how best to address the wide-char issue John-Mark Gurney strongly favours just doing UTF-8, I and I believe Marcel and Phil would tend to agree (since AFAICT it has little overhead for ASCII) Based on this thread, it would seem there is still demand for this functionality so I think we should proceed with putting libxo into our internal tree and generate some diffs for folk to review... Thanks --sjg From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 00:09:42 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 971DD21C; Thu, 31 Jul 2014 00:09:42 +0000 (UTC) Received: from mail.xcllnt.net (mail.xcllnt.net [50.0.150.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 674C527F8; Thu, 31 Jul 2014 00:09:42 +0000 (UTC) Received: from jhayes-sslvpn-nc.jnpr.net ([66.129.239.11]) (authenticated bits=0) by mail.xcllnt.net (8.14.9/8.14.9) with ESMTP id s6V09cZf094559 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Wed, 30 Jul 2014 17:09:40 -0700 (PDT) (envelope-from marcel@xcllnt.net) Content-Type: multipart/signed; boundary="Apple-Mail=_5CA6EAF9-A833-4B1F-8BE4-0BA79EFE4BB5"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Marcel Moolenaar In-Reply-To: <20140730235123.AC944580A2@chaos.jnpr.net> Date: Wed, 30 Jul 2014 17:09:33 -0700 Message-Id: References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> <20140729230345.31E9B580A2@chaos.jnpr.net> <53D85495.4050408@mu.org> <20140730053446.DCE8D580A2@chaos.jnpr.net> <53D944F5.7000207@freebsd.org> <20140730194522.B58FE580A2@chaos.jnpr.net> <20140730235123.AC944580A2@chaos.jnpr.net> To: "Simon J. Gerraty" X-Mailer: Apple Mail (2.1878.6) Cc: Alfred Perlstein , phil@juniper.net, freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 00:09:42 -0000 --Apple-Mail=_5CA6EAF9-A833-4B1F-8BE4-0BA79EFE4BB5 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii On Jul 30, 2014, at 4:51 PM, Simon J. Gerraty wrote: > > Our only concern was how best to address the wide-char issue John-Mark > Gurney strongly favours just doing UTF-8, I and I believe Marcel and > Phil would tend to agree (since AFAICT it has little overhead for ASCII) Yes. I would even be ok with only having wide-character support. I do tend to agree with Phil that the standard C functions that work on wide characters are more of an eye sore than the "regular" C functions. -- Marcel Moolenaar marcel@xcllnt.net --Apple-Mail=_5CA6EAF9-A833-4B1F-8BE4-0BA79EFE4BB5 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlPZiT0ACgkQpgWlLWHuifa8fgCbBXztn9mOGrcirnXu598Zzz94 P+sAn1kGqJmSUAmZwY1QC04pPjeExyyj =kjuF -----END PGP SIGNATURE----- --Apple-Mail=_5CA6EAF9-A833-4B1F-8BE4-0BA79EFE4BB5-- From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 00:25:17 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A38DB554; Thu, 31 Jul 2014 00:25:17 +0000 (UTC) Received: from mail.xcllnt.net (mail.xcllnt.net [50.0.150.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5965A295E; Thu, 31 Jul 2014 00:25:17 +0000 (UTC) Received: from jhayes-sslvpn-nc.jnpr.net ([66.129.239.11]) (authenticated bits=0) by mail.xcllnt.net (8.14.9/8.14.9) with ESMTP id s6V0PERw094606 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Wed, 30 Jul 2014 17:25:15 -0700 (PDT) (envelope-from marcel@xcllnt.net) Content-Type: multipart/signed; boundary="Apple-Mail=_48265AB6-342B-4B6A-A5BE-772C31DFAA79"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Marcel Moolenaar In-Reply-To: <20140730235123.AC944580A2@chaos.jnpr.net> Date: Wed, 30 Jul 2014 17:25:09 -0700 Message-Id: <06AE7354-CF5D-409F-BCAB-7CD4B728DF15@xcllnt.net> References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> <20140729230345.31E9B580A2@chaos.jnpr.net> <53D85495.4050408@mu.org> <20140730053446.DCE8D580A2@chaos.jnpr.net> <53D944F5.7000207@freebsd.org> <20140730194522.B58FE580A2@chaos.jnpr.net> <20140730235123.AC944580A2@chaos.jnpr.net> To: "Simon J. Gerraty" X-Mailer: Apple Mail (2.1878.6) Cc: Alfred Perlstein , phil@juniper.net, freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 00:25:17 -0000 --Apple-Mail=_48265AB6-342B-4B6A-A5BE-772C31DFAA79 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii On Jul 30, 2014, at 4:51 PM, Simon J. Gerraty wrote: > This looks like the same sort of approach that Juniper used a > over a decade ago for the BSD apps to get XML out, and which we felt was > just too ugly to upstream, I don't see how we could support this. > > Unfortunately it took me a couple of years to get a few hours of Phil's > time, to come up with a neat API to avoid that #ifdef and if/else mess. > On the plus side that means it covers a lot more than just XML. > > We now have an API and library that can avoid the need to double the > cost of adding new output to apps, ie you only write to libxo's api > and it does plain TXT for you if needed. Something that any implementation should handle is that warn(3) and err(3) functions should do the right thing. More precisely: I would expect that the output of a utility when emitting some ML handles both stdout and stderr consistently. Also: I would hope that if we emit XML (for example), we can do so according to a well-defined schema. The GSoC approach seems less likely to have a good solution for those than libxo has. -- Marcel Moolenaar marcel@xcllnt.net --Apple-Mail=_48265AB6-342B-4B6A-A5BE-772C31DFAA79 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlPZjOUACgkQpgWlLWHuifaXRwCeLbL9tbaOdsi++lSRyt1iWTve dHsAoIOco8fBpBMa6TfW8OxC5OHqyG4N =2IhH -----END PGP SIGNATURE----- --Apple-Mail=_48265AB6-342B-4B6A-A5BE-772C31DFAA79-- From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 00:50:42 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 31D60853; Thu, 31 Jul 2014 00:50:42 +0000 (UTC) Received: from smtp10.server.rpi.edu (gateway.canit.rpi.edu [128.113.2.230]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E3C872BAA; Thu, 31 Jul 2014 00:50:41 +0000 (UTC) Received: from smtp-auth1.server.rpi.edu (route.canit.rpi.edu [128.113.2.231]) by smtp10.server.rpi.edu (8.14.3/8.14.3/Debian-9.4) with ESMTP id s6V0odV8010331; Wed, 30 Jul 2014 20:50:39 -0400 Received: from smtp-auth1.server.rpi.edu (localhost [127.0.0.1]) by smtp-auth1.server.rpi.edu (Postfix) with ESMTP id 587DF5820D; Wed, 30 Jul 2014 20:50:39 -0400 (EDT) Received: from [128.113.24.47] (gilead-qc124.netel.rpi.edu [128.113.124.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: drosih) by smtp-auth1.server.rpi.edu (Postfix) with ESMTPSA id 4384A5800C; Wed, 30 Jul 2014 20:50:38 -0400 (EDT) From: "Garance A Drosehn" To: "Alfred Perlstein" Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Date: Wed, 30 Jul 2014 20:50:37 -0400 Message-ID: <07923DF8-48CC-4A05-9260-529C51922DA7@rpi.edu> In-Reply-To: <53D944F5.7000207@freebsd.org> References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140728054217.AC1A0580A2@chaos.jnpr.net> <20140728055336.GJ50802@ivaldir.etoilebsd.net> <20140729230345.31E9B580A2@chaos.jnpr.net> <53D85495.4050408@mu.org> <20140730053446.DCE8D580A2@chaos.jnpr.net> <53D944F5.7000207@freebsd.org> MIME-Version: 1.0 X-Mailer: MailMate (1.7.2r3905) X-Virus-Scanned: ClamAV using ClamSMTP X-Bayes-Prob: 0.0001 (Score 0, tokens from: outgoing, @@RPTN) X-Spam-Score: 0.00 () [Hold at 15.10] X-CanIt-Incident-Id: 03MwMODFM X-CanIt-Geo: ip=128.113.124.17; country=US; region=New York; city=Troy; latitude=42.7495; longitude=-73.5951; http://maps.google.com/maps?q=42.7495,-73.5951&z=6 X-CanItPRO-Stream: outgoing X-Canit-Stats-ID: Bayes signature not available X-Scanned-By: CanIt (www . roaringpenguin . com) on 128.113.2.230 Cc: freebsd-arch@freebsd.org, phil@juniper.net, "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 00:50:42 -0000 On 30 Jul 2014, at 15:18, Alfred Perlstein wrote: > The goal of a GSOC project is to get the code into FreeBSD. > > The code can be seen here: > https://socsvn.freebsd.org/socsvn/soc2014/zkorchev/ > [...skip...] > The details for the code are here: > https://socsvn.freebsd.org/socsvn/soc2014/zkorchev/ > > You should be able to do an svn checkout and then get diffs > to see what is going on. If you require any assistance please > let me know. Those two URL's look extremely similar to me. Was the second one supposed to point to some other page? I haven't taken the time to check out the tree and skim through all the changes, but I looked at a few specific files. In .../lib/libsol/sol.c, it looks like the only format implemented so far is JSON. Is that true? Some dumb questions I should probably be able to figure out for myself: Where is SOL_JSON defined? sol.c includes sol.h, but sol.h does not seem to define that value. Also, sol.h includes yajl/yajl_gen.h, but I don't see where that file comes from. I looked at .../usr.bin/du/du.c just to see a simple example. I notice the '#if defined(SOL_ON)'. I assume that's just meant for the initial debugging, so that one could turn off all the SOL support if it was suspected of causing some problem. Is that expected to stay in the code once the code goes into production? If so, I'd rather see it as 'if (SOL_ON) { ... }', so that it's proper C code which the compiler would optimize away if SOL_ON was defined as FALSE. As a general rule I prefer to have the compiler *always* compiling&checking that code, even if some of it ends up producing no object code because SOL_ON is false. -- Garance Alistair Drosehn = drosih@rpi.edu Senior Systems Programmer or gad@FreeBSD.org Rensselaer Polytechnic Institute; Troy, NY; USA From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 01:37:58 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 43931DC6; Thu, 31 Jul 2014 01:37:58 +0000 (UTC) Received: from smtp10.server.rpi.edu (gateway.canit.rpi.edu [128.113.2.230]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 005C5208F; Thu, 31 Jul 2014 01:37:57 +0000 (UTC) Received: from smtp-auth2.server.rpi.edu (smtp-auth2.server.rpi.edu [128.113.2.232]) by smtp10.server.rpi.edu (8.14.3/8.14.3/Debian-9.4) with ESMTP id s6V1btZl018390; Wed, 30 Jul 2014 21:37:55 -0400 Received: from smtp-auth2.server.rpi.edu (localhost [127.0.0.1]) by smtp-auth2.server.rpi.edu (Postfix) with ESMTP id 99F0F1825C; Wed, 30 Jul 2014 21:37:55 -0400 (EDT) Received: from [128.113.24.47] (gilead-qc124.netel.rpi.edu [128.113.124.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: drosih) by smtp-auth2.server.rpi.edu (Postfix) with ESMTPSA id 8832918009; Wed, 30 Jul 2014 21:37:55 -0400 (EDT) From: "Garance A Drosehn" To: "Simon Gerraty" Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Date: Wed, 30 Jul 2014 21:37:54 -0400 Message-ID: <82CFA67F-BA93-44EE-BD4B-9105F89AD157@rpi.edu> In-Reply-To: <20140725044921.9F0D3580A2@chaos.jnpr.net> References: <20140725044921.9F0D3580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain; format=flowed X-Mailer: MailMate (1.7.2r3905) X-Virus-Scanned: ClamAV using ClamSMTP X-Bayes-Prob: 0.0001 (Score 0, tokens from: outgoing, @@RPTN) X-Spam-Score: 0.00 () [Hold at 15.10] X-CanIt-Incident-Id: 03MwNBTaX X-CanIt-Geo: ip=128.113.124.17; country=US; region=New York; city=Troy; latitude=42.7495; longitude=-73.5951; http://maps.google.com/maps?q=42.7495,-73.5951&z=6 X-CanItPRO-Stream: outgoing X-Canit-Stats-ID: Bayes signature not available X-Scanned-By: CanIt (www . roaringpenguin . com) on 128.113.2.230 Cc: sjg@freebsd.org, arch@freebsd.org, marcel@freebsd.org, phil@juniper.net X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 01:37:58 -0000 On 25 Jul 2014, at 0:49, Simon Gerraty wrote: > > The following from Phil provides some idea of the functionality > available and the API. > The one shown here uses the default output handle (stdout), but there > are variants that allow multiple output handles. > >> Here's some sample output (from my libxo-ified "w"): >> >> (This uses an environment variable to trigger the mode and options: >> T X J and H are the modes (text, xml, json, html); [...etc...] >> >> % foreach i ( T XP JP HP HPIx ) >> echo === $i === >> env LIBXO_OPTIONS=$i ./xtest -n | head -10 >> end >> === T === >> 6:47PM up 18 days, 2:01, 9 user%s, load averages: 0.00, 0.00, 0.00 >> USER TTY FROM LOGIN@ IDLE WHAT >> phil pts/0 76.182.32.73 5:09PM 33 /bin/sh >> phil pts/1 76.182.32.73 05Jul14 2 >> /usr/bin/perl /u/phil/bin/plum ( >> phil pts/2 76.182.32.73 05Jul14 1 /bin/tcsh >> phil pts/3 76.182.32.73 05Jul14 2days ssh dent >> phil pts/4 76.182.32.73 Tue02PM 2days ssh >> svl-junos-d026.juniper.net >> phil pts/5 76.182.32.73 Wed01AM 2days telnet >> man-o-war 2006 >> phil pts/6 76.182.32.73 Fri10PM 2days ssh >> 198.85.229.65 >> phil pts/7 76.182.32.73 Fri10PM 2days ssh zap >> === XP === >> >> 6:47PM >> 18 days >> 2:01 >> 9 >> 0.00 >> 0.00 >> 0.00 >> >> >> === JP === Do you have links to the library itself? Over the years I've dabbled with doing something like this for the lpr/lpc/lpq programs, so I've done a fair amount of thinking about it. I didn't do as much *work* as either libxo or the GSOC project, but I have done some thinking! Mind you, even that thinking is based only on the lpr-programs, and not a larger set of utilities. I'd suggest that the above isn't quite what one would want, either. For the text version it's fine to have a time-of-day value as '6:47PM', but if you're going for machine-readable output then you'd want that in some format which was much more specific and *standard* (as opposed to arbitrary pretty-printed strings). Something like the ISO 8601 format used in obscure parts of lpd: #define LPD_TIMESTAMP_PATTERN "%Y-%m-%dT%T%z %a" (actually the %a part is not part of ISO 8601, but it is useful for some programs which might want to process the time). Or you might want to print it as a unix-epoch integer. Something that makes it easy for a program to process it. Or you could follow the example of EDN, and use rfc-3339-format (see '#inst' at https://github.com/edn-format/edn). The nice thing about standards is that there are so many to choose from. It's hard to tell based on your sample output, but there's also the question of truncating strings. In the text output of your example, it obviously makes sense to truncate the 'WHAT' value to 'ssh svl-junos-d026.juniper.net', but when printing the same output in a machine-readable format you wouldn't want to truncate it. Let the application which *reads* the data decide how many characters *it* wants to use. -- Garance Alistair Drosehn = drosih@rpi.edu Senior Systems Programmer or gad@FreeBSD.org Rensselaer Polytechnic Institute; Troy, NY; USA From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 01:47:45 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BF306FA2; Thu, 31 Jul 2014 01:47:45 +0000 (UTC) Received: from smtp9.server.rpi.edu (gateway.canit.rpi.edu [128.113.2.229]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 62D08214E; Thu, 31 Jul 2014 01:47:44 +0000 (UTC) Received: from smtp-auth2.server.rpi.edu (route.canit.rpi.edu [128.113.2.232]) by smtp9.server.rpi.edu (8.14.3/8.14.3/Debian-9.4) with ESMTP id s6V1jG98015323; Wed, 30 Jul 2014 21:45:16 -0400 Received: from smtp-auth2.server.rpi.edu (localhost [127.0.0.1]) by smtp-auth2.server.rpi.edu (Postfix) with ESMTP id 7DAA318079; Wed, 30 Jul 2014 21:45:16 -0400 (EDT) Received: from [128.113.24.47] (gilead-qc124.netel.rpi.edu [128.113.124.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: drosih) by smtp-auth2.server.rpi.edu (Postfix) with ESMTPSA id 6CBD218009; Wed, 30 Jul 2014 21:45:16 -0400 (EDT) From: "Garance A Drosehn" To: "Simon Gerraty" Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Date: Wed, 30 Jul 2014 21:45:15 -0400 Message-ID: In-Reply-To: <82CFA67F-BA93-44EE-BD4B-9105F89AD157@rpi.edu> References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <82CFA67F-BA93-44EE-BD4B-9105F89AD157@rpi.edu> MIME-Version: 1.0 X-Mailer: MailMate (1.7.2r3905) X-Virus-Scanned: ClamAV using ClamSMTP X-Bayes-Prob: 0.0001 (Score 0, tokens from: outgoing, @@RPTN) X-Spam-Score: 0.00 () [Hold at 15.10] X-CanIt-Incident-Id: 02MwNJgfn X-CanIt-Geo: ip=128.113.124.17; country=US; region=New York; city=Troy; latitude=42.7495; longitude=-73.5951; http://maps.google.com/maps?q=42.7495,-73.5951&z=6 X-CanItPRO-Stream: outgoing X-Canit-Stats-ID: Bayes signature not available X-Scanned-By: CanIt (www . roaringpenguin . com) on 128.113.2.229 Cc: sjg@freebsd.org, arch@freebsd.org, marcel@freebsd.org, phil@juniper.net X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 01:47:45 -0000 On 30 Jul 2014, at 21:37, Garance A Drosehn wrote: > > [...] if you're going for machine-readable output then > you'd want that in some format which was much more specific and > *standard* (as opposed to arbitrary pretty-printed strings). > Something like the ISO 8601 format used in obscure parts of lpd: > > #define LPD_TIMESTAMP_PATTERN "%Y-%m-%dT%T%z %a" > Or you could follow the example of EDN, and use rfc-3339-format > (see '#inst' at https://github.com/edn-format/edn). The nice > thing about standards is that there are so many to choose from. I should note that these two formats are very similar, and in fact may be exactly the same. I kept meaning to see if there was any difference between them. I just noticed that the rfc has the 'Z' suffix as an option for a timezone, and I don't think that the ISO one does. -- Garance Alistair Drosehn = drosih@rpi.edu Senior Systems Programmer or gad@FreeBSD.org Rensselaer Polytechnic Institute; Troy, NY; USA From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 01:51:23 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4863EF8; Thu, 31 Jul 2014 01:51:23 +0000 (UTC) Received: from mail-la0-x229.google.com (mail-la0-x229.google.com [IPv6:2a00:1450:4010:c03::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 996F321D4; Thu, 31 Jul 2014 01:51:22 +0000 (UTC) Received: by mail-la0-f41.google.com with SMTP id s18so1557308lam.28 for ; Wed, 30 Jul 2014 18:51:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=htDwpTKLF/mkQORBz7du2bWdDfzEjX/9MMLcyZygFgc=; b=awo+grd9mW4qecLgn9zMaGstY4TCRnF+BZOuLKnRJ5eh8RGOuw48+yWIF6ryvoVH5D qLF/LhOqkzCtdAZg458eh8B5eNbBoWJxfb09ExDLFqtl0ZGVydtP9KrlwbcwJhQ9ZnGE eEs+aARF8ytCnNoun5uDJZt148kNWHGJW5T6I8RAzCGG+bnsfjPgXriKlMF6Kofz47o4 WYZeTOzClRguUhQFgsf/q5bfamhq9h3rjswYP6J0ziOsRoWXe36A/xcS9lUm8gKlB996 hnfuUkjxXBBOjNvMvyVonHFgw/heQyh7Hh01TzbloEedvb0seO7uzTK8YIxuXEOaeXZ/ uz3A== MIME-Version: 1.0 X-Received: by 10.112.60.5 with SMTP id d5mr8259683lbr.26.1406771479615; Wed, 30 Jul 2014 18:51:19 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.112.225.34 with HTTP; Wed, 30 Jul 2014 18:51:19 -0700 (PDT) In-Reply-To: <20140730071933.GA20122@pc5.home> References: <20140730071933.GA20122@pc5.home> Date: Wed, 30 Jul 2014 18:51:19 -0700 X-Google-Sender-Auth: 1kF4cWStzhp-eEg1DwP4hcTkjKQ Message-ID: Subject: Re: [CFT] Autofs. From: Craig Rodrigues To: freebsd-current Current , freebsd-arch Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 01:51:23 -0000 On Wed, Jul 30, 2014 at 12:19 AM, Edward Tomasz Napiera=C5=82a wrote: > At the link below you will find a patch that adds the new automounter. > The patch is against yesterdays 11.0-CURRENT. > > http://people.freebsd.org/~trasz/autofs-head-20140729.diff > > Hi, Nice stuff! I updated a current system, and tried the patch. I put autofs_enable=3D"YES" in /etc/rc.conf In my network, I have a system, testhost with some NFS exports. /dir1/export1 /dir1/export2 /dir1/export3 I did: cd /net/testhost This operation seemed to timeout. In dmesg, I saw these errors: WARNING: autofs_callout: request 0 for /net/ timed out after 30 seconds WARNING: autofs_trigger_one: request for /net/ completed with error 60 autofs_trigger: trigger failed with error 60; will retry in 1 seconds, 2 attempts left WARNING: autofs_callout: request 1 for /net/ timed out after 30 seconds WARNING: autofs_trigger_one: request for /net/ completed with error 60 autofs_trigger: trigger failed with error 60; will retry in 1 seconds, 1 attempts left WARNING: autofs_callout: request 2 for /net/ timed out after 30 seconds WARNING: autofs_trigger_one: request for /net/ completed with error 60 autofs_trigger: trigger failed 3 times; returning error 60 I used my Mac laptop in the same network, and did: cd /net/testhost and the command did not timeout, and I could do an ls and see that /net/testhost/dir1/ was visible. Any ideas? This is cool stuff, and I hope you can get it into the 10.1 RELEASE. -- Craig From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 02:09:45 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 067F9CDE for ; Thu, 31 Jul 2014 02:09:45 +0000 (UTC) Received: from mail-pd0-f171.google.com (mail-pd0-f171.google.com [209.85.192.171]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CC1692400 for ; Thu, 31 Jul 2014 02:09:44 +0000 (UTC) Received: by mail-pd0-f171.google.com with SMTP id z10so2505029pdj.16 for ; Wed, 30 Jul 2014 19:09:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=ilwxu2yw3dznfn2iIsCE19Zpa5Az4gQ/gL9GZUHSUUc=; b=NztCEfCF7jHA2Hjza28xD/njQiYCgyGlh3XxV1TX2/3tjGhzn4o3eay7yXUn3RGYBL vBzL1gQkoJdi+Z5skMExJ1E97Oez5sl5c5hRpb+Ezvsh+PVji4VxAKWnTd/d58V6QbUx RX6E1XfPXF8D0QTYzyDJqMX2JLjxnubTyg+mz2vCIKt0rcz6jZQyhD5bPbQXc9Ux89DJ brTq6qYRLYKvV2GZRZbQykz8Tp0FlM+VbEJ3rDyUMw2AvWT9N4Nbi4+xH+2L5MAd4asr 1KpW5ZKjTx4DSBLjIiPrPMx8G6oYuJ33MO6aCT3ypvtLzQ9bxbJxj6TOCz88KMPOik2S uteQ== X-Gm-Message-State: ALoCoQmpN4WFopwgWEE5MKMnVyDpq0s7xP67W5q4WXi60gmqpcYo3piy5+7voAoHpoGcdTRnlBhA X-Received: by 10.68.93.65 with SMTP id cs1mr916227pbb.52.1406772583483; Wed, 30 Jul 2014 19:09:43 -0700 (PDT) Received: from [192.168.1.100] (c-24-6-220-224.hsd1.ca.comcast.net. [24.6.220.224]) by mx.google.com with ESMTPSA id vz7sm150285pbc.95.2014.07.30.19.09.41 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 30 Jul 2014 19:09:42 -0700 (PDT) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Tim Kientzle In-Reply-To: Date: Wed, 30 Jul 2014 19:09:40 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <82CFA67F-BA93-44EE-BD4B-9105F89AD157@rpi.edu> To: Garance A Drosehn X-Mailer: Apple Mail (2.1878.6) Cc: sjg@freebsd.org, arch@freebsd.org, marcel@freebsd.org, phil@juniper.net, Simon Gerraty X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 02:09:45 -0000 On Jul 30, 2014, at 6:45 PM, Garance A Drosehn wrote: > On 30 Jul 2014, at 21:37, Garance A Drosehn wrote: >>=20 >> [...] if you're going for machine-readable output then >> you'd want that in some format which was much more specific and >> *standard* (as opposed to arbitrary pretty-printed strings). >> Something like the ISO 8601 format used in obscure parts of lpd: >>=20 >> #define LPD_TIMESTAMP_PATTERN "%Y-%m-%dT%T%z %a" >=20 >> Or you could follow the example of EDN, and use rfc-3339-format >> (see '#inst' at https://github.com/edn-format/edn). The nice >> thing about standards is that there are so many to choose from. >=20 > I should note that these two formats are very similar, and in fact > may be exactly the same. Essentially, ISO8601 is the same as RFC3339 except that ISO8601 also has a bunch of additional notations for partial date/time, durations, and repeat intervals. Trivia: * RFC3339 claims to be a =93profile of ISO8601=94 * RFC3339 requires a timezone specifier * Both allow fractional seconds (period followed by one or more digits) * RFC3339 allows a timezone of =91-00:00=92; ISO8601 requires a =91+=92 = for a zero offset > I kept meaning to see if there was any > difference between them. I just noticed that the rfc has the 'Z' > suffix as an option for a timezone, and I don't think that the ISO > one does. Both allow =91Z=92.=20 = http://stackoverflow.com/questions/522251/whats-the-difference-between-iso= -8601-and-rfc-3339-date-formats Tim From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 03:00:14 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D0C815F2 for ; Thu, 31 Jul 2014 03:00:14 +0000 (UTC) Received: from nm7.bullet.mail.bf1.yahoo.com (nm7.bullet.mail.bf1.yahoo.com [98.139.212.166]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7CF1A27CA for ; Thu, 31 Jul 2014 03:00:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1406775607; bh=nK+8ewItBakKIkJSYsuxVZ/ewRmX8ScjxUFvCNFdaAc=; h=Received:Received:Received:X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:Cc:Message-Id:References:To:X-Mailer; b=dUN5UOU0DeFGjYf0PX4dnpDB7Dlsx1dlpgo0p4rA2RqUBtgEyynL6ZBA03d6LcitJcXfM0+Uo91fbbXENf1tZ7W/mo6Wb+X4FW7P4nKEflGr5tCt9n4XuMh2EHnp3fAp6h/0zkMXTTVBrSfBWFu4jdXc0uhoRDoRboNixNVA9WFelsaSzdg3NYPj32FXwnVbcznASOq+nNjGix1/vYOwsanhf7Otl3YwYe6aLReSxIIL5xjec3Qznbc92aepdZ+yuBVedFDyRv2Ka0ygIdXMnxDAmGrm8K30NuSvAq6VWKaraZih8g+NQeY4G8Rslq45G0NPQfS/DdnHXkhmLQ2rTA== DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s2048; d=yahoo.com; b=qnVJ0DZVtojq7tIUd4O7vf3okCg302ohgc7/TGxMSSfmFxis5vy//aHKrnqpjR7FYHqUFiLVNfp/mbUY4b464bolTtXegHqDIpNP4BSVHaZ0Vid+60AHfrsqqF0P34XJNGryhSl1HegMC1iJDhgFND19Is0uaUDyd4CL2G0IvvlQQini7KgUPmh9V5BCMOp1y3RhwNvNVhyw76FmVRyrp39tc4b1rvjO1Cq4jEHBANqB0zJJhT59cMZ0Y7gBtzfY5SCfexkEprmH4plZ3S5Qkiq5Sy6HZxZ99aBFzp9+sxs283P85uI1rkmoPGZDr4JezfZ5qFeJZlhF8eJDP8EeCw==; Received: from [66.196.81.173] by nm7.bullet.mail.bf1.yahoo.com with NNFMP; 31 Jul 2014 03:00:07 -0000 Received: from [98.139.211.163] by tm19.bullet.mail.bf1.yahoo.com with NNFMP; 31 Jul 2014 03:00:07 -0000 Received: from [127.0.0.1] by smtp220.mail.bf1.yahoo.com with NNFMP; 31 Jul 2014 03:00:07 -0000 X-Yahoo-Newman-Id: 761958.64427.bm@smtp220.mail.bf1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: ZLjC2LUVM1mCfnrKg_ChLBKfWJJrp0.4EFE5TLKVFHxeRmi 3JvWSL4fQujsZciSzqNIqBxYFoQywHY_Lvh6iwdSI5vo9dM8q2ifdWxMlEXy byndjEpyLoreAdXY5.vaU5bBk3OjicpD0YcYcPxr53EKGUTySHQNvr1Qq96Y L3IrAsTXkJgWdG3MDNKtbV8mXQeS_MJjSL38RT3o5_yprLTvLcNgvLch66qO YSLwRj.xHocH_fFSMBWe8cJoIPxrIPMqATXyKw9PWrPgYn3MRaLLHWB4kUo6 QU3.8g9NvSpKBVqHTyLOcXiMV6.7OYrOTc7My4MJxmaeV.gGpI0RWq8aV6vD j3ULMvDYRobMUGuavoY.eca9lOyyaVvKPXdDG2Nk02ewrRcwfDUlNk502Szg _kARwffpe7fv3GCIXqT.5pik3My3uj1QYgrv8lgNbMKe0pLo3LS.vGbpItgs HyREF10ucbSgSlOk12Mnt72XwuFRmAe1yiFns3pE9SG0Kpt5DSPjNNKw90qV yqRngVCMyK_AeUHfu.adCHOWqBn8JNoxPXlJhvBv8YE8QjgAEAFiL4JBUgSM z9PQvyN.1ct9XWmee2Fkqb8LXfg9FRRulGz1E8RNL_LQfI20._Zm0xpk1DHP toIE4W_YgbAAAZKobImWwevIUAzKMBCgFwmA2hAvrG7nmkMJYf0NveeziPbg W1XrSrmG9QGcDWHib.4LhpLqDm6sgqIsxEiVVj61alGuFvVeANw-- X-Yahoo-SMTP: xcjD0guswBAZaPPIbxpWwLcp9Unf Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: [RFC] ASLR Whitepaper and Candidate Final Patch From: Pedro Giffuni In-Reply-To: <20140724175704.GT29618@pwnie.vrt.sourcefire.com> Date: Wed, 30 Jul 2014 22:00:02 -0500 Message-Id: <112E989D-607B-47AC-8942-0FB049DA6C3D@freebsd.org> References: <96C72773-3239-427E-A90B-D05FF0F5B782@freebsd.org> <20140720201858.GB29618@pwnie.vrt.sourcefire.com> <20140723004543.GH29618@pwnie.vrt.sourcefire.com> <20140724175704.GT29618@pwnie.vrt.sourcefire.com> To: Shawn Webb X-Mailer: Apple Mail (2.1878.6) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: PaX Team , Oliver Pinter , Robert Watson , Bryan Drewery , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 03:00:14 -0000 Il giorno 24/lug/2014, alle ore 12:57, Shawn Webb ha = scritto: > On Jul 22, 2014 08:45 PM -0400, Shawn Webb wrote: >> On Jul 23, 2014 12:28 AM +0100, Robert Watson wrote: >>> On Sun, 20 Jul 2014, Shawn Webb wrote: >>>=20 >>>>> - It is yet undetermined what the performance effect will be, and = it is not=20 >>>>> clear (but seems likely from past measurements) if there will be a=20= >>>>> performance hit even when ASLR is off. -Apparently there are = applications=20 >>>>> that will segfault (?). >>>>=20 >>>> So I have an old Dell Latitude E6500 that I bought at Defcon a year = or >>>> so ago that I'm doing testing on. Even though it's quite an = underpowered >>>> laptop, I'm running ZFS on it for BE support (in case one of our = changes >>>> kills it). I'll run unixbench on it a few times to benchmark the = ASLR >>>> patch. I'll test these three scenarios: >>>> 1) ASLR compiled in and enabled; >>>> 2) ASLR compiled in and disabled; >>>> 3) ASLR compiled out (GENERIC kernel). >>>>=20 >>>> In each of these three scenarios, I'll have the kernel debugging = features=20 >>>> (WITNESS, INVARIANTS, etc.) turned off to better simulate a = production=20 >>>> system and to remove just one more variable in the tests. >>>>=20 >>>> I'll run unixbench ten times under each scenario and I'll compute = averages. >>>>=20 >>>> Since this is an older laptop (and it's running ZFS), these tests = will take=20 >>>> a couple days. I'll have an answer for you soon. >>>=20 >>> Hi Shawn: >>>=20 >>> Great news that this work is coming to fruition -- ASLR is long = overdue. >>>=20 >>> Are you having any luck with performance measurements? Unixbench = seems like a=20 >>> good starting point, but I wonder if it would be useful to look, in=20= >>> particular, at memory-mapping intensive workloads that might be = affected as a=20 >>> result of changes in kernel VM data-structure use, or greater = fragmentation of=20 >>> the address space. I'm not sure I have a specific application here = in mind --=20 >>> in the past I might have pointed out tools such as ElectricFence = that tend to=20 >>> increase fragmentation themselves. >>=20 >> The unixbench tests on that laptop have finished. However, I've been >> fighting a pesky migraine these last couple days, so I haven't had = the >> opportunity to aggregate the results into a nice little spreadsheet. = I'm >> hoping to finish it up by the end of the week. >>=20 >> I'll take a look at ElectricFence this weekend. Additionally, I have = a >> netbook somewhere. Once I find it and its power cord, I'll install >> FreeBSD/x86 and re-run the same tests on that. >>=20 >>>=20 >>> Also, could you say a little more about the effects that the change = might have=20 >>> on transparent superpage use -- other than suitable alignment of = large=20 >>> mappings, it's not clear to me what effect it might have. >>=20 >> Since we're just modifying the hint passed to the underlying VM = system, >> superpage support works as it should with ASLR enabled. The VM system >> will modify the hint in order to be able to use superpages. In those >> cases, we might lose a little bit of entropy. However, due to = superpages >> (on amd64, at least) requring 2MB alignment, you'd lose some entropy = no >> matter how ASLR was implemented--at the end of the day, you need that >> alignment for superpages to work. >>=20 >>>=20 >>> I wonder if some equipment in the FreeBSD Netperf cluster might be = used to=20 >>> help with performance characterisation -- in particular, very recent = high-end=20 >>> server hardware, and also, lower-end embedded-style systems that = have markedly=20 >>> different virtual-memory implementations in hardware and software. = Often=20 >>> those two classes of systems see markedly different = performance-change=20 >>> characteristics as a result of greater cache-centrism and = instruction-level=20 >>> parallelism in the higher-end designs that can mask increases in = instruction=20 >>> count. >>=20 >> Any additional testing would be very much welcome. Our ASLR >> implementation misbehaves on ARM, so testing on ARM-based embedded >> devices is pretty limited. My next goal is to figure out why it bugs = out >> on ARM. Essentially, when a child process exits/dies and the parent >> process gets sent SIGCHLD, the parent process' pc register somehow = gets >> set to 0xc0000000 and segfaults. Here's a screenshot of the process: >> https://twitter.com/lattera/status/490529645997998080 >>=20 >> FreeBSD 11-CURRENT hasn't been stable at all on sparc64, even without >> the ASLR patches. I have an SunFire 280R box that I've attempted to = test >> ASLR our on, but I couldn't get a stable enough installation of = vanilla >> FreeBSD to work long enough to recompile world/kernel. And generating = an >> installation ISO from my amd64 box doesn't work as the VTOC8 = bootloader >> isn't recognized by the BIOS (not sure if that's what it's called in >> sparc land). >>=20 >>>=20 >>> I think someone has already commented that Peter Holm's help might = be=20 >>> enlisted; you have have seen his 'stress2' suite, which could help = with=20 >>> stability testing. >>=20 >> I'll take a look at that, too. Thanks a lot for your suggestions and >> feedback. >=20 > The unixbench results are in. The overall scores are below. >=20 > ASLR Disabled: 456.33 > ASLR Enabled: 357.05 > No ASLR: 474.03 >=20 > I've uploaded the raw results to > http://0xfeedface.org/~shawn/aslr/2014-07-24_benchmark.tar.gz >=20 > Take these results with a grain of salt, given that some of = unixbench's > test are filesystem-related and I'm running ZFS on an old laptop with > little RAM. It does show that there is a performance impact when ASLR = is > enabled. >=20 The effect of the available RAM and/or age of the system is independent of ASLR being on or off si I think the results are valid. As Robert said, the particularly bad result is the effect of having ASLR disabled vs not having the patch at all. The priority should certainly = be to fix this.=20 The effect of ASLR seems significant, I did expect it less but it is = something that can be expected. People that enable ASLR have to be aware that = there are consequences performance-wise. A further effect that hasn=92t been = quantified but could be obtained from the raw data is the standard deviation. I = would expect that the randomization induced from using ASLR will cause an important = increase in the standard deviation and that performance will be basically = unpredictable. The last issue is a show stopper from enabling ASLR by default but the = first issue is a show stopper from committing the patch at all :(. Of course, all just IMHO. Pedro. From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 03:22:54 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E2C80ADE; Thu, 31 Jul 2014 03:22:53 +0000 (UTC) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2lp0240.outbound.protection.outlook.com [207.46.163.240]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 731452ACE; Thu, 31 Jul 2014 03:22:52 +0000 (UTC) Received: from BY2PR05CA005.namprd05.prod.outlook.com (10.242.32.35) by DM2PR05MB733.namprd05.prod.outlook.com (10.141.178.18) with Microsoft SMTP Server (TLS) id 15.0.995.14; Thu, 31 Jul 2014 03:22:44 +0000 Received: from BL2FFO11FD039.protection.gbl (2a01:111:f400:7c09::177) by BY2PR05CA005.outlook.office365.com (2a01:111:e400:2c2a::35) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Thu, 31 Jul 2014 03:22:44 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BL2FFO11FD039.mail.protection.outlook.com (10.173.161.135) with Microsoft SMTP Server (TLS) id 15.0.990.10 via Frontend Transport; Thu, 31 Jul 2014 03:22:43 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Wed, 30 Jul 2014 20:22:42 -0700 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6V3Men16865; Wed, 30 Jul 2014 20:22:40 -0700 (PDT) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id 8F711580A2; Wed, 30 Jul 2014 20:22:40 -0700 (PDT) To: Garance A Drosehn Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <82CFA67F-BA93-44EE-BD4B-9105F89AD157@rpi.edu> References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <82CFA67F-BA93-44EE-BD4B-9105F89AD157@rpi.edu> Comments: In-reply-to: Garance A Drosehn message dated "Wed, 30 Jul 2014 21:37:54 -0400." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Wed, 30 Jul 2014 20:22:40 -0700 Message-ID: <20140731032240.8F711580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(199002)(189002)(377424004)(87936001)(87286001)(2171001)(88136002)(31966008)(81156004)(106466001)(19580395003)(83322001)(76482001)(46102001)(19580405001)(6806004)(47776003)(89996001)(15975445006)(20776003)(84676001)(44976005)(92726001)(74502001)(74662001)(86362001)(92566001)(95666004)(85306003)(70486001)(105596002)(97736001)(104166001)(102836001)(93546004)(110136001)(107046002)(102176002)(80022001)(77156001)(69596002)(76176999)(21056001)(50226001)(77982001)(101356003)(57986006)(99396002)(15202345003)(90896003)(48376002)(83072002)(81342001)(85852003)(33656002)(50986999)(50466002)(64706001)(81542001)(68736004)(79102001)(62966002)(76506005)(93916002)(42262001); DIR:OUT; SFP:; SCL:1; SRVR:DM2PR05MB733; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0289B6431E Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=sjg@juniper.net; X-OriginatorOrg: juniper.net Cc: arch@freebsd.org, marcel@freebsd.org, phil@juniper.net X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 03:22:54 -0000 On Wed, 30 Jul 2014 21:37:54 -0400, Garance A Drosehn writes: >Do you have links to the library itself? Not yet. >I'd suggest that the above isn't quite what one would want, either. >For the text version it's fine to have a time-of-day value as >'6:47PM', but if you're going for machine-readable output then Actually for machine readable output the most useful thing is the utc seconds. Eg (in case you don't have access to a Junos router): root@vjb5> show system uptime | display xml 2014-07-30 20:19:25 PDT etc. The patch to w(1) was mostly to demo the API I think. >It's hard to tell based on your sample output, but there's also >the question of truncating strings. In the text output of your >example, it obviously makes sense to truncate the 'WHAT' value >to 'ssh svl-junos-d026.juniper.net', but when printing the same >output in a machine-readable format you wouldn't want to truncate >it. Let the application which *reads* the data decide how many >characters *it* wants to use. Yes. From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 03:41:11 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CD6A6FB; Thu, 31 Jul 2014 03:41:11 +0000 (UTC) Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2lp0204.outbound.protection.outlook.com [207.46.163.204]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9EF502CEC; Thu, 31 Jul 2014 03:41:10 +0000 (UTC) Received: from BLUPR05CA0081.namprd05.prod.outlook.com (10.141.20.51) by DM2PR05MB733.namprd05.prod.outlook.com (10.141.178.18) with Microsoft SMTP Server (TLS) id 15.0.995.14; Thu, 31 Jul 2014 03:41:07 +0000 Received: from BL2FFO11FD015.protection.gbl (2a01:111:f400:7c09::176) by BLUPR05CA0081.outlook.office365.com (2a01:111:e400:855::51) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Thu, 31 Jul 2014 03:41:07 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BL2FFO11FD015.mail.protection.outlook.com (10.173.160.223) with Microsoft SMTP Server (TLS) id 15.0.990.10 via Frontend Transport; Thu, 31 Jul 2014 03:41:07 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Wed, 30 Jul 2014 20:41:04 -0700 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6V3eVn27065; Wed, 30 Jul 2014 20:41:01 -0700 (PDT) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id B2FC5580A2; Wed, 30 Jul 2014 20:40:31 -0700 (PDT) To: Garance A Drosehn Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <82CFA67F-BA93-44EE-BD4B-9105F89AD157@rpi.edu> References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <82CFA67F-BA93-44EE-BD4B-9105F89AD157@rpi.edu> Comments: In-reply-to: Garance A Drosehn message dated "Wed, 30 Jul 2014 21:37:54 -0400." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Wed, 30 Jul 2014 20:40:31 -0700 Message-ID: <20140731034031.B2FC5580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(979002)(6009001)(189002)(199002)(69596002)(77156001)(4396001)(77982001)(50226001)(76176999)(21056001)(93546004)(110136001)(102836001)(80022001)(107046002)(102176002)(81542001)(79102001)(68736004)(50466002)(50986999)(64706001)(93916002)(62966002)(76506005)(57986006)(101356003)(99396002)(81342001)(85852003)(33656002)(83072002)(90896003)(48376002)(83322001)(76482001)(46102001)(44976005)(92726001)(20776003)(84676001)(6806004)(89996001)(47776003)(88136002)(2171001)(87286001)(87936001)(31966008)(81156004)(106466001)(92566001)(95666004)(86362001)(97736001)(104166001)(70486001)(85306003)(105596002)(74662001)(74502001)(42262001)(969003)(989001)(999001)(1009001)(1019001); DIR:OUT; SFP:; SCL:1; SRVR:DM2PR05MB733; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:ovrnspm; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0289B6431E Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=sjg@juniper.net; X-OriginatorOrg: juniper.net Cc: arch@freebsd.org, marcel@freebsd.org, phil@juniper.net X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 03:41:12 -0000 On Wed, 30 Jul 2014 21:37:54 -0400, Garance A Drosehn writes: >It's hard to tell based on your sample output, but there's also >the question of truncating strings. In the text output of your Perhaps more than a "yes" answer is warranted. w(1) sizes those strings based on its expectation of the width of the tty. That can obviously be fixed/improved - but involves exposing knowledge of the out format (or at least that it isn't TXT) to the application. I'm sure there will be many cases where that sort of thing may be needed/useful, but that's almost orthogonal to the API question. As I mentioned, this patch was demoing the API not changing how w(1) works, when apps are converted for real people can go as crazy as they like (ok not true; it really is important to have consitency in terms of the structure eg that the same XML tags are used for the same data regardless of application - we have a team of people to keep an eye on that stuff). --sjg From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 03:53:44 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9C83C2AD; Thu, 31 Jul 2014 03:53:44 +0000 (UTC) Received: from smtp9.server.rpi.edu (gateway.canit.rpi.edu [128.113.2.229]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5AA2B2EC1; Thu, 31 Jul 2014 03:53:43 +0000 (UTC) Received: from smtp-auth1.server.rpi.edu (smtp-auth1.server.rpi.edu [128.113.2.231]) by smtp9.server.rpi.edu (8.14.3/8.14.3/Debian-9.4) with ESMTP id s6V3rfR3027464; Wed, 30 Jul 2014 23:53:42 -0400 Received: from smtp-auth1.server.rpi.edu (localhost [127.0.0.1]) by smtp-auth1.server.rpi.edu (Postfix) with ESMTP id C8D2F5821B; Wed, 30 Jul 2014 23:53:41 -0400 (EDT) Received: from [192.168.1.6] (cpe-72-231-155-88.nycap.res.rr.com [72.231.155.88]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: drosih) by smtp-auth1.server.rpi.edu (Postfix) with ESMTPSA id 837565800C; Wed, 30 Jul 2014 23:53:41 -0400 (EDT) From: "Garance A Drosehn" To: "Simon J. Gerraty" Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Date: Wed, 30 Jul 2014 23:53:12 -0400 Message-ID: <34EB8587-0F01-4F34-89D5-F0849062DC6F@rpi.edu> In-Reply-To: <20140731032240.8F711580A2@chaos.jnpr.net> References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <82CFA67F-BA93-44EE-BD4B-9105F89AD157@rpi.edu> <20140731032240.8F711580A2@chaos.jnpr.net> MIME-Version: 1.0 X-Mailer: MailMate (1.7.2r3905) X-Virus-Scanned: ClamAV using ClamSMTP X-Bayes-Prob: 0.0001 (Score 0, tokens from: outgoing, @@RPTN) X-Spam-Score: 0.00 () [Hold at 15.10] X-CanIt-Incident-Id: 02MwPRGFq X-CanIt-Geo: ip=72.231.155.88; country=US; region=New York; city=Troy; latitude=42.7495; longitude=-73.5951; http://maps.google.com/maps?q=42.7495,-73.5951&z=6 X-CanItPRO-Stream: outgoing X-Canit-Stats-ID: Bayes signature not available X-Scanned-By: CanIt (www . roaringpenguin . com) on 128.113.2.229 Cc: arch@freebsd.org, marcel@freebsd.org, phil@juniper.net X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 03:53:44 -0000 On 30 Jul 2014, at 23:22, Simon J. Gerraty wrote: > On Wed, 30 Jul 2014 21:37:54 -0400, Garance A Drosehn writes: > >> I'd suggest that the above isn't quite what one would want, either. >> For the text version it's fine to have a time-of-day value as >> '6:47PM', but if you're going for machine-readable output then > > Actually for machine readable output the most useful thing is the utc > seconds. Eg (in case you don't have access to a Junos router): > While that may be the most useful thing for the applications on Junos, it is not necessarily the most useful thing for other apps. And in the case of EDN, if one was going to write a timestamp they might as well write it in the format expected by other EDN-based apps. IMO. -- Garance Alistair Drosehn = drosih@rpi.edu Senior Systems Programmer or gad@FreeBSD.org Rensselaer Polytechnic Institute; Troy, NY; USA From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 05:02:04 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 38749F32; Thu, 31 Jul 2014 05:02:04 +0000 (UTC) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id DECDD2557; Thu, 31 Jul 2014 05:02:03 +0000 (UTC) Received: from Julian-MBP3.local (ppp121-45-250-191.lns20.per2.internode.on.net [121.45.250.191]) (authenticated bits=0) by vps1.elischer.org (8.14.9/8.14.9) with ESMTP id s6V51vc5024599 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Wed, 30 Jul 2014 22:01:59 -0700 (PDT) (envelope-from julian@freebsd.org) Message-ID: <53D9CDBF.6030003@freebsd.org> Date: Thu, 31 Jul 2014 13:01:51 +0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Pedro Giffuni , Shawn Webb Subject: Re: [RFC] ASLR Whitepaper and Candidate Final Patch References: <96C72773-3239-427E-A90B-D05FF0F5B782@freebsd.org> <20140720201858.GB29618@pwnie.vrt.sourcefire.com> <20140723004543.GH29618@pwnie.vrt.sourcefire.com> <20140724175704.GT29618@pwnie.vrt.sourcefire.com> <112E989D-607B-47AC-8942-0FB049DA6C3D@freebsd.org> In-Reply-To: <112E989D-607B-47AC-8942-0FB049DA6C3D@freebsd.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Cc: PaX Team , freebsd-arch@freebsd.org, Bryan Drewery , Robert Watson , Oliver Pinter X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 05:02:04 -0000 On 7/31/14, 11:00 AM, Pedro Giffuni wrote: > Il giorno 24/lug/2014, alle ore 12:57, Shawn Webb ha scritto: > >> >> The unixbench results are in. The overall scores are below. >> >> ASLR Disabled: 456.33 >> ASLR Enabled: 357.05 >> No ASLR: 474.03 is bigger better or worse? >> >> I've uploaded the raw results to >> http://0xfeedface.org/~shawn/aslr/2014-07-24_benchmark.tar.gz >> >> Take these results with a grain of salt, given that some of unixbench's >> test are filesystem-related and I'm running ZFS on an old laptop with >> little RAM. It does show that there is a performance impact when ASLR is >> enabled. and even from being included. I don't believe that should be forgotten. 1% here ,2% there.. soon you have lost noticeable performance. > The effect of the available RAM and/or age of the system is > independent of ASLR being on or off si I think the results are valid. > > As Robert said, the particularly bad result is the effect of having ASLR > disabled vs not having the patch at all. The priority should certainly be > to fix this. > > The effect of ASLR seems significant, I did expect it less but it is something > that can be expected. People that enable ASLR have to be aware that there > are consequences performance-wise. A further effect that hasnt been quantified > but could be obtained from the raw data is the standard deviation. I would expect > that the randomization induced from using ASLR will cause an important increase > in the standard deviation and that performance will be basically unpredictable. > > The last issue is a show stopper from enabling ASLR by default but the first > issue is a show stopper from committing the patch at all :(. > > Of course, all just IMHO. > > Pedro. > > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > > > From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 06:31:43 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B8910EB9; Thu, 31 Jul 2014 06:31:43 +0000 (UTC) Received: from mail-wi0-x236.google.com (mail-wi0-x236.google.com [IPv6:2a00:1450:400c:c05::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 062432D47; Thu, 31 Jul 2014 06:31:42 +0000 (UTC) Received: by mail-wi0-f182.google.com with SMTP id d1so3525145wiv.3 for ; Wed, 30 Jul 2014 23:31:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:mail-followup-to :references:mime-version:content-type:content-disposition :content-transfer-encoding:in-reply-to:user-agent; bh=/QnkKsq7MQuDAGX9dwh0j9zpU5aEg7Uo3c8DXgV5Dyo=; b=G5/Q4S9P1o/Xiv4jx0Dfo+pTLX+IwRoetVGZnDwam9AaBVHkkYuvreyeZgnAaaH1XR dRMRM5M9w7dLatuIikFnXhxRGGQGQkZj63Q2W5XIZ+5ZnzIVnzp8td7PK9voiB/80BG7 neS0jkZ+0sQjjQzQgqAg6TfPG2gexnYNlphKv8YgzuyfYPcHiavY+O6XrV13juKKj6oD V1QL5vNKsZ9bywb6izUEBKDVrDSp39kEuEI6yUCW4V4a9C20T6RKmofPhXxKBYbB1ZDb Jy7qlnIj9blhFhtsiyTgEWZjusW6ceOmLAn25Hw5OCNFiwb3gSlILMG7ztTpuuprhtcn TCrw== X-Received: by 10.194.122.73 with SMTP id lq9mr13054924wjb.133.1406788301201; Wed, 30 Jul 2014 23:31:41 -0700 (PDT) Received: from brick.home (acd38.neoplus.adsl.tpnet.pl. [83.25.55.38]) by mx.google.com with ESMTPSA id u10sm17328847wix.14.2014.07.30.23.31.39 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 30 Jul 2014 23:31:40 -0700 (PDT) Sender: =?UTF-8?Q?Edward_Tomasz_Napiera=C5=82a?= Date: Thu, 31 Jul 2014 08:31:38 +0200 From: Edward Tomasz =?utf-8?Q?Napiera=C5=82a?= To: Craig Rodrigues Subject: Re: [CFT] Autofs. Message-ID: <20140731063138.GA12000@brick.home> Mail-Followup-To: Craig Rodrigues , freebsd-current Current , freebsd-arch References: <20140730071933.GA20122@pc5.home> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Cc: freebsd-current Current , freebsd-arch X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 06:31:43 -0000 On 0730T1851, Craig Rodrigues wrote: > On Wed, Jul 30, 2014 at 12:19 AM, Edward Tomasz Napierała > wrote: > > > At the link below you will find a patch that adds the new automounter. > > The patch is against yesterdays 11.0-CURRENT. > > > > http://people.freebsd.org/~trasz/autofs-head-20140729.diff > > > > > Hi, > > Nice stuff! I updated a current system, and tried the patch. > I put autofs_enable="YES" in /etc/rc.conf > > In my network, I have a system, testhost with some NFS exports. > > /dir1/export1 > /dir1/export2 > /dir1/export3 > > I did: > > cd /net/testhost > > This operation seemed to timeout. > In dmesg, I saw these errors: > > WARNING: autofs_callout: request 0 for /net/ timed out after 30 seconds > WARNING: autofs_trigger_one: request for /net/ completed with error 60 > autofs_trigger: trigger failed with error 60; will retry in 1 seconds, 2 > attempts left Here is how to debug it: kill automountd, and then start it like this: while :; do automountd -d; done This will make automountd print debug information to standard output, which should provide a clue on what's wrong. From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 08:30:18 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 253A77AA; Thu, 31 Jul 2014 08:30:18 +0000 (UTC) Received: from cyrus.watson.org (cyrus.watson.org [198.74.231.69]) by mx1.freebsd.org (Postfix) with ESMTP id D43482993; Thu, 31 Jul 2014 08:30:17 +0000 (UTC) Received: from [10.0.1.9] (host86-132-108-36.range86-132.btcentralplus.com [86.132.108.36]) by cyrus.watson.org (Postfix) with ESMTPSA id 7D6D246B2C; Thu, 31 Jul 2014 04:30:15 -0400 (EDT) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: [RFC] ASLR Whitepaper and Candidate Final Patch From: "Robert N. M. Watson" In-Reply-To: <112E989D-607B-47AC-8942-0FB049DA6C3D@freebsd.org> Date: Thu, 31 Jul 2014 09:30:11 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <96C72773-3239-427E-A90B-D05FF0F5B782@freebsd.org> <20140720201858.GB29618@pwnie.vrt.sourcefire.com> <20140723004543.GH29618@pwnie.vrt.sourcefire.com> <20140724175704.GT29618@pwnie.vrt.sourcefire.com> <112E989D-607B-47AC-8942-0FB049DA6C3D@freebsd.org> To: Pedro Giffuni X-Mailer: Apple Mail (2.1878.6) Cc: PaX Team , freebsd-arch@freebsd.org, Oliver Pinter , Bryan Drewery , Shawn Webb X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 08:30:18 -0000 On 31 Jul 2014, at 04:00, Pedro Giffuni wrote: > The effect of the available RAM and/or age of the system is > independent of ASLR being on or off si I think the results are valid. >=20 > As Robert said, the particularly bad result is the effect of having = ASLR > disabled vs not having the patch at all. The priority should certainly = be > to fix this.=20 >=20 > The effect of ASLR seems significant, I did expect it less but it is = something > that can be expected. People that enable ASLR have to be aware that = there > are consequences performance-wise. A further effect that hasn=92t been = quantified > but could be obtained from the raw data is the standard deviation. I = would expect > that the randomization induced from using ASLR will cause an important = increase > in the standard deviation and that performance will be basically = unpredictable. >=20 > The last issue is a show stopper from enabling ASLR by default but the = first > issue is a show stopper from committing the patch at all :(. >=20 > Of course, all just IMHO. I'm not sure that the three numbers presented are meaningful at all: we = can't tell if the differences are significant or not, because we don't = have a sense of the distributions from which the numbers fall. If the = difference between 'no patch' and 'disabled' is in fact negligible as = one would expect given the contents of the patch, then it suggests the = variance in the tool's output number is quite high -- calling into = question any interpretation of the difference between 'disabled' and = 'enabled'. Selecting some suitable benchmarks and doing a proper = analysis is definitely required. The other more abstract point here is not the immediate performance = consequences, but also the effect they have on future possible = optimisations. In the past, we (and many others) have toyed with = prelinking techniques, in which run-time linking costs are amortised by = preestablishing virtual-address bindings for frequently used libraries. = That optimisation is basically entirely antithetical to ASLR, which is = premised on doing precisely the opposite. I don't take a strong view on = the value of prelinking, although will observe that it is a technique = that pops in and out of fashion pretty frequently; it's currently 'out' = but presumably will be back 'in' again in the future. With prelinking, = there is presumably a sharp cliff associated with ASLR being enabled. I would quite like to understand where the modest 'expected' performance = drop in ASLR is supposed to come from; as the PaX folk have suggested, a = decent implementation might have little or no overhead if we think it = doesn't seriously disturb VM/TLB use on 64-bit platforms -- there's not = very much cost to pulling a bit of entropy from Yarrow/Fortuna if it is = suitably amortised, and unlike on Linux, you won't block waiting on it = due to Yarrow/Fortuna sitting between the consumer and the entropy pool. = On 32-bit systems you'd expect ASLR to be both pretty ineffective in = terms of bits of entropy available, and catastrophic for address-space = fragmentation, but on 64-bit systems much more reasonable. Does anyone = have any thoughts on this? Robert From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 08:44:14 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 62795AD6; Thu, 31 Jul 2014 08:44:14 +0000 (UTC) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id 25E102B1C; Thu, 31 Jul 2014 08:44:13 +0000 (UTC) Received: from critter.freebsd.dk (unknown [192.168.48.2]) by phk.freebsd.dk (Postfix) with ESMTP id 505101578; Thu, 31 Jul 2014 08:44:06 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.14.9/8.14.9) with ESMTP id s6V8i33h094842; Thu, 31 Jul 2014 08:44:03 GMT (envelope-from phk@phk.freebsd.dk) To: Phil Shafer Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-reply-to: <201407302324.s6UNOB2H087915@idle.juniper.net> From: "Poul-Henning Kamp" References: <201407302324.s6UNOB2H087915@idle.juniper.net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <94840.1406796243.1@critter.freebsd.dk> Date: Thu, 31 Jul 2014 08:44:03 +0000 Message-ID: <94841.1406796243@critter.freebsd.dk> Cc: sjg@freebsd.org, arch@freebsd.org, John-Mark Gurney , marcel@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 08:44:14 -0000 -------- In message <201407302324.s6UNOB2H087915@idle.juniper.net>, Phil Shafer writes: >John-Mark Gurney writes: >>My vote would be to use and *enforce* UTF-8 by the API. That means if >>someone passes a string in, it must be properly formed UTF-8... Given that libxo is just now starting to stare at trunk and that the next .0 release is some way of still, the consequences of enforcing UTF-8 will not be seen by users at large until mid 2015 or so. The real question is how it will be perceived if we are *not* actively going UTF-8 by that time? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 09:08:06 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3294E63A; Thu, 31 Jul 2014 09:08:06 +0000 (UTC) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2lp0242.outbound.protection.outlook.com [207.46.163.242]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 454F62D7C; Thu, 31 Jul 2014 09:08:05 +0000 (UTC) Received: from BY2PR05CA043.namprd05.prod.outlook.com (10.141.250.33) by BLUPR05MB722.namprd05.prod.outlook.com (10.141.207.150) with Microsoft SMTP Server (TLS) id 15.0.995.14; Thu, 31 Jul 2014 09:08:03 +0000 Received: from BL2FFO11FD023.protection.gbl (2a01:111:f400:7c09::176) by BY2PR05CA043.outlook.office365.com (2a01:111:e400:2c5f::33) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Thu, 31 Jul 2014 09:08:02 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BL2FFO11FD023.mail.protection.outlook.com (10.173.161.102) with Microsoft SMTP Server (TLS) id 15.0.990.10 via Frontend Transport; Thu, 31 Jul 2014 09:08:02 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Thu, 31 Jul 2014 02:08:01 -0700 Received: from idle.juniper.net (idleski.juniper.net [172.25.4.26]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6V97tn41852; Thu, 31 Jul 2014 02:07:55 -0700 (PDT) (envelope-from phil@juniper.net) Received: from idle.juniper.net (localhost [127.0.0.1]) by idle.juniper.net (8.14.4/8.14.3) with ESMTP id s6V97Qsw092908; Thu, 31 Jul 2014 05:07:41 -0400 (EDT) (envelope-from phil@idle.juniper.net) Message-ID: <201407310907.s6V97Qsw092908@idle.juniper.net> To: "Simon J. Gerraty" Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <20140730235123.AC944580A2@chaos.jnpr.net> Date: Thu, 31 Jul 2014 05:07:26 -0400 From: Phil Shafer MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(377424004)(164054003)(189002)(199002)(54356999)(81342001)(74662001)(99396002)(102836001)(21056001)(64706001)(86362001)(110136001)(50986999)(74502001)(107046002)(20776003)(47776003)(31966008)(80022001)(48376002)(97736001)(76482001)(81542001)(77982001)(50466002)(15202345003)(84676001)(53416004)(103666002)(83322001)(105596002)(69596002)(44976005)(1941001)(106466001)(92726001)(19580395003)(46102001)(15975445006)(85306003)(79102001)(68736004)(6806004)(81156004)(83072002)(4396001)(85852003)(76506005)(95666004)(87936001)(92566001); DIR:OUT; SFP:; SCL:1; SRVR:BLUPR05MB722; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0289B6431E Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=phil@juniper.net; X-OriginatorOrg: juniper.net Cc: Alfred Perlstein , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 09:08:06 -0000 "Simon J. Gerraty" writes: >We now have an API and library that can avoid the need to double the >cost of adding new output to apps, ie you only write to libxo's api >and it does plain TXT for you if needed. I'm appending another example, using "wc". I've added the "xo" equivalent of warn/warnx/err/errx/etc. Normally, they'll make the expected noise on stderr, but with the XO_WARN_XML flag, they generate XML content on the stdout (or whatever xo_handle_t is in use). I don't have a JSON equivalent because, well, I don't know what the json equivalent would look like. There's also a xo_attr function that will add an attribute to the next element emitted, so one can: xo_attr("seconds", stop - start); xo_emit("{:time}", fancy_time(stop - start)); to make: There's also the "printf(1)" equivalent command "xo" for use in shell scripts: % xo --wrap response/data 'The {:animal} is {:color} and {:mood}\n' bear brown angry The bear is brown and angry % xo -X -p --wrap response/data 'The {:animal} is {:color} and {:mood}\n' bear brown angry bear brown angry The code is in github: https://github.com/Juniper/libxo HTML docs are here: http://juniper.github.io/libxo/libxo-manual.html Text/source docs are here: https://raw.githubusercontent.com/Juniper/libxo/master/doc/libxo.txt I'm using the common master/develop branching scheme, so "master" should be stable, while "develop" is where I'm working. Simon said there's a github mirror for freebsd, so I make fork that and add the patches for the various utilities I'm working thru. Thanks, Phil ----------- diff -rbu /usr/src/usr.bin/wc/wc.c ./wc.c --- /usr/src/usr.bin/wc/wc.c 2010-12-21 12:09:25.000000000 -0500 +++ ./wc.c 2014-07-31 04:20:15.000000000 -0400 @@ -61,6 +61,7 @@ #include #include #include +#include uintmax_t tlinect, twordct, tcharct, tlongline; int doline, doword, dochar, domulti, dolongline; @@ -105,33 +106,45 @@ if (doline + doword + dochar + domulti + dolongline == 0) doline = doword = dochar = 1; + xo_open_container("wc"); + xo_open_list("file"); errors = 0; total = 0; if (!*argv) { + xo_open_instance("file"); if (cnt((char *)NULL) != 0) ++errors; else - (void)printf("\n"); + xo_emit("\n"); + xo_close_instance("file"); } else do { + xo_open_instance("file"); + xo_emit(" {ek:filename/%s}\n", *argv); if (cnt(*argv) != 0) ++errors; else - (void)printf(" %s\n", *argv); + xo_emit(" {d:filename/%s}\n", *argv); + xo_close_instance("file"); ++total; } while(*++argv); + xo_close_list("file"); if (total > 1) { + xo_open_container("total"); if (doline) - (void)printf(" %7ju", tlinect); + xo_emit(" {:lines/%7ju/%ju}", tlinect); if (doword) - (void)printf(" %7ju", twordct); + xo_emit(" {:words/%7ju/%ju", twordct); if (dochar || domulti) - (void)printf(" %7ju", tcharct); + xo_emit(" {:characters/%7ju/%ju}", tcharct); if (dolongline) - (void)printf(" %7ju", tlongline); - (void)printf(" total\n"); + xo_emit(" {:long-lines/%7ju/%ju}", tlongline); + xo_emit(" total\n"); + xo_close_container("total"); } + xo_close_container("wc"); + xo_flush(); exit(errors == 0 ? 0 : 1); } @@ -154,7 +167,7 @@ fd = STDIN_FILENO; } else { if ((fd = open(file, O_RDONLY, 0)) < 0) { - warn("%s: open", file); + xo_warn("%s: open", file); return (1); } if (doword || (domulti && MB_CUR_MAX != 1)) @@ -167,7 +180,7 @@ if (doline) { while ((len = read(fd, buf, MAXBSIZE))) { if (len == -1) { - warn("%s: read", file); + xo_warn("%s: read", file); (void)close(fd); return (1); } @@ -182,15 +195,15 @@ tmpll++; } tlinect += linect; - (void)printf(" %7ju", linect); + xo_emit(" {:lines/%7ju/%ju}", linect); if (dochar) { tcharct += charct; - (void)printf(" %7ju", charct); + xo_emit(" {:characters/%7ju/%ju}", charct); } if (dolongline) { if (llct > tlongline) tlongline = llct; - (void)printf(" %7ju", tlongline); + xo_emit(" {:long-lines/%7ju/%ju}", tlongline); } (void)close(fd); return (0); @@ -201,12 +214,13 @@ */ if (dochar || domulti) { if (fstat(fd, &sb)) { - warn("%s: fstat", file); + xo_warn("%s: fstat", file); (void)close(fd); return (1); } if (S_ISREG(sb.st_mode)) { - (void)printf(" %7lld", (long long)sb.st_size); + xo_emit(" {:characters/%7lld/%lld}", + (long long)sb.st_size); tcharct += sb.st_size; (void)close(fd); return (0); @@ -220,7 +234,7 @@ memset(&mbs, 0, sizeof(mbs)); while ((len = read(fd, buf, MAXBSIZE)) != 0) { if (len == -1) { - warn("%s: read", file); + xo_warn("%s: read", file); (void)close(fd); return (1); } @@ -233,7 +247,7 @@ (size_t)-1) { if (!warned) { errno = EILSEQ; - warn("%s", file); + xo_warn("%s", file); warned = 1; } memset(&mbs, 0, sizeof(mbs)); @@ -264,23 +278,23 @@ } if (domulti && MB_CUR_MAX > 1) if (mbrtowc(NULL, NULL, 0, &mbs) == (size_t)-1 && !warned) - warn("%s", file); + xo_warn("%s", file); if (doline) { tlinect += linect; - (void)printf(" %7ju", linect); + xo_emit(" {:lines/%7ju/%ju}", linect); } if (doword) { twordct += wordct; - (void)printf(" %7ju", wordct); + xo_emit(" {:words/%7ju/%ju}", wordct); } if (dochar || domulti) { tcharct += charct; - (void)printf(" %7ju", charct); + xo_emit(" {:characters/%7ju/%ju}", charct); } if (dolongline) { if (llct > tlongline) tlongline = llct; - (void)printf(" %7ju", llct); + xo_emit(" {:long-lines/%7ju/%ju}", llct); } (void)close(fd); return (0); @@ -289,6 +303,6 @@ static void usage() { - (void)fprintf(stderr, "usage: wc [-Lclmw] [file ...]\n"); + xo_error("usage: wc [-Lclmw] [file ...]\n"); exit(1); } From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 09:19:01 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EFDCCBF0; Thu, 31 Jul 2014 09:19:00 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1blp0190.outbound.protection.outlook.com [207.46.163.190]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7DF812F95; Thu, 31 Jul 2014 09:18:59 +0000 (UTC) Received: from BLUPR05CA006.namprd05.prod.outlook.com (10.255.219.164) by BLUPR05MB722.namprd05.prod.outlook.com (10.141.207.150) with Microsoft SMTP Server (TLS) id 15.0.995.14; Thu, 31 Jul 2014 09:18:50 +0000 Received: from BN1AFFO11FD047.protection.gbl (2a01:111:f400:7c10::145) by BLUPR05CA006.outlook.office365.com (2a01:111:e400:83f::36) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Thu, 31 Jul 2014 09:18:50 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BN1AFFO11FD047.mail.protection.outlook.com (10.58.53.62) with Microsoft SMTP Server (TLS) id 15.0.990.10 via Frontend Transport; Thu, 31 Jul 2014 09:18:50 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Thu, 31 Jul 2014 02:18:48 -0700 Received: from idle.juniper.net (idleski.juniper.net [172.25.4.26]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6V9Iin49714; Thu, 31 Jul 2014 02:18:44 -0700 (PDT) (envelope-from phil@juniper.net) Received: from idle.juniper.net (localhost [127.0.0.1]) by idle.juniper.net (8.14.4/8.14.3) with ESMTP id s6V9IRj5092988; Thu, 31 Jul 2014 05:18:31 -0400 (EDT) (envelope-from phil@idle.juniper.net) Message-ID: <201407310918.s6V9IRj5092988@idle.juniper.net> To: "Simon J. Gerraty" Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <20140731034031.B2FC5580A2@chaos.jnpr.net> Date: Thu, 31 Jul 2014 05:18:27 -0400 From: Phil Shafer MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(164054003)(189002)(199002)(69596002)(83322001)(105596002)(1941001)(44976005)(103666002)(79102001)(85306003)(83072002)(68736004)(81156004)(6806004)(87936001)(92566001)(95666004)(4396001)(76506005)(85852003)(92726001)(106466001)(46102001)(110136001)(50986999)(74502001)(81342001)(74662001)(54356999)(86362001)(102836001)(99396002)(64706001)(21056001)(77982001)(81542001)(76482001)(97736001)(48376002)(53416004)(84676001)(50466002)(20776003)(107046002)(31966008)(80022001)(47776003); DIR:OUT; SFP:; SCL:1; SRVR:BLUPR05MB722; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0289B6431E Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=phil@juniper.net; X-OriginatorOrg: juniper.net Cc: arch@freebsd.org, marcel@freebsd.org, Garance A Drosehn X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 09:19:01 -0000 "Simon J. Gerraty" writes: >w(1) sizes those strings based on its expectation of the width of the >tty. That can obviously be fixed/improved - but involves exposing >knowledge of the out format (or at least that it isn't TXT) to the >application. libxo allows the field description to carry two distinct format descriptors, one for text/html and one for xml/json. The latter defaults to the former: xo_emit(" {:words/%7ju/%ju}", twordct); For "w", this is does the right thing; in text mode, the command string is truncated: 5:08AM up 27 days, 12:22, 11 user%s, load averages: 0.00, 0.00, 0.00 USER TTY FROM LOGIN@ IDLE WHAT phil pts/1 76.182.32.73 05Jul14 - /usr/bin/perl /u/phil/bin/plum ( ... In XML mode, the command is not truncated. 5:09AM 27 days 12:22 11 0.00 0.00 0.00 phil pts/1 76.182.32.73 05Jul14 1 /usr/bin/perl /u/phil/bin/plum (perl5.12.4) ... (Yes, I'm likely the only plum user left in the wild.) Thanks, Phil From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 15:48:03 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 62E50929; Thu, 31 Jul 2014 15:48:03 +0000 (UTC) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2lp0239.outbound.protection.outlook.com [207.46.163.239]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7C1682104; Thu, 31 Jul 2014 15:48:01 +0000 (UTC) Received: from BY2PR05CA051.namprd05.prod.outlook.com (10.141.250.41) by BLUPR05MB721.namprd05.prod.outlook.com (10.141.207.144) with Microsoft SMTP Server (TLS) id 15.0.995.14; Thu, 31 Jul 2014 15:32:23 +0000 Received: from BY2FFO11FD004.protection.gbl (2a01:111:f400:7c0c::159) by BY2PR05CA051.outlook.office365.com (2a01:111:e400:2c5f::41) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Thu, 31 Jul 2014 15:32:23 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BY2FFO11FD004.mail.protection.outlook.com (10.1.14.158) with Microsoft SMTP Server (TLS) id 15.0.990.10 via Frontend Transport; Thu, 31 Jul 2014 15:32:22 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Thu, 31 Jul 2014 08:32:21 -0700 Received: from idle.juniper.net (idleski.juniper.net [172.25.4.26]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6VFWIn29075; Thu, 31 Jul 2014 08:32:18 -0700 (PDT) (envelope-from phil@juniper.net) Received: from idle.juniper.net (localhost [127.0.0.1]) by idle.juniper.net (8.14.4/8.14.3) with ESMTP id s6VFVbJn094888; Thu, 31 Jul 2014 11:31:42 -0400 (EDT) (envelope-from phil@idle.juniper.net) Message-ID: <201407311531.s6VFVbJn094888@idle.juniper.net> To: Poul-Henning Kamp Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <94841.1406796243@critter.freebsd.dk> Date: Thu, 31 Jul 2014 11:31:37 -0400 From: Phil Shafer MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(164054003)(199002)(189002)(48376002)(84676001)(87936001)(19580395003)(102836001)(85852003)(69596002)(80022001)(64706001)(47776003)(15202345003)(20776003)(83322001)(15975445006)(81342001)(4396001)(50466002)(92566001)(83072002)(76482001)(44976005)(110136001)(46102001)(6806004)(31966008)(107046002)(54356999)(103666002)(106466001)(74662001)(92726001)(86362001)(76506005)(85306003)(81156004)(97736001)(21056001)(50986999)(74502001)(79102001)(81542001)(105596002)(53416004)(99396002)(95666004)(77982001)(68736004); DIR:OUT; SFP:; SCL:1; SRVR:BLUPR05MB721; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0289B6431E Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=phil@juniper.net; X-OriginatorOrg: juniper.net Cc: sjg@freebsd.org, arch@freebsd.org, John-Mark Gurney , marcel@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 15:48:03 -0000 Poul-Henning Kamp writes: >The real question is how it will be perceived if we are *not* actively >going UTF-8 by that time? And moving toward UTF-8 won't be simple. I just tossed a couple of file from http://www.cl.cam.ac.uk/~mgk25/ucs/examples/ into a few apps that are supposed to support UTF-8 (emacs, vim, firefox) and the results were underwhelming. Skimming the description of xterm support for UTF-8 gives some appreciation for how complex this crap is (http://www.cl.cam.ac.uk/~mgk25/unicode.html#xterm). That said, I'm a bit tired of explaining to my kids why I'm typing in black and white terminal windows in the age of HTML5 ;^) Thanks, Phil From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 16:41:07 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B84EB709; Thu, 31 Jul 2014 16:41:07 +0000 (UTC) Received: from smtp9.server.rpi.edu (gateway.canit.rpi.edu [128.113.2.229]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5CB092876; Thu, 31 Jul 2014 16:41:06 +0000 (UTC) Received: from smtp-auth2.server.rpi.edu (smtp-auth2.server.rpi.edu [128.113.2.232]) by smtp9.server.rpi.edu (8.14.3/8.14.3/Debian-9.4) with ESMTP id s6VGf4CL004714; Thu, 31 Jul 2014 12:41:05 -0400 Received: from smtp-auth2.server.rpi.edu (localhost [127.0.0.1]) by smtp-auth2.server.rpi.edu (Postfix) with ESMTP id C015F1814F; Thu, 31 Jul 2014 12:41:04 -0400 (EDT) Received: from [128.113.24.47] (gilead-qc124.netel.rpi.edu [128.113.124.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: drosih) by smtp-auth2.server.rpi.edu (Postfix) with ESMTPSA id AF98C180BD; Thu, 31 Jul 2014 12:41:04 -0400 (EDT) From: "Garance A Drosehn" To: "Phil Shafer" Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Date: Thu, 31 Jul 2014 12:41:03 -0400 Message-ID: <08B4AE10-1747-4BB1-AA1A-BA4491A57F95@rpi.edu> In-Reply-To: <201407310918.s6V9IRj5092988@idle.juniper.net> References: <201407310918.s6V9IRj5092988@idle.juniper.net> MIME-Version: 1.0 X-Mailer: MailMate (1.7.2r3905) X-Virus-Scanned: ClamAV using ClamSMTP X-Bayes-Prob: 0.0001 (Score 0, tokens from: outgoing, @@RPTN) X-Spam-Score: 0.00 () [Hold at 15.10] X-CanIt-Incident-Id: 02Mx4F4E5 X-CanIt-Geo: ip=128.113.124.17; country=US; region=New York; city=Troy; latitude=42.7495; longitude=-73.5951; http://maps.google.com/maps?q=42.7495,-73.5951&z=6 X-CanItPRO-Stream: outgoing X-Canit-Stats-ID: Bayes signature not available X-Scanned-By: CanIt (www . roaringpenguin . com) on 128.113.2.229 Cc: arch@freebsd.org, marcel@freebsd.org, "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 16:41:07 -0000 On 31 Jul 2014, at 5:18, Phil Shafer wrote: > "Simon J. Gerraty" writes: >> w(1) sizes those strings based on its expectation of the width of the >> tty. That can obviously be fixed/improved - but involves exposing >> knowledge of the out format (or at least that it isn't TXT) to the >> application. > > libxo allows the field description to carry two distinct format > descriptors, one for text/html and one for xml/json. The latter > defaults to the former: > > xo_emit(" {:words/%7ju/%ju}", twordct); > > For "w", this is does the right thing; in text mode, the command > string is truncated: Ah, that's helpful. What if there's something you want to print out for xml/json but *not* for plain-text? (it's easy to imagine some commands might print out more values when they are not constrained by an 80-char width). Also, given that machine-readable outputs might change over time, is there the idea of including a version-number with the output for each command? I hope it doesn't seem like I'm just complaining about the work everyone else is doing (both this and the GSOC project). Please note that I've attempted to do this a few times myself, and I haven't been happy with any of *my* attempts either! -- Garance Alistair Drosehn = drosih@rpi.edu Senior Systems Programmer or gad@FreeBSD.org Rensselaer Polytechnic Institute; Troy, NY; USA From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 17:47:32 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CCFF37B1; Thu, 31 Jul 2014 17:47:32 +0000 (UTC) Received: from mail-qg0-x233.google.com (mail-qg0-x233.google.com [IPv6:2607:f8b0:400d:c04::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7BFCE20DB; Thu, 31 Jul 2014 17:47:32 +0000 (UTC) Received: by mail-qg0-f51.google.com with SMTP id a108so4403348qge.24 for ; Thu, 31 Jul 2014 10:47:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=4WCTctv1rV01g+nilN6EpaCze3S1FJT3D1Vu8Y1npfk=; b=cXNCJ34k7JJoVD2VpxNLGhhMkyXYQjczjE0cJcm66ePR7zHKeS3zaO5oz1R+WJUjgn cGD5a/UwL2x/4skmHSQFDHBFrwlXXDXf9PnRKQ1Ff+ed2wADcEi4zAEc5w0Nwn9X0pf/ UwFCawPagLJ4feJbQfJzE9ZCw3IwBf9/3Lvb8eRSYF5lyxewLQeu126rGh+rhaitSWip H4NsbArXOagaoGk31oQIfSC5zvo1w8axcEzu3FjohUqKNTMOEP3MRx+ATEpj+K6toCWw 4Ukec7mE8tOwHRZjbrwm6XSeDsgYRylorC9HFaDdzI1cw8AxcFhHbJt2laagJvvsTK3E EATw== MIME-Version: 1.0 X-Received: by 10.224.53.137 with SMTP id m9mr20893065qag.66.1406828851550; Thu, 31 Jul 2014 10:47:31 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.140.30.230 with HTTP; Thu, 31 Jul 2014 10:47:31 -0700 (PDT) In-Reply-To: References: <20140730071933.GA20122@pc5.home> Date: Thu, 31 Jul 2014 10:47:31 -0700 X-Google-Sender-Auth: YFjgXg5PZibWk2qN0V8JQn4-OfA Message-ID: Subject: Re: [CFT] Autofs. From: Craig Rodrigues To: freebsd-current Current , freebsd-arch Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 17:47:32 -0000 On Wed, Jul 30, 2014 at 6:51 PM, Craig Rodrigues wrote: > On Wed, Jul 30, 2014 at 12:19 AM, Edward Tomasz Napiera=C5=82a < > trasz@freebsd.org> wrote: > >> At the link below you will find a patch that adds the new automounter. >> The patch is against yesterdays 11.0-CURRENT. >> >> http://people.freebsd.org/~trasz/autofs-head-20140729.diff >> >> > Hi, > > Nice stuff! I updated a current system, and tried the patch. > I put autofs_enable=3D"YES" in /etc/rc.conf > > In my network, I have a system, testhost with some NFS exports. > > /dir1/export1 > /dir1/export2 > /dir1/export3 > > I did: > > cd /net/testhost > > This operation seemed to timeout. > In dmesg, I saw these errors: > > WARNING: autofs_callout: request 0 for /net/ timed out after 30 seconds > WARNING: autofs_trigger_one: request for /net/ completed with error 60 > autofs_trigger: trigger failed with error 60; will retry in 1 seconds, = 2 > attempts left > WARNING: autofs_callout: request 1 for /net/ timed out after 30 seconds > WARNING: autofs_trigger_one: request for /net/ completed with error 60 > autofs_trigger: trigger failed with error 60; will retry in 1 seconds, = 1 > attempts left > WARNING: autofs_callout: request 2 for /net/ timed out after 30 seconds > WARNING: autofs_trigger_one: request for /net/ completed with error 60 > autofs_trigger: trigger failed 3 times; returning error 60 > > I used my Mac laptop in the same network, and did: > > cd /net/testhost > > and the command did not timeout, and I could do an ls and see that > /net/testhost/dir1/ was visible. > > Any ideas? > > This is cool stuff, and I hope you can get it into the 10.1 RELEASE. > > -- > Craig > > Hi, I cannot reproduce this problem at all. Now, when I do: cd /net/testhost it "just works", and I can access all the NFS exports. My previous problem might have been due to a temporary networking problem on my side. Yay! The only additional comment that I have for you is that the rc.conf variable is "autofs_enable", but the /etc/rc.d/ scripts are /etc/rc.d/automount and /etc/rc.d/automountd. It's not a big deal, but it would be nice if there was a single /etc/rc.d/autofs script to consolidate all this stuff, that would make it easier for admins to figure out where all the rc.d stuff related to autofs lives. Also, if your rc script introduced an autofs_flags (or automount_flags) variable, so that people could put the "-d" flag in there to enable more automount debugging, that would be nice. Other than that, I would say: Great work! I hope you can get this into 10.1-RELEASE! -- Craig From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 17:55:54 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1CEA4C1B; Thu, 31 Jul 2014 17:55:54 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D2CE021C4; Thu, 31 Jul 2014 17:55:53 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s6VHtlvG032115 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 31 Jul 2014 10:55:47 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s6VHtl0k032114; Thu, 31 Jul 2014 10:55:47 -0700 (PDT) (envelope-from jmg) Date: Thu, 31 Jul 2014 10:55:47 -0700 From: John-Mark Gurney To: Phil Shafer Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Message-ID: <20140731175547.GO43962@funkthat.com> Mail-Followup-To: Phil Shafer , arch@freebsd.org, sjg@freebsd.org, marcel@freebsd.org References: <20140730193819.GM43962@funkthat.com> <201407302324.s6UNOB2H087915@idle.juniper.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201407302324.s6UNOB2H087915@idle.juniper.net> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Thu, 31 Jul 2014 10:55:47 -0700 (PDT) Cc: sjg@freebsd.org, arch@freebsd.org, marcel@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 17:55:54 -0000 Phil Shafer wrote this message on Wed, Jul 30, 2014 at 19:24 -0400: > John-Mark Gurney writes: > >My vote would be to use and *enforce* UTF-8 by the API. That means if > >someone passes a string in, it must be properly formed UTF-8... > > I can certainly see making this an option, detecting the high-bit > and inspecting the following 1-5 bytes to ensure the corresponding > high two bits are set appropriately. But what action would you > expect the library to take when invalid strings are passed in? Return an error? printf can return an error, yet most people don't check it.. so no real difference in API/bugs... The reason I even suggest this is that JSON requires the output to be in Unicode... Not some special locale encoding.. See section 3 of: https://www.ietf.org/rfc/rfc4627.txt Besides we should finally move to UTF-8 for file system and other parts of the system... I do like the idea of random binary filenames, but we really should stop sticking our head in the sand.. We will only make ourselves look silly when 2020 roles around if we don't... > libxo supports a warning flag, that will trigger warnings on stderr > for things like invalid or malformed format strings, but I'm not > sure I'd be happy if the library skipped invalid strings. printf may skip parts of your strings if you don't check it's return value... Plus, if the API states you must pass in UTF-8 strings, and someone doesn't properly encode/convert to UTF-8, it's their bug, not the library's bug... We have too many encoding issues already in our source tree, and we need to get better about making sure we don't have them, and this will help... > BTW, this issue is driven by "w"s use of wide characters (for > days of the week). Plus, enforcing UTF-8 will make the w versions easier, and allow the library to output other width of UTF if wanted/requested.. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 18:17:03 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A1DC34B2; Thu, 31 Jul 2014 18:17:03 +0000 (UTC) Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2lp0204.outbound.protection.outlook.com [207.46.163.204]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2F69424D2; Thu, 31 Jul 2014 18:17:02 +0000 (UTC) Received: from BY2PR05CA044.namprd05.prod.outlook.com (10.141.250.34) by BLUPR05MB723.namprd05.prod.outlook.com (10.141.207.153) with Microsoft SMTP Server (TLS) id 15.0.995.14; Thu, 31 Jul 2014 18:16:59 +0000 Received: from BN1BFFO11FD049.protection.gbl (2a01:111:f400:7c10::1:167) by BY2PR05CA044.outlook.office365.com (2a01:111:e400:2c5f::34) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Thu, 31 Jul 2014 18:16:58 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BN1BFFO11FD049.mail.protection.outlook.com (10.58.145.4) with Microsoft SMTP Server (TLS) id 15.0.990.10 via Frontend Transport; Thu, 31 Jul 2014 18:16:58 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Thu, 31 Jul 2014 11:16:57 -0700 Received: from idle.juniper.net (idleski.juniper.net [172.25.4.26]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6VIGmn45624; Thu, 31 Jul 2014 11:16:48 -0700 (PDT) (envelope-from phil@juniper.net) Received: from idle.juniper.net (localhost [127.0.0.1]) by idle.juniper.net (8.14.4/8.14.3) with ESMTP id s6VIGR07096191; Thu, 31 Jul 2014 14:16:31 -0400 (EDT) (envelope-from phil@idle.juniper.net) Message-ID: <201407311816.s6VIGR07096191@idle.juniper.net> To: Garance A Drosehn Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <08B4AE10-1747-4BB1-AA1A-BA4491A57F95@rpi.edu> Date: Thu, 31 Jul 2014 14:16:27 -0400 From: Phil Shafer MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(164054003)(199002)(189002)(69596002)(105596002)(103666002)(46102001)(31966008)(21056001)(74662001)(74502001)(77982001)(79102001)(76482001)(84676001)(110136001)(6806004)(68736004)(44976005)(85306003)(95666004)(47776003)(102836001)(106466001)(48376002)(53416004)(86362001)(92726001)(97736001)(85852003)(64706001)(83322001)(99396002)(81342001)(81542001)(83072002)(50986999)(76506005)(92566001)(4396001)(50466002)(54356999)(87936001)(107046002)(81156004)(80022001)(2171001)(20776003); DIR:OUT; SFP:; SCL:1; SRVR:BLUPR05MB723; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0289B6431E Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=phil@juniper.net; X-OriginatorOrg: juniper.net Cc: arch@freebsd.org, marcel@freebsd.org, "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 18:17:03 -0000 Garance A Drosehn writes: >What if there's something you want to print out for xml/json but >*not* for plain-text? (it's easy to imagine some commands might >print out more values when they are not constrained by an 80-char >width). The "d" flag instructs libxo so only emit for "display" formats (TEXT, HTML) and the "e" flag for "encode" formats (XML, JSON). xo_emit("{d:count}{e:count}", "fourth", "4"); >Also, given that machine-readable outputs might change over time, >is there the idea of including a version-number with the output >for each command? I imagine at some point in the future we could see YANG models for this output, with revision strings and more. In practice, this turns out to be a minor issue, if strong discipline is maintained. Future versions can emit new tags, but not remove old tags. So a script that uses an XPath expression like: user-entry[user == 'phil' && idle/@seconds > 6000] it should work forever. XML becomes an API, where your API makes a contract saying "I won't change" and deprecating XML elements becomes as rare as deprecating functions in libc. >I hope it doesn't seem like I'm just complaining about the work >everyone else is doing (both this and the GSOC project). Please >note that I've attempted to do this a few times myself, and I haven't >been happy with any of *my* attempts either! I'm been kicking this idea around in my head for years. The first XML API in JUNOS was 2001. But the way we do this in JUNOS is not a good fit for BSD; we emit XML from our daemons and the CLI uses an output definition language to know how to turn XML into text. BSD needs point-of-creation content generation flexibility, and some scheme for the shell/parent to tell the what sort of data should be generated. I'm using the LIBXO_OPTIONS variable to do this, so your RESTful API daemon can set this env var, run the command, and funnel the output back to the client. There needs to be a mechanism for marking the executable to say which types of data formats are support, or if libxo is supported at all. Thanks, Phil From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 18:40:33 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AB508B31; Thu, 31 Jul 2014 18:40:33 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (dns-bn1lp0143.outbound.protection.outlook.com [207.46.163.143]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9196026A8; Thu, 31 Jul 2014 18:40:31 +0000 (UTC) Received: from CO2PR05CA033.namprd05.prod.outlook.com (10.141.241.161) by DM2PR05MB734.namprd05.prod.outlook.com (10.141.178.22) with Microsoft SMTP Server (TLS) id 15.0.995.14; Thu, 31 Jul 2014 18:40:29 +0000 Received: from BY2FFO11FD028.protection.gbl (2a01:111:f400:7c0c::115) by CO2PR05CA033.outlook.office365.com (2a01:111:e400:1429::33) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Thu, 31 Jul 2014 18:40:28 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BY2FFO11FD028.mail.protection.outlook.com (10.1.15.217) with Microsoft SMTP Server (TLS) id 15.0.990.10 via Frontend Transport; Thu, 31 Jul 2014 18:40:28 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Thu, 31 Jul 2014 11:40:28 -0700 Received: from idle.juniper.net (idleski.juniper.net [172.25.4.26]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6VIeAn61811; Thu, 31 Jul 2014 11:40:22 -0700 (PDT) (envelope-from phil@juniper.net) Received: from idle.juniper.net (localhost [127.0.0.1]) by idle.juniper.net (8.14.4/8.14.3) with ESMTP id s6VIdlMK096434; Thu, 31 Jul 2014 14:39:54 -0400 (EDT) (envelope-from phil@idle.juniper.net) Message-ID: <201407311839.s6VIdlMK096434@idle.juniper.net> To: John-Mark Gurney Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <20140731175547.GO43962@funkthat.com> Date: Thu, 31 Jul 2014 14:39:47 -0400 From: Phil Shafer MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(164054003)(189002)(199002)(77982001)(99396002)(106466001)(46102001)(110136001)(107046002)(21056001)(84676001)(81156004)(48376002)(50466002)(87936001)(44976005)(79102001)(76506005)(81342001)(86362001)(20776003)(92566001)(103666002)(95666004)(105596002)(74502001)(6806004)(85852003)(92726001)(83072002)(97736001)(31966008)(47776003)(81542001)(83322001)(54356999)(76482001)(80022001)(53416004)(4396001)(69596002)(68736004)(102836001)(85306003)(50986999); DIR:OUT; SFP:; SCL:1; SRVR:DM2PR05MB734; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0289B6431E Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=phil@juniper.net; X-OriginatorOrg: juniper.net Cc: sjg@freebsd.org, arch@freebsd.org, marcel@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 18:40:33 -0000 John-Mark Gurney writes: >Return an error? printf can return an error, yet most people don't >check it.. so no real difference in API/bugs... My concern is emitting half a string, where the half we don't emit is something important. I don't want to make the opposite of an injection attack, where arranging some daemon to call xo_emit with a broken UTF-8 string allows an evil-doer to fix their evil content into the other half of the string. I'm escaping XML, JSON, and HTML content already, so the simplest scheme is to: a) UTF-8 check the format string; if it fails, nothing is emitted b) for each format descriptor, check the content generared; if it fails, nothing is emitted from the xo_emit call anything already generated is discarded Simple and easy. Seem reasonable? The other option would be to discard only that specific format descriptor or only that field description. xo_emit("{:good/%d}{:bad/%d%s}{:ugly}", 0, 55, "\xff\x01\xff", "cat"); Does the "cat" get emitted? Is "55" emitted? If "ugly" was phil, and the bogus string blocked the generation of that vital bit of info, life could be bad. Unfortunately, even this isn't a simple fix for "w", which wants call wcsftime() to get wide values for month and day-of-the-week names. Does wcsrtombs() convert this to UTF-8? Is there a locale for UTF-8? Thanks, Phil From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 20:07:41 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0A5AA45B; Thu, 31 Jul 2014 20:07:41 +0000 (UTC) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id C3FD3211D; Thu, 31 Jul 2014 20:07:40 +0000 (UTC) Received: from critter.freebsd.dk (unknown [192.168.48.2]) by phk.freebsd.dk (Postfix) with ESMTP id EF9011578; Thu, 31 Jul 2014 20:07:38 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.14.9/8.14.9) with ESMTP id s6VK7b9K058088; Thu, 31 Jul 2014 20:07:38 GMT (envelope-from phk@phk.freebsd.dk) To: John-Mark Gurney Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-reply-to: <20140731175547.GO43962@funkthat.com> From: "Poul-Henning Kamp" References: <20140730193819.GM43962@funkthat.com> <201407302324.s6UNOB2H087915@idle.juniper.net> <20140731175547.GO43962@funkthat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <58086.1406837257.1@critter.freebsd.dk> Date: Thu, 31 Jul 2014 20:07:37 +0000 Message-ID: <58087.1406837257@critter.freebsd.dk> Cc: sjg@freebsd.org, arch@freebsd.org, marcel@freebsd.org, Phil Shafer X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 20:07:41 -0000 -------- In message <20140731175547.GO43962@funkthat.com>, John-Mark Gurney writes: >Phil Shafer wrote this message on Wed, Jul 30, 2014 at 19:24 -0400: >> John-Mark Gurney writes: >> >My vote would be to use and *enforce* UTF-8 by the API. That means if >> >someone passes a string in, it must be properly formed UTF-8... >> >> I can certainly see making this an option, detecting the high-bit >> and inspecting the following 1-5 bytes to ensure the corresponding >> high two bits are set appropriately. But what action would you >> expect the library to take when invalid strings are passed in? > >Return an error? printf can return an error, yet most people don't >check it.. so no real difference in API/bugs... This is why we ended up with SIGPIPE in the first place. Can I point discreetely at sbuf(3)'s accumulative error handling and suggest that libxo does something similar ? That way applications only need to check for errors once, rather than after every single call to every single function in the libxo library. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 20:50:57 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 329CB568; Thu, 31 Jul 2014 20:50:57 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E42B325ED; Thu, 31 Jul 2014 20:50:56 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s6VKot8Z034481 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 31 Jul 2014 13:50:55 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s6VKosKB034480; Thu, 31 Jul 2014 13:50:54 -0700 (PDT) (envelope-from jmg) Date: Thu, 31 Jul 2014 13:50:54 -0700 From: John-Mark Gurney To: Phil Shafer Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Message-ID: <20140731205054.GT43962@funkthat.com> Mail-Followup-To: Phil Shafer , Poul-Henning Kamp , sjg@freebsd.org, arch@freebsd.org, marcel@freebsd.org References: <94841.1406796243@critter.freebsd.dk> <201407311531.s6VFVbJn094888@idle.juniper.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201407311531.s6VFVbJn094888@idle.juniper.net> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Thu, 31 Jul 2014 13:50:56 -0700 (PDT) Cc: sjg@freebsd.org, arch@freebsd.org, Poul-Henning Kamp , marcel@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 20:50:57 -0000 Phil Shafer wrote this message on Thu, Jul 31, 2014 at 11:31 -0400: > Poul-Henning Kamp writes: > >The real question is how it will be perceived if we are *not* actively > >going UTF-8 by that time? > > And moving toward UTF-8 won't be simple. I just tossed a couple If we don't start, we won't ever move forward... > of file from http://www.cl.cam.ac.uk/~mgk25/ucs/examples/ into a > few apps that are supposed to support UTF-8 (emacs, vim, firefox) Why not nvi? :) I just tried out nvi w/ UTF-8-demo.txt, and was surprised that it worked well.. there are issues w/ combining characters, both stargate and Thai, but that's an issue w/ nvi escaping those characters instead of displaying them.. my terminal was Terminal.app from MacOSX... less and more appear to work, and handle the combining characters properly... vt works, but we need much better font support.. The default font is missing lots of math, linguistic, APL, georgian, Thai, Amharic, runes (though MacOS's font I'm using misses these too), Braille (this should be easy for anyone to add), and I believe Japanese... > and the results were underwhelming. Skimming the description of > xterm support for UTF-8 gives some appreciation for how complex > this crap is (http://www.cl.cam.ac.uk/~mgk25/unicode.html#xterm). I must say, from my brief test, I'm surprised it worked as well as it did... :) -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 20:58:34 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A5CEFA02; Thu, 31 Jul 2014 20:58:34 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 64D2826DB; Thu, 31 Jul 2014 20:58:33 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s6VKwXOh034612 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 31 Jul 2014 13:58:33 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s6VKwXG5034611; Thu, 31 Jul 2014 13:58:33 -0700 (PDT) (envelope-from jmg) Date: Thu, 31 Jul 2014 13:58:33 -0700 From: John-Mark Gurney To: Poul-Henning Kamp Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Message-ID: <20140731205833.GU43962@funkthat.com> Mail-Followup-To: Poul-Henning Kamp , Phil Shafer , sjg@freebsd.org, arch@freebsd.org, marcel@freebsd.org References: <20140730193819.GM43962@funkthat.com> <201407302324.s6UNOB2H087915@idle.juniper.net> <20140731175547.GO43962@funkthat.com> <58087.1406837257@critter.freebsd.dk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <58087.1406837257@critter.freebsd.dk> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Thu, 31 Jul 2014 13:58:33 -0700 (PDT) Cc: sjg@freebsd.org, arch@freebsd.org, marcel@freebsd.org, Phil Shafer X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 20:58:34 -0000 Poul-Henning Kamp wrote this message on Thu, Jul 31, 2014 at 20:07 +0000: > -------- > In message <20140731175547.GO43962@funkthat.com>, John-Mark Gurney writes: > >Phil Shafer wrote this message on Wed, Jul 30, 2014 at 19:24 -0400: > >> John-Mark Gurney writes: > >> >My vote would be to use and *enforce* UTF-8 by the API. That means if > >> >someone passes a string in, it must be properly formed UTF-8... > >> > >> I can certainly see making this an option, detecting the high-bit > >> and inspecting the following 1-5 bytes to ensure the corresponding > >> high two bits are set appropriately. But what action would you > >> expect the library to take when invalid strings are passed in? > > > >Return an error? printf can return an error, yet most people don't > >check it.. so no real difference in API/bugs... > > This is why we ended up with SIGPIPE in the first place. > > Can I point discreetely at sbuf(3)'s accumulative error handling > and suggest that libxo does something similar ? That way applications $ man 3 sbuf No manual entry for sbuf And looks like it isn't in libc: $ cc -o q q.c /tmp/q-41cfbe.o: In function `main': q.c:(.text+0x23): undefined reference to `sbuf_new' q.c:(.text+0x38): undefined reference to `sbuf_cat' q.c:(.text+0x4c): undefined reference to `sbuf_cat' q.c:(.text+0x58): undefined reference to `sbuf_finish' q.c:(.text+0x64): undefined reference to `sbuf_data' q.c:(.text+0x82): undefined reference to `sbuf_delete' cc: error: linker command failed with exit code 1 (use -v to see invocation) Hmm... looks like libsbuf exists, and causes the above program to compile and work... Please write a man page for sbuf(3)... Thanks. > only need to check for errors once, rather than after every single > call to every single function in the libxo library. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 21:01:22 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EA710C9C; Thu, 31 Jul 2014 21:01:22 +0000 (UTC) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id ACD832786; Thu, 31 Jul 2014 21:01:22 +0000 (UTC) Received: from critter.freebsd.dk (unknown [192.168.48.2]) by phk.freebsd.dk (Postfix) with ESMTP id 88A0616D2; Thu, 31 Jul 2014 21:01:21 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.14.9/8.14.9) with ESMTP id s6VL1LVT058440; Thu, 31 Jul 2014 21:01:21 GMT (envelope-from phk@phk.freebsd.dk) To: John-Mark Gurney Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-reply-to: <20140731205054.GT43962@funkthat.com> From: "Poul-Henning Kamp" References: <94841.1406796243@critter.freebsd.dk> <201407311531.s6VFVbJn094888@idle.juniper.net> <20140731205054.GT43962@funkthat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <58438.1406840481.1@critter.freebsd.dk> Date: Thu, 31 Jul 2014 21:01:21 +0000 Message-ID: <58439.1406840481@critter.freebsd.dk> Cc: sjg@freebsd.org, arch@freebsd.org, marcel@freebsd.org, Phil Shafer X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 21:01:23 -0000 -------- In message <20140731205054.GT43962@funkthat.com>, John-Mark Gurney writes: >Phil Shafer wrote this message on Thu, Jul 31, 2014 at 11:31 -0400: >Why not nvi? :) I just tried out nvi w/ UTF-8-demo.txt, and was >surprised that it worked well.. Having seen peter brave the wemm-field and go utf-8 I recently converted too and most stuff seems to work without trouble. What doesn't work with nvi is that it *recently* started mangling files with invalid byte sequences no matter what charset you set it for except 'C'. I write recently, because I'm quite sure it started in the last year or so, before that it did the right thing. You do get a warning on reading in the file if there are conversion issues, but I really think nvi should be able to edit any file, no matter what charset you have set for it, and invalid byte sequences should just show up as hex-expansions like they do in 'C' mode. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 21:01:54 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 54342D46; Thu, 31 Jul 2014 21:01:54 +0000 (UTC) Received: from mail.crittercasa.com (mail.turbofuzz.com [208.87.221.144]) by mx1.freebsd.org (Postfix) with ESMTP id 3358E278C; Thu, 31 Jul 2014 21:01:53 +0000 (UTC) Received: from [10.2.0.80] (unknown [69.198.165.132]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.crittercasa.com (Postfix) with ESMTPS id E05DA1648DE; Thu, 31 Jul 2014 14:01:52 -0700 (PDT) Mime-Version: 1.0 (Mac OS X Mail 8.0 \(1971.5\)) Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Jordan Hubbard In-Reply-To: Date: Thu, 31 Jul 2014 14:02:12 -0700 Message-Id: <7DBE2572-5803-4D0B-A48F-D983785090D8@turbofuzz.com> References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <82CFA67F-BA93-44EE-BD4B-9105F89AD157@rpi.edu> To: Tim Kientzle X-Mailer: Apple Mail (2.1971.5) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: marcel@freebsd.org, phil@juniper.net, Simon Gerraty , arch@freebsd.org, sjg@freebsd.org, Garance A Drosehn X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 21:01:54 -0000 > On Jul 30, 2014, at 7:09 PM, Tim Kientzle wrote: >=20 >> I kept meaning to see if there was any difference between them. I = just noticed that the rfc has the =91Z' suffix as an option for a = timezone, and I don't think that the ISO one does. >=20 > Both allow =91Z=92.=20 That will keep us pilots happy. ;-) - Jordan From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 21:06:09 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 65719E68; Thu, 31 Jul 2014 21:06:09 +0000 (UTC) Received: from mail.crittercasa.com (mail.turbofuzz.com [208.87.221.144]) by mx1.freebsd.org (Postfix) with ESMTP id 4D96527B7; Thu, 31 Jul 2014 21:06:08 +0000 (UTC) Received: from [10.2.0.80] (unknown [69.198.165.132]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.crittercasa.com (Postfix) with ESMTPS id 8B73E164882; Thu, 31 Jul 2014 14:00:21 -0700 (PDT) Mime-Version: 1.0 (Mac OS X Mail 8.0 \(1971.5\)) Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Jordan Hubbard In-Reply-To: <20140731205054.GT43962@funkthat.com> Date: Thu, 31 Jul 2014 14:00:41 -0700 Message-Id: <07AA5159-16A2-45D4-890D-4FB794F99B79@turbofuzz.com> References: <94841.1406796243@critter.freebsd.dk> <201407311531.s6VFVbJn094888@idle.juniper.net> <20140731205054.GT43962@funkthat.com> To: John-Mark Gurney X-Mailer: Apple Mail (2.1971.5) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: sjg@freebsd.org, arch@freebsd.org, Poul-Henning Kamp , marcel@freebsd.org, Phil Shafer X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 21:06:09 -0000 > On Jul 31, 2014, at 1:50 PM, John-Mark Gurney = wrote: >=20 >> And moving toward UTF-8 won't be simple. I just tossed a couple >=20 > If we don't start, we won't ever move forward... Amen. If you look at $LANG on an OS X box, it=E2=80=99s been set to = en_US.UTF-8 for awhile, and that wasn=E2=80=99t without pain. We had to = deal with various performance regressions (8x hit to grep(1) over = ISO-Latin1!) and various other interoperability problems, but it was = ultimately like ripping off a band-aid - better done quickly than = slowly, and once done, the constant trickle of I18N bugs more or less = shut off entirely. To put it another way, if FreeBSD doesn=E2=80=99t do it, its downstream = vendors will have to. Everything from file names to filesystem names = need to be UTF-8 just so they display properly and Japanese users can = put their files on /mnt/=E3=83=9B=E3=83=BC=E3=83=A0/=E7=A7=81=E3=81=AE=E4=BC= =9A=E7=A4=BE/ Downstream here in FreeNAS-land, we get I18N requests = all the time, and it=E2=80=99s sure a pita when you have the various = technologies all *mostly* able to speak UTF-8 but there are routines = here and there that just don=E2=80=99t, and trying to make everything = work from a user land agent like Samba or Netatalk all the way down the = stack is just a PITA, but we still have to do it because it=E2=80=99s = not an all-english-all-the-time world we live in. I=E2=80=99d also not even worry about wide characters. They are a = historical artifact and not the direction everyone is going in. UTF-8 = FTW. - Jordan From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 21:09:39 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 24285FDE; Thu, 31 Jul 2014 21:09:39 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id F043D27F7; Thu, 31 Jul 2014 21:09:38 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s6VL9bDd034796 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 31 Jul 2014 14:09:38 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s6VL9bSc034795; Thu, 31 Jul 2014 14:09:37 -0700 (PDT) (envelope-from jmg) Date: Thu, 31 Jul 2014 14:09:37 -0700 From: John-Mark Gurney To: Phil Shafer Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Message-ID: <20140731210937.GV43962@funkthat.com> Mail-Followup-To: Phil Shafer , arch@freebsd.org, sjg@freebsd.org, marcel@freebsd.org References: <20140731175547.GO43962@funkthat.com> <201407311839.s6VIdlMK096434@idle.juniper.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201407311839.s6VIdlMK096434@idle.juniper.net> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Thu, 31 Jul 2014 14:09:38 -0700 (PDT) Cc: sjg@freebsd.org, arch@freebsd.org, marcel@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 21:09:39 -0000 Phil Shafer wrote this message on Thu, Jul 31, 2014 at 14:39 -0400: > John-Mark Gurney writes: > >Return an error? printf can return an error, yet most people don't > >check it.. so no real difference in API/bugs... > > My concern is emitting half a string, where the half we don't emit > is something important. I don't want to make the opposite of an > injection attack, where arranging some daemon to call xo_emit with > a broken UTF-8 string allows an evil-doer to fix their evil content > into the other half of the string. > > I'm escaping XML, JSON, and HTML content already, so the simplest > scheme is to: > > a) UTF-8 check the format string; > if it fails, nothing is emitted > b) for each format descriptor, check the content generared; > if it fails, nothing is emitted from the xo_emit call > anything already generated is discarded > > Simple and easy. Seem reasonable? The other option would be to > discard only that specific format descriptor or only that field > description. > > xo_emit("{:good/%d}{:bad/%d%s}{:ugly}", 0, 55, "\xff\x01\xff", "cat"); > > Does the "cat" get emitted? Is "55" emitted? > > If "ugly" was phil, and the bogus > string blocked the generation of that vital bit of info, life could > be bad. I agree... > Unfortunately, even this isn't a simple fix for "w", which wants > call wcsftime() to get wide values for month and day-of-the-week > names. Does wcsrtombs() convert this to UTF-8? Is there a locale > for UTF-8? Well, from my understanding there can't be a "locale" that is UTF-8 as a locale contains more than just character encoding... It also includes month/day names, sorting, etc... I think you can get a C locale (the default) w/ UTF-8 by setting the correct environment variables, but I don't know them well enough to say... Should we add a locale that does this? There is UTF-8 in /usr/share/locale, but if you set LANG to it, things don't work.. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 21:18:22 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8119535C; Thu, 31 Jul 2014 21:18:22 +0000 (UTC) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id 435B628EA; Thu, 31 Jul 2014 21:18:21 +0000 (UTC) Received: from critter.freebsd.dk (unknown [192.168.48.2]) by phk.freebsd.dk (Postfix) with ESMTP id 4F6C31578; Thu, 31 Jul 2014 21:18:20 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.14.9/8.14.9) with ESMTP id s6VLIJoA058557; Thu, 31 Jul 2014 21:18:19 GMT (envelope-from phk@phk.freebsd.dk) To: John-Mark Gurney Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-reply-to: <20140731205833.GU43962@funkthat.com> From: "Poul-Henning Kamp" References: <20140730193819.GM43962@funkthat.com> <201407302324.s6UNOB2H087915@idle.juniper.net> <20140731175547.GO43962@funkthat.com> <58087.1406837257@critter.freebsd.dk> <20140731205833.GU43962@funkthat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <58555.1406841499.1@critter.freebsd.dk> Date: Thu, 31 Jul 2014 21:18:19 +0000 Message-ID: <58556.1406841499@critter.freebsd.dk> Cc: sjg@freebsd.org, arch@freebsd.org, marcel@freebsd.org, Phil Shafer X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 21:18:22 -0000 -------- In message <20140731205833.GU43962@funkthat.com>, John-Mark Gurney writes: >$ man 3 sbuf >No manual entry for sbuf Sorry: sbuf(9) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 21:31:24 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 637D4856; Thu, 31 Jul 2014 21:31:24 +0000 (UTC) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2lp0238.outbound.protection.outlook.com [207.46.163.238]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BF0D52A17; Thu, 31 Jul 2014 21:31:22 +0000 (UTC) Received: from BLUPR05CA006.namprd05.prod.outlook.com (10.255.219.164) by CO2PR05MB729.namprd05.prod.outlook.com (10.141.228.12) with Microsoft SMTP Server (TLS) id 15.0.995.14; Thu, 31 Jul 2014 21:31:20 +0000 Received: from BY2FFO11FD050.protection.gbl (2a01:111:f400:7c0c::138) by BLUPR05CA006.outlook.office365.com (2a01:111:e400:83f::36) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Thu, 31 Jul 2014 21:31:19 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BY2FFO11FD050.mail.protection.outlook.com (10.1.15.187) with Microsoft SMTP Server (TLS) id 15.0.990.10 via Frontend Transport; Thu, 31 Jul 2014 21:31:19 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Thu, 31 Jul 2014 14:31:12 -0700 Received: from idle.juniper.net (idleski.juniper.net [172.25.4.26]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6VLUqn85662; Thu, 31 Jul 2014 14:31:09 -0700 (PDT) (envelope-from phil@juniper.net) Received: from idle.juniper.net (localhost [127.0.0.1]) by idle.juniper.net (8.14.4/8.14.3) with ESMTP id s6VLUFSP097778; Thu, 31 Jul 2014 17:30:24 -0400 (EDT) (envelope-from phil@idle.juniper.net) Message-ID: <201407312130.s6VLUFSP097778@idle.juniper.net> To: Poul-Henning Kamp Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <58087.1406837257@critter.freebsd.dk> Date: Thu, 31 Jul 2014 17:30:15 -0400 From: Phil Shafer MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(189002)(199002)(164054003)(77982001)(99396002)(54356999)(107046002)(102836001)(20776003)(44976005)(50466002)(83072002)(47776003)(69596002)(46102001)(85852003)(86362001)(92566001)(68736004)(92726001)(87936001)(83322001)(50986999)(76482001)(48376002)(4396001)(84676001)(6806004)(81342001)(21056001)(97736001)(81542001)(53416004)(95666004)(85306003)(106466001)(110136001)(31966008)(79102001)(103666002)(76506005)(105596002)(74502001)(80022001)(81156004); DIR:OUT; SFP:; SCL:1; SRVR:CO2PR05MB729; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0289B6431E Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=phil@juniper.net; X-OriginatorOrg: juniper.net Cc: sjg@freebsd.org, arch@freebsd.org, John-Mark Gurney , marcel@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 21:31:24 -0000 Poul-Henning Kamp writes: >Can I point discreetely at sbuf(3)'s accumulative error handling >and suggest that libxo does something similar ? That way applications >only need to check for errors once, rather than after every single >call to every single function in the libxo library. sbuf looks like a simple case, returning either ENOMEM or the error code from the flush function. libxo can keep a "there's been an error" flag that the user can retrieve, but all the details of what's gone wrong would be lost. Or it can buffer the contents of warning messages and deliver it to the caller. Currently you need to turn on the per-handle XOF_WARN flag to get warnings displayed; perhaps that's enough. Thanks, Phil From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 21:32:45 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 55B1691A; Thu, 31 Jul 2014 21:32:45 +0000 (UTC) Received: from mail.crittercasa.com (mail.turbofuzz.com [208.87.221.144]) by mx1.freebsd.org (Postfix) with ESMTP id 35EE82AB0; Thu, 31 Jul 2014 21:32:44 +0000 (UTC) Received: from [10.2.0.80] (unknown [69.198.165.132]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.crittercasa.com (Postfix) with ESMTPS id 630C41648DE; Thu, 31 Jul 2014 14:32:44 -0700 (PDT) Mime-Version: 1.0 (Mac OS X Mail 8.0 \(1971.5\)) Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Jordan Hubbard In-Reply-To: <20140731210937.GV43962@funkthat.com> Date: Thu, 31 Jul 2014 14:33:03 -0700 Message-Id: <92A1A2E4-6154-460F-8682-A39F0EB6198C@turbofuzz.com> References: <20140731175547.GO43962@funkthat.com> <201407311839.s6VIdlMK096434@idle.juniper.net> <20140731210937.GV43962@funkthat.com> To: John-Mark Gurney X-Mailer: Apple Mail (2.1971.5) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: sjg@freebsd.org, arch@freebsd.org, marcel@freebsd.org, Phil Shafer X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 21:32:45 -0000 > On Jul 31, 2014, at 2:09 PM, John-Mark Gurney wrote: > > Well, from my understanding there can't be a "locale" that is UTF-8 > as a locale contains more than just character encoding... It also > includes month/day names, sorting, etc... I think you can get a > C locale (the default) w/ UTF-8 by setting the correct environment > variables, but I don't know them well enough to say... Should we add > a locale that does this? There is UTF-8 in /usr/share/locale, but if > you set LANG to it, things don't work.. en_US.UTF-8 From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 21:36:45 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C61EFA40; Thu, 31 Jul 2014 21:36:45 +0000 (UTC) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id 8B16B2ADD; Thu, 31 Jul 2014 21:36:45 +0000 (UTC) Received: from critter.freebsd.dk (unknown [192.168.48.2]) by phk.freebsd.dk (Postfix) with ESMTP id 5FCC71578; Thu, 31 Jul 2014 21:36:44 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.14.9/8.14.9) with ESMTP id s6VLahku058666; Thu, 31 Jul 2014 21:36:43 GMT (envelope-from phk@phk.freebsd.dk) To: Phil Shafer Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-reply-to: <201407312130.s6VLUFSP097778@idle.juniper.net> From: "Poul-Henning Kamp" References: <201407312130.s6VLUFSP097778@idle.juniper.net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <58664.1406842603.1@critter.freebsd.dk> Date: Thu, 31 Jul 2014 21:36:43 +0000 Message-ID: <58665.1406842603@critter.freebsd.dk> Cc: sjg@freebsd.org, arch@freebsd.org, John-Mark Gurney , marcel@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 21:36:45 -0000 -------- In message <201407312130.s6VLUFSP097778@idle.juniper.net>, Phil Shafer writes: >Poul-Henning Kamp writes: >>Can I point discreetely at sbuf(3)'s accumulative error handling >>and suggest that libxo does something similar ? That way applications >>only need to check for errors once, rather than after every single >>call to every single function in the libxo library. > >sbuf looks like a simple case, returning either ENOMEM or the >error code from the flush function. libxo can keep a "there's >been an error" flag that the user can retrieve, but all the >details of what's gone wrong would be lost. Or it can buffer >the contents of warning messages and deliver it to the caller. We can afford to dedicate a buffer of a reasonable size for that purpose if we need to. The point here is one of API design, and experience has shown that either error-handling is convenient or it doesn't happen. I don't see the libxo case being any different from sbuf in this respect, in fact I see it being almost even more important because the readers are non-humans. libxo should latch on error like libsbuf, and valid output should only be emitted if no errors were encountered during production. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 21:41:13 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 18E01B50; Thu, 31 Jul 2014 21:41:13 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1lp0144.outbound.protection.outlook.com [207.46.163.144]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EF3FD2AFF; Thu, 31 Jul 2014 21:41:11 +0000 (UTC) Received: from BLUPR05CA0076.namprd05.prod.outlook.com (10.141.20.46) by BY2PR05MB726.namprd05.prod.outlook.com (10.141.223.19) with Microsoft SMTP Server (TLS) id 15.0.995.14; Thu, 31 Jul 2014 21:41:02 +0000 Received: from BN1AFFO11FD045.protection.gbl (2a01:111:f400:7c10::144) by BLUPR05CA0076.outlook.office365.com (2a01:111:e400:855::46) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Thu, 31 Jul 2014 21:41:01 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BN1AFFO11FD045.mail.protection.outlook.com (10.58.53.60) with Microsoft SMTP Server (TLS) id 15.0.990.10 via Frontend Transport; Thu, 31 Jul 2014 21:41:00 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Thu, 31 Jul 2014 14:40:58 -0700 Received: from idle.juniper.net (idleski.juniper.net [172.25.4.26]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6VLejn91464; Thu, 31 Jul 2014 14:40:56 -0700 (PDT) (envelope-from phil@juniper.net) Received: from idle.juniper.net (localhost [127.0.0.1]) by idle.juniper.net (8.14.4/8.14.3) with ESMTP id s6VLeTS4097915; Thu, 31 Jul 2014 17:40:32 -0400 (EDT) (envelope-from phil@idle.juniper.net) Message-ID: <201407312140.s6VLeTS4097915@idle.juniper.net> To: John-Mark Gurney Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <20140731210937.GV43962@funkthat.com> Date: Thu, 31 Jul 2014 17:40:29 -0400 From: Phil Shafer MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(189002)(164054003)(199002)(6806004)(80022001)(44976005)(87936001)(81156004)(21056001)(95666004)(79102001)(47776003)(83322001)(76506005)(85306003)(20776003)(76482001)(4396001)(105596002)(77982001)(74502001)(107046002)(102836001)(97736001)(53416004)(103666002)(110136001)(81342001)(92566001)(54356999)(106466001)(85852003)(69596002)(31966008)(48376002)(81542001)(50466002)(50986999)(92726001)(68736004)(86362001)(99396002)(46102001)(83072002)(84676001); DIR:OUT; SFP:; SCL:1; SRVR:BY2PR05MB726; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0289B6431E Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=phil@juniper.net; X-OriginatorOrg: juniper.net Cc: sjg@freebsd.org, arch@freebsd.org, marcel@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 21:41:13 -0000 John-Mark Gurney writes: >Well, from my understanding there can't be a "locale" that is UTF-8 >as a locale contains more than just character encoding... It also >includes month/day names, sorting, etc... I think you can get a >C locale (the default) w/ UTF-8 by setting the correct environment >variables, but I don't know them well enough to say... Should we add >a locale that does this? There is UTF-8 in /usr/share/locale, but if >you set LANG to it, things don't work.. I'll change the library to follow the settings of the user's env var and assuming they've set it correctly, all will work well. Since libxo uses vsnprintf under the covers, all this should work fine. (void)wcsftime(buf, sizeof(buf), fmt, &tp); ... xo_emit("{:login-time/%ls}", buf); Or something like that...... Thanks, Phil From owner-freebsd-arch@FreeBSD.ORG Thu Jul 31 21:42:46 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DE2A2C18; Thu, 31 Jul 2014 21:42:46 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id B40E22B86; Thu, 31 Jul 2014 21:42:46 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s6VLgjk2035361 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 31 Jul 2014 14:42:46 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s6VLgj0F035360; Thu, 31 Jul 2014 14:42:45 -0700 (PDT) (envelope-from jmg) Date: Thu, 31 Jul 2014 14:42:45 -0700 From: John-Mark Gurney To: Poul-Henning Kamp Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Message-ID: <20140731214245.GW43962@funkthat.com> Mail-Followup-To: Poul-Henning Kamp , sjg@freebsd.org, arch@freebsd.org, marcel@freebsd.org, Phil Shafer References: <20140730193819.GM43962@funkthat.com> <201407302324.s6UNOB2H087915@idle.juniper.net> <20140731175547.GO43962@funkthat.com> <58087.1406837257@critter.freebsd.dk> <20140731205833.GU43962@funkthat.com> <58556.1406841499@critter.freebsd.dk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <58556.1406841499@critter.freebsd.dk> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Thu, 31 Jul 2014 14:42:46 -0700 (PDT) Cc: sjg@freebsd.org, arch@freebsd.org, marcel@freebsd.org, Phil Shafer X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 21:42:46 -0000 Poul-Henning Kamp wrote this message on Thu, Jul 31, 2014 at 21:18 +0000: > -------- > In message <20140731205833.GU43962@funkthat.com>, John-Mark Gurney writes: > > >$ man 3 sbuf > >No manual entry for sbuf > > Sorry: sbuf(9) I'm fine w/ an sbuf(3) man page that says, the API is the same as documented in sbuf(9) and you need to link against libsbuf, but we don't have that... I had to grep around in /usr/lib to find what library to use... P.S. If someone writes said man page, I will word smith/fixup/complete and commit it. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Fri Aug 1 05:04:41 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DCF6E662; Fri, 1 Aug 2014 05:04:40 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1blp0183.outbound.protection.outlook.com [207.46.163.183]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5DDBF208A; Fri, 1 Aug 2014 05:04:40 +0000 (UTC) Received: from BLUPR05MB724.namprd05.prod.outlook.com (10.141.207.154) by BLUPR05MB054.namprd05.prod.outlook.com (10.255.210.149) with Microsoft SMTP Server (TLS) id 15.0.995.14; Fri, 1 Aug 2014 05:04:38 +0000 Received: from BLUPR05CA010.namprd05.prod.outlook.com (10.255.219.168) by BLUPR05MB724.namprd05.prod.outlook.com (10.141.207.154) with Microsoft SMTP Server (TLS) id 15.0.995.14; Fri, 1 Aug 2014 05:04:37 +0000 Received: from BN1AFFO11FD035.protection.gbl (2a01:111:f400:7c10::197) by BLUPR05CA010.outlook.office365.com (2a01:111:e400:83f::40) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Fri, 1 Aug 2014 05:04:37 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BN1AFFO11FD035.mail.protection.outlook.com (10.58.52.159) with Microsoft SMTP Server (TLS) id 15.0.990.10 via Frontend Transport; Fri, 1 Aug 2014 05:04:37 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Thu, 31 Jul 2014 22:04:36 -0700 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s7154Yn91048; Thu, 31 Jul 2014 22:04:34 -0700 (PDT) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id 82C9E580A2; Thu, 31 Jul 2014 22:04:34 -0700 (PDT) To: Poul-Henning Kamp Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <58665.1406842603@critter.freebsd.dk> References: <201407312130.s6VLUFSP097778@idle.juniper.net> <58665.1406842603@critter.freebsd.dk> Comments: In-reply-to: Poul-Henning Kamp message dated "Thu, 31 Jul 2014 21:36:43 -0000." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Thu, 31 Jul 2014 22:04:34 -0700 Message-ID: <20140801050434.82C9E580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(51704005)(189002)(199002)(76482001)(21056001)(83072002)(97736001)(107046002)(95666004)(76506005)(81156004)(110136001)(106466001)(44976005)(48376002)(62966002)(89996001)(57986006)(70486001)(84676001)(76176999)(81542001)(50986999)(46102001)(99396002)(68736004)(74502001)(31966008)(93546004)(64706001)(92726001)(33656002)(77156001)(79102001)(69596002)(92566001)(87286001)(6806004)(20776003)(86362001)(4396001)(83322001)(74662001)(102836001)(105596002)(87936001)(102176002)(93916002)(85852003)(81342001)(104166001)(88136002)(50226001)(47776003)(80022001)(77982001)(85306004)(42262001); DIR:OUT; SFP:; SCL:1; SRVR:BLUPR05MB724; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 029097202E Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=sjg@juniper.net; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-OriginatorOrg: juniper.net Cc: arch@freebsd.org, John-Mark Gurney , marcel@freebsd.org, Phil Shafer X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Aug 2014 05:04:41 -0000 On Thu, 31 Jul 2014 21:36:43 +0000, Poul-Henning Kamp writes: >>sbuf looks like a simple case, returning either ENOMEM or the >>error code from the flush function. libxo can keep a "there's >>been an error" flag that the user can retrieve, but all the >>details of what's gone wrong would be lost. Or it can buffer >>the contents of warning messages and deliver it to the caller. >The point here is one of API design, and experience has shown >that either error-handling is convenient or it doesn't happen. > >I don't see the libxo case being any different from sbuf >in this respect, in fact I see it being almost even more >important because the readers are non-humans. The libxo case can be complicated by the structured output. If you have emitted and then you encounter an exception, even if you output a nice sorry about that your peer may not cope unless you also close all the open elements: etc. Of course I wouldn't be surprised if the lib already handles all that ;-) From owner-freebsd-arch@FreeBSD.ORG Fri Aug 1 15:51:32 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 208E2D64; Fri, 1 Aug 2014 15:51:32 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1blp0181.outbound.protection.outlook.com [207.46.163.181]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9516D21B5; Fri, 1 Aug 2014 15:51:30 +0000 (UTC) Received: from BLUPR05CA0074.namprd05.prod.outlook.com (10.141.20.44) by DM2PR05MB734.namprd05.prod.outlook.com (10.141.178.22) with Microsoft SMTP Server (TLS) id 15.0.995.14; Fri, 1 Aug 2014 15:51:28 +0000 Received: from BN1AFFO11FD026.protection.gbl (2a01:111:f400:7c10::135) by BLUPR05CA0074.outlook.office365.com (2a01:111:e400:855::44) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Fri, 1 Aug 2014 15:51:27 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BN1AFFO11FD026.mail.protection.outlook.com (10.58.52.86) with Microsoft SMTP Server (TLS) id 15.0.990.10 via Frontend Transport; Fri, 1 Aug 2014 15:51:27 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Fri, 1 Aug 2014 08:51:26 -0700 Received: from idle.juniper.net (idleski.juniper.net [172.25.4.26]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s71FpCn97304; Fri, 1 Aug 2014 08:51:13 -0700 (PDT) (envelope-from phil@juniper.net) Received: from idle.juniper.net (localhost [127.0.0.1]) by idle.juniper.net (8.14.4/8.14.3) with ESMTP id s71FoiQo003612; Fri, 1 Aug 2014 11:50:51 -0400 (EDT) (envelope-from phil@idle.juniper.net) Message-ID: <201408011550.s71FoiQo003612@idle.juniper.net> To: "Simon J. Gerraty" Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <20140801050434.82C9E580A2@chaos.jnpr.net> Date: Fri, 1 Aug 2014 11:50:44 -0400 From: Phil Shafer MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(189002)(199002)(164054003)(47776003)(54356999)(81542001)(69596002)(80022001)(4396001)(83322001)(85852003)(74662001)(95666004)(103666002)(92726001)(83072002)(97736001)(31966008)(74502001)(1941001)(105596002)(6806004)(50986999)(53416004)(68736004)(85306004)(76482001)(64706001)(86362001)(102836001)(21056001)(92566001)(81156004)(84676001)(50466002)(48376002)(87936001)(99396002)(106466001)(46102001)(77982001)(107046002)(110136001)(76506005)(79102001)(81342001)(20776003)(44976005); DIR:OUT; SFP:; SCL:1; SRVR:DM2PR05MB734; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 029097202E Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=phil@juniper.net; X-OriginatorOrg: juniper.net Cc: arch@freebsd.org, Poul-Henning Kamp , marcel@freebsd.org, John-Mark Gurney X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Aug 2014 15:51:32 -0000 "Simon J. Gerraty" writes: >Of course I wouldn't be surprised if the lib already handles all that ;-) No, I can't manufacture memory on the fly ;^) Hmmm... I could have an emergency mode where I stop trying to buffer and churn out a series of small write() calls to send close tags. Or perhaps just punting and refusing to write more once ENOMEM is seen is the right thing. Seeing broken output is better than limping along with output that looks right but isn't. Thanks, Phil From owner-freebsd-arch@FreeBSD.ORG Fri Aug 1 16:25:43 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4CABA2A1; Fri, 1 Aug 2014 16:25:43 +0000 (UTC) Received: from mail.xcllnt.net (mail.xcllnt.net [50.0.150.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 19F8825DF; Fri, 1 Aug 2014 16:25:42 +0000 (UTC) Received: from [192.168.2.25] (atc.xcllnt.net [50.0.150.213]) (authenticated bits=0) by mail.xcllnt.net (8.14.9/8.14.9) with ESMTP id s71GPYMb001144 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Fri, 1 Aug 2014 09:25:35 -0700 (PDT) (envelope-from marcel@xcllnt.net) Content-Type: multipart/signed; boundary="Apple-Mail=_134438E9-D07D-4C1D-A44D-D177AC448546"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Marcel Moolenaar In-Reply-To: <201408011550.s71FoiQo003612@idle.juniper.net> Date: Fri, 1 Aug 2014 09:25:34 -0700 Message-Id: References: <201408011550.s71FoiQo003612@idle.juniper.net> To: Phil Shafer X-Mailer: Apple Mail (2.1878.6) Cc: arch@freebsd.org, Poul-Henning Kamp , Marcel Moolenaar , John-Mark Gurney , "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Aug 2014 16:25:43 -0000 --Apple-Mail=_134438E9-D07D-4C1D-A44D-D177AC448546 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii On Aug 1, 2014, at 8:50 AM, Phil Shafer wrote: > "Simon J. Gerraty" writes: >> Of course I wouldn't be surprised if the lib already handles all that ;-) > > No, I can't manufacture memory on the fly ;^) > > Hmmm... I could have an emergency mode where I stop trying to buffer > and churn out a series of small write() calls to send close tags. > Or perhaps just punting and refusing to write more once ENOMEM is > seen is the right thing. Seeing broken output is better than limping > along with output that looks right but isn't. But broken output can have nasty side-effects due to parsers tripping over. What we try to prevent here (limping along) can very easily be introduced by us by emitting something that trips over parsers. We merely pushed the limp-along problem downstream from us. To get a reliable chain from producer to consumer, it may be best to have correct structure at all times and tackle the limp-along problem with constructs within that structure to signal that the data within that structure is complete and/or sound. For example: if stdout and stderr are separate tags in the XML output, then a third tag, say status, can be added to indicate that the stdout and stderr tags are accurate/complete. Or something along those lines. To me machine parseable output is only that if it is parseable at all times. Machines suck at interpreting their input when it suddenly stops being parseable. The best you can expect is garbage in, garbage out. While it sounds easy to just reject the entire input in that case (i.e. when some tool emits broken output), the fact that we may opt to just abort the output based on the fact that we may have emitted bits of it and can't undo it should be treated as an indication that pretty much any program is really behaving very similar and act upon it's input right away and not wait until it's seen the end and likes what it has seen. Rejecting the entire input when the structure is broken is just not something to bank on. $0.02 -- Marcel Moolenaar marcel@xcllnt.net --Apple-Mail=_134438E9-D07D-4C1D-A44D-D177AC448546 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlPbv34ACgkQpgWlLWHuifbtVACfRe8mobKENTajml41cAx0Y4Kk EzgAni0s1bjetR4Uu/gtrhpMoLQx/5Dy =DH8K -----END PGP SIGNATURE----- --Apple-Mail=_134438E9-D07D-4C1D-A44D-D177AC448546-- From owner-freebsd-arch@FreeBSD.ORG Fri Aug 1 17:59:15 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7048177C; Fri, 1 Aug 2014 17:59:15 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 2C7F9208C; Fri, 1 Aug 2014 17:59:14 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s71Hx6pC050606 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 1 Aug 2014 10:59:07 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s71Hx63m050605; Fri, 1 Aug 2014 10:59:06 -0700 (PDT) (envelope-from jmg) Date: Fri, 1 Aug 2014 10:59:06 -0700 From: John-Mark Gurney To: Marcel Moolenaar Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Message-ID: <20140801175906.GA50495@funkthat.com> Mail-Followup-To: Marcel Moolenaar , Phil Shafer , "Simon J. Gerraty" , Poul-Henning Kamp , arch@freebsd.org, Marcel Moolenaar References: <201408011550.s71FoiQo003612@idle.juniper.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Fri, 01 Aug 2014 10:59:07 -0700 (PDT) Cc: arch@freebsd.org, Poul-Henning Kamp , Marcel Moolenaar , Phil Shafer , "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Aug 2014 17:59:15 -0000 Marcel Moolenaar wrote this message on Fri, Aug 01, 2014 at 09:25 -0700: > On Aug 1, 2014, at 8:50 AM, Phil Shafer wrote: > > > "Simon J. Gerraty" writes: > >> Of course I wouldn't be surprised if the lib already handles all that ;-) > > > > No, I can't manufacture memory on the fly ;^) > > > > Hmmm... I could have an emergency mode where I stop trying to buffer > > and churn out a series of small write() calls to send close tags. > > Or perhaps just punting and refusing to write more once ENOMEM is > > seen is the right thing. Seeing broken output is better than limping > > along with output that looks right but isn't. > > But broken output can have nasty side-effects due to > parsers tripping over. What we try to prevent here > (limping along) can very easily be introduced by us > by emitting something that trips over parsers. We > merely pushed the limp-along problem downstream from > us. The just pushed the error handling further down stream... I'd trust a parse error more than someone handling all the odd edge cases of missing tags, or even failure to parse the error tag... IFF we have a proper DTD/spec that includes an error tag can we even think of making an error document parsable when an error occured... Either choice we make pushes the error handling down stream... One case, it's the parser, the other case it's the consumer of the parser, but in both cases the downstream HAS to properly handle the error however it is signaled... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Fri Aug 1 20:28:17 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 482A3A32; Fri, 1 Aug 2014 20:28:17 +0000 (UTC) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id 0DE1D228D; Fri, 1 Aug 2014 20:28:16 +0000 (UTC) Received: from critter.freebsd.dk (unknown [192.168.48.2]) by phk.freebsd.dk (Postfix) with ESMTP id B717F1578; Fri, 1 Aug 2014 20:28:09 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.14.9/8.14.9) with ESMTP id s71KS7u9063133; Fri, 1 Aug 2014 20:28:08 GMT (envelope-from phk@phk.freebsd.dk) To: Phil Shafer Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-reply-to: <201408011550.s71FoiQo003612@idle.juniper.net> From: "Poul-Henning Kamp" References: <201408011550.s71FoiQo003612@idle.juniper.net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <63131.1406924887.1@critter.freebsd.dk> Date: Fri, 01 Aug 2014 20:28:07 +0000 Message-ID: <63132.1406924887@critter.freebsd.dk> Cc: arch@freebsd.org, John-Mark Gurney , marcel@freebsd.org, "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Aug 2014 20:28:17 -0000 -------- In message <201408011550.s71FoiQo003612@idle.juniper.net>, Phil Shafer writes: >"Simon J. Gerraty" writes: >>Of course I wouldn't be surprised if the lib already handles all that ;-) > >No, I can't manufacture memory on the fly ;^) > >Hmmm... I could have an emergency mode where I stop trying to buffer >and churn out a series of small write() calls to send close tags. >Or perhaps just punting and refusing to write more once ENOMEM is >seen is the right thing. First of, this is not just ENOMEM, this is also invalid UTF-8 strings, NULL pointers and much more bogosity. >Seeing broken output is better than limping >along with output that looks right but isn't. The output should preferably be explicitly broken, so that nobody downstream mistakenly takes it and runs with it. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence.